serious performance problems - halp!!1

[wilsontarbuckles]

in summary i am having problems making a hardware raid controller work for my softraid plans. i bought a supermicro case that came with an older motherboard/cpu combo and a megaraid 300-8x when my fileserver started dying:

http://cgi.ebay.com/3U-Supermicro-H...emQQptZLH_DefaultDomain_0?hash=item563a7b24d8

auction says a pair of 3ware 9500s, but that's not what arrived. anyways, all i want is 8 ports for linux softraid. i tried all these things and nothing has worked so far:
- create single-drive raid-0 logical drives and combine them with linux softraid. single drive performance is awesome- the controller cache works at 1GB/s, and single drive read speed is >90MB/s for cheap 1.5tb caviar greens once i do the partition alignment for their big sectors. as soon as i make an array out of them they sync at only 15MB/s, and the performance for synced filesystems is just horrible (much worse than unaligned single drives). besides the slowness, it seems stable. enabling writeback, adaptive read ahead, and cached writes on the controller don't fix the problem.
- disabling the bios with a jumper. some installers won't boot (udevadm settle times out), others boot but slowly and if they do no drives are presented to the operating system.
- disabling the bios in the megaraid bios utility. no more slow booting. megaraid.ko has a module parameter called unconf_disks that presents unconfigured disks to the OS. but, when you load the module udev goes crazy and fills up, like, sda through sdbf with devices. four of them (in my case, with four drives) actually appear to be the underlying drives. when i add them to an array scary page fault messages show up in dmesg, but it works at about single drive speeds. when i make a filesystem on top of the raid and try to mount it, it appears corrupt.

i haven't done softraid with anything but onboard controllers before, and these problems have really taken me by surprise.

i want to take my old, smaller, faster drives and make them into a 1+0 array under lvm for iscsi targets for VMs (ghetto san). random access is probably faster on the hardware raid, so i wouldn't mind using that. for raid 5, i'd much prefer software raid for a few reasons: it will be easier to recover / more recoverable if there's a serious failure, it supports >2TB devices where this card doesn't, and the performance metrics i care about are a little better. to wit, maybe i should just buy one of those 8-port supermicro things for $99.

 

[keenan]

Have you done any benchmarking accessing multiple single drives simultaneously?

 

[wilsontarbuckles]

Have you done any benchmarking accessing multiple single drives simultaneously?

no, this is a great idea.