Building a TB RAID5 array - help!

[SilverMK3]

I have the following parts in the mail arriving within the next few days:
4 x 320GB Seagate Barracuda 7200.10 Perpendicular SATA-II (NCIX.com)
1 x HighPoint RocketRAID 1740 SATA-II RAID Controller (Comtec.ca)
2 x 256MB DDR400 ([H]Forums)
1 x ATI Radeon 9600XT (local)
1 x 3COM Gigabit Ethernet NIC ([H]Forums)

Add this used barebones box I picked up earlier this week:
1 x AMD AthlonXP 2800+
1 x Asus A7N8X-X Motherboard
1 x big ol' case & 350Watt power supply

I'm in the process of building a 4-disk RAID-5 array on the cheap (relatively), but I'm concerned that the generic power supply won't be able to spin up 4 drives on boot. The controller supports staggered spinup, but not with the Seagate drives . The PS is rated at 17A on the 12V rails, does anyone know how much the Seagate 7200.10's draw on spinup?

Also, any suggestions for a n00b at building servers? I've used RAID-0 before, but I've never configured a WinXP installation to act as a server. Should I be disabling any extra services or worried about write-caching?

Thanks.

 

[RavenD]

Dont get a generic PSU. The PSU is the only component that can kill the rest of the system when it dies.

 

[AARGH!]

Not to rain on your parade, but your write speeds will be slower than a standalone drive, so I hope you weren't counting on blazing speeds with this. Software raid controllers do not do well. I returned the 1640 version of this card because of how slow it was. Proc was a 3.2Ghz P4 (478) with 2 gigs ram. I was horribly dissappointed.

AARGH!

 

[SilverMK3]

hmm. well that sucks. I know you get a penalty when writing to a RAID-5 array because of the distributed parity, but I don't really know how much of one to expect. I'll have multiple users reading and only a couple of us writing to the array, however, so it shouldn't be too bad I hope.

Maybe I'll do some HD-Tach measurements with one drive by itself, 3 drives in RAID5, and 4 drives in RAID5.

Either way, it should still be noticably faster than the Hawking Technologies HNAS1 enclosure that I've been using until now. It was limited to ~30mbits/sec or <4 MBytes/sec reading & writing. Its gonna be painful backing up 230GB+ at that speed (my project for the weekend).

I guess I'll have to start poking around in the FS/FT section for a new PSU :/

 

[unhappy_mage]

The 1740 is a completely different model than the 1640. It's based on the same architecture (I think...) as the 1820 and 22[23]0 cards, which is significantly faster than the older 1640 engine. Some simple benchmarks will show you STR and access times (HDTach being one of them). Give them a run and post the results here.

I think it'll be pretty obvious whether it's "fast" or not

 

[AARGH!]

All the parity checks are still being done by the CPU, not the card. Google some comparisions between the software based raid cards and hardware and you will see. They also mention the limitiation of the PCI bus as well in comparision to PCI-X cards.

AARGH!

 

[Madwand]

With the storage controller and GbE NIC both on the same PCI bus (Via socket A to boot), worrying about the performance of either component is somewhat moot -- you're not going to get that great performance out of the combination.

 

[Dew]

With the storage controller and GbE NIC both on the same PCI bus (Via socket A to boot), worrying about the performance of either component is somewhat moot -- you're not going to get that great performance out of the combination.

Agreed. My write speeds are acceptable on my fileserver, but it is very powerful at what its does, and is all PCIe for all the vital interconnects(LAN, Raid card). You are probably going to be looking at maybe 10MB/sec write speeds. I average 20MB/sec write on mine over LAN.

 

[Farva]

Dont get a generic PSU. The PSU is the only component that can kill the rest of the system when it dies.

QFT. i would not get a generic psu for this reason, or even a crappy psu, especially with "critial" data on my drives.

 

[InorganicMatter]

Get a caching controller. RAID-5 has very poor write speeds to the drives, and the more cache you have the less you will notice the poor writes. I've got a bunch of old SCSI drives in RAID-5, but I have 256MB of cache. It absoloutely flies because I've got the controller set to write-through mode. The OS just trows data into the cache, and immediately the system is responsive again. When I play Quake 4 on my array, it just pops up and says "Game Saved," and doesn't even hiccup. I played on a single drive system the other day, and every time it saved the game froze for a few seconds.

 

[unhappy_mage]

All the parity checks are still being done by the CPU, not the card. Google some comparisions between the software based raid cards and hardware and you will see. They also mention the limitiation of the PCI bus as well in comparision to PCI-X cards.

Yes, but with proper DMA you can reduce the impact of writes a lot, using system memory. The HPT driver, I think, takes the write request from the OS, performs the XOR calculation in memory, and then tells the controller to suck the data out of system memory. I have no evidence that this is how they implemented it, but it seems like a reasonable way to go about it.

Dew: What's the STR like locally? You may want to turn on jumbo packets and see if that improves things.

PS: AARGH!, do you put AARGH! at the end of your messages to sign them, or to express frustration with the situation? It's confusing

 

[Dew]

Dew: What's the STR like locally? You may want to turn on jumbo packets and see if that improves things.

It's definately an array limitation. I can read from the array over LAN at 47MB/sec sustained. iperf(100 connections) results in an average of 920MBps(~700MBps @ 1 connection). What is most annoying is that read performance drops to 25MB/sec when I have more than 2 concurrent file transfers. (This is the main fileserver for the LAN I host every month, average of 8 users leeching my 1.5TB TV/Movies collection. Average output after 12 hours: 1TB).

I wonder if it will improve any once I upgrade the fileserver (2GB Ram, X2 3800+@2.5GHz, basically moving my current gaming MOBO/CPU/RAM to the fileserver when I upgrade to C2D).

Performance when array was first brought online, 5 Drives, nothing on the array:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Fedora5         10G 34637  90 71612  25 38922  15 38857  92 176716  28 193.7   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  6665  62 +++++ +++  5498  51  6996  67 +++++ +++  3825  42
Fedora5,10G,34637,90,71612,25,38922,15,38857,92,176716,28,193.7,0,16,6665,62,+++++,+++,5498,51,6996,67,+++++,+++,3825,42
[dew@Fedora5 root]$ df -B 1M | grep terabyte
/dev/sda1              1220475         9   1220467   1% /mnt/terabyte

Performance now, 8 Drives, 1.8TB used:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Fedora5         10G 38184  82 106188  52 37533  25 44553  86 178953  46 127.3   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2205  77 +++++ +++  3663  71  2206  74 +++++ +++  1251  45
Fedora5,10G,38184,82,106188,52,37533,25,44553,86,178953,46,127.3,0,16,2205,77,+++++,+++,3663,71,2206,74,+++++,+++,1251,45
[dew@Fedora5 terabyte]$ df -B 1G | grep terabyte
/dev/sdb1                 2048      1860       189  91% /mnt/terabyte

 

[Madwand]

I average 20MB/sec write on mine over LAN.

Do you have write back caching enabled?

I can write to my 2320 (with write back caching, running RAID 5) at 50-70 MB/s over the network (very large file transfers).

E.g. ~64 MB/s:

-------------------------------------------------------------------------------
M:\test\test0\10.gb 10,000,000,000
-------------------------------------------------------------------------------
Directories processed = 1
Total data in bytes = 10,000,000,000
Elapsed time in sec. = 155.3
Action speed (MB/min) = 3864
Files copied = 1

 

[Dew]

Write Through = ~21MB/sec
Write Back = ~23MB/sec

I'm not overly concerned. This array gets a big brother in a few months. Then I'll set things up properly (Matching filesystem and raid block sizes, XFS on the physical disk[no more partitioning], etc).

 

[unhappy_mage]

I'd look at your samba configuration, and see if other protocols go faster. Set Apache's wwwroot to somewhere on the array, and see if you can pull anything at more than 20 MB/s off it, try NFS, try SCP. Since you're getting ~100 MB/s to disk and ~170 from, it shouldn't be an issue to read at full gigabit speeds. I'm guessing something is subtly wrong with Samba.

 

[Dew]

Heh, UM. Any chance you know what this line from top means:

Cpu(s): 1.3% us, 27.9% sy, 0.0% ni, 0.0% id, 54.8% wa, 3.0% hi, 13.0% si

That was the scenarion when pulling from 10 concurrent ftp connections and 1 samba connection. I can't find any reference as to what wa means, I'm assuming wait, as in the cpu is being tied up. If so, this may be cause of the problem.

BTW, I was able to write to the array at 40MB/sec using samba and ftp concurrently.

 

[unhappy_mage]

IOWait. That (usually) means that the process is running away from the disk - it's ready to send more data, but it's waiting on the disk to retrieve what it asked for. You might see if this thread is relevant, or try mounting your XFS filesystem with "sunit=128,swidth=(N-1)" (for 64k stripes across N disks) and see if that helps. The noatime option will also make things go faster; check 'man mount' for more details.

XFS will try to do writes in multiple places on the disk if it can, so this may mean you get much better read throughput off the disk than you'd expect. I dunno, worth a try

 

[Dew]

Well, looks like I found at least part of the problem. The sunit size is 0, cant mount with it larger than 8 (which means I created the filesystem that).
Here's the xfs_info (mounted with sunit=8,swidth=8):

meta-data=/dev/sdb1              isize=256    agcount=55, agsize=9764821 blks
         =                       sectsz=512
data     =                       bsize=4096   blocks=536870202, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

Also, looks like that tweak won't for for FC5 x64.
Here is my file list along with values for /proc/sys/vm/
block_dump = 0
dirty_background_ratio = 10
dirty_expire_centisecs = 3000
dirty_ratio = 40
dirty_writeback_centisecs = 500
drop_caches = 0
hugetlb_shm_group = 0
laptop_mode = 0
legacy_va_layout = 0
lowmem_reserve_ratio = 256 256 32
max_map_count = 65536
min_free_kbytes = 4053
nr_hugepages = 0
nr_pdflush_threads = 2
overcommit_memory = 0
overcommit_ratio = 50
page-cluster = 3
percpu_pagelist_fraction = 0
swappiness = 60
swap_token_timeout = 300 0
vfs_cache_pressure = 100
zone_reclaim_interval = 30
zone_reclaim_mode = 0


I'll play around with the atime settings, see if that makes a difference.
Edit found a xfs tweak guide. It suggested the following:
-o noatime,nodiratime,osyncisdsync

Seems to have worked. Pulling 8 symultaneous files from two systems at a total BW of 45MB/sec average

IOWAIT still hovers at 45%, but as long as it can do 45MB/sec with 10 separate users, I'll be happy.

BTW, the best I can do is 50MB/sec at the start of a file(4GB file), it drops down to 48MB/sec after about 10 secs. This is writing to my 160Gig Raptor.

Write speeds were unaffected.

 

[Dew]

Well, I fixed it up.

Read Speed for a single file: 49.5MB/sec Average
Write Speed for a single file: 39.2MB/sec Average

Read Speed for multiple files(More than 4): 43.4MB/sec Average
Write Speed for multiple files(More than 4): 35.8MB/sec Average


Fix action:
in /etc/samba/smb.conf
socket options = TCP_NODELAY SO_RCVBUF=262140 SO_SNDBUF=262140

This makes me happy!

 

[unhappy_mage]

I was right, Samba it is Glad you got things working better.

 

[general]

That's a .9TB array, not a TB one.

 

[SilverMK3]

That's a .9TB array, not a TB one.

shhhh... Its a 0.96TB array

Here are the benchmarks as promised! Definately bumping into a serious CPU & PCI Bus bottleneck here:

Single 320GB Seagate 7200.10 on my main rig (see sig):

Three x 320GB in RAID5 on my server:

Four x 320GB in RAID5 on my server:

I'm sure the performance will only decrease once I get my gigabit NIC in the mail (Though I currently have no problems saturating the 100Mbit onboard LAN). I currently pin the CPU usage and Kernal time at 95% whenever I write to the array.

Anyone have any solutions that DON'T involve getting a new RAID controller and rebuilding the array? I'm seriously contemplating upgrading the mobo & cpu... anyone have an extra Athlon64 & S939 mobo kicking around?? lol.

 

[Madwand]

The PCI bus seems to be performing pretty well. Since you already have all this HW set up, and the transition to something faster would be wasteful and not really cheap, I'd suggest just running with it. If then you find yourself wishing for faster data transfer performance in practice, then, think again if an upgrade would give you a significant boost, and go from there.

 

[SilverMK3]

Madwand: You make a good point about the cost/benefit thing... You really think there wouldn't be much of a performance increase in going to a Athlon64 platform? I'm maxing out my poor little AthlonXP 2800+ every time I write to the array over the network.

My next issue stems from my naively thinking that I could use the RAID controller's OCE (Online Capacity Expansion) to add the fourth disk to the array and simply resize the Windows partition to include the added space. Oops. Looks like there's issues doing this with a RAID array. Does anyone know of a good open source utility to do this with? I was going to use the GParted LiveUSB image but it can't see the array at all.

Does PartitionMagic or whatever the current commercial application for this type of thing support RAID arrays?

Anyway. TIA for the help!

 

[Madwand]

Partition Magic's a bit dated, but IIRC, I've used Acronis Disk Director to expand a RAID partition. But I prefer to have a full backup for greater flexibility and so that I don't rely on such tools to work perfectly.

I don't know why you're getting such high CPU utilization; I don't think it's just the CPU itself.. I suggest trying the new NIC when it arrives; enabling jumbo frames (enable it on both NICs and try a direct wire connection with static IPs if your switch doesn't support jumbo frames). Also try varying interrupt moderation. Also try modifying Tcp1323Opts and TcpWindowSize (Google for more on that).

You can probably gain something with a new MB by moving the NIC off the PCI bus (and also thereby potentially reduce the impact of jumbo frames), and something more if you get a PCI bus which supports 66 MHz for your RAID controller (at even more expense). But you may be bottlenecked on the other computers' hard drives, and the relative gain might not be worth the expense.

 

[AARGH!]

As I brought up earlier, the non-proc PCI cards do not perform well, due to CPU load, and the PCI issues. You can either go with PCI card with onboard proc to take load off CPU, but you will still have bus issues, but less costly than jumping up to whole new motherboard and a PCIE/X card to avoid both issues.

I eventually went and found an ABit IC7 Max3 that has 6 SATA ports onboard and running RAID 0 on two separate pairs, and the other two ports running a 3 drive hot swap bay to use for backups. (before anyone asks 3rd bay is for spare) This config I have 1.2TB for storage and right now 640gig for backups, but it's only limited by number of spare drives I want to have.

Good luck!

 

[SilverMK3]

I used Acronis Disk Director and it worked great! It resized the partition in ~45 seconds without even having to reboot. Sweet.

Now I need to contemplate the rest of the system. I have a hookup on a nearly-free Athlon64 3200+, just need to find a S939 motherboard with a chipset-level GBe NIC and hopefully an AGP slot or integrated video... The reality is that this IS fast enough for everything I'm going to with it though - I just can't stop looking at the potential to go even faster. lol.

 

[unhappy_mage]

As I brought up earlier, the non-proc PCI cards do not perform well, due to CPU load, and the PCI issues.

Really? Tweakers.net might have something to say about that...

If STR is your thing, host-based raid is the way to go. The Highpoint and Broadcom cards up there around 350 MB/s are both host-based; the Areca at 250ish is hardware.

Granted, this doesn't happen in every test - in desktop-type workloads, for example, 8*r5 on the Areca with 1GB cache scores around three times as many points as the Highpoint - but that simply furthers my point. All generalizations are false, dammit