Areca 1220 + Hitach 2TB + XFS Writeup

fibersnet2

n00b
Joined
Dec 28, 2009
Messages
13
Hello,

Thank you all for the guidance and help with upgrading my file server. As promised, here is the writeup that I made while going through it. (http://www.hardforum.com/showthread.php?t=1481766)

I had 7 x 300GB 7200.9 Segates that performed flawlessly for 4.5 years and it was time to upgrade. I chose the 2TB Hitachi drives because of the issues with the other drives. The Hitachis didn't have so many reviews so decided to take a chance and buy them.




Background:
Copied over 7x300GB drives to one 2TB drive (it wasn't full)
Initialize 4 x 2TB drives in 1 raidset, 2 volume sets.
One for the OS (64GB) and one for the files (3936GB)

Copy over the data from the lone 2TB drive onto the array of 4 x 2TB drives

Make the last 2TB drive part of the array, so in total 5 x 2TB in the raidset

This allowed me to measure how long it takes to expand the raidset, volume set and also expand the xfs filesytem was well




1) Instant to create Raidset
2) 35 Hours to Initialize a Volume Set with 4 Hard drives in Raid6
3) Instant to execute:
Code:
Monster:~# mkfs.xfs -d sunit=128,swidth=384 -i size=1k -l version=2,su=64k,lazy-count=1,size=128m /dev/sdb
meta-data=/dev/sdb               isize=1024   agcount=4, agsize=240234320 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=960937280, imaxpct=5
         =                       sunit=16     swidth=48 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

**Note, for sunit and swidth, xfs always displays it in terms of its own block size
sunit=16 is actually 16 xfs blocks which is 16 * 4096 = 64kB
swidth=48 blks = 48 xfs blocks, and 48/16 = 3 (3 data disks)

4) mount -o noatime,logbsize=256k,inode64,sunit=128,swidth=384,nobarrier /dev/sdb /monster


5) Quick Performance Tests with 4 Drives
Code:
Monster:/monster# dd if=/dev/zero of=10GB bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 55.4964 s, 189 MB/s
Monster:/monster# dd if=10GB of=/dev/null bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 42.7833 s, 245 MB/s
Monster:/monster#


6) /usr/sbin/bonnie++ -d /monster/fibersnet -m MonsterUpgraded_4*2TB_RAID6 -s 8g

Code:
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
MonsterUpgraded_ 8G   578  99 180080  44 95911  29  1008  99 239588  34 380.6   6
Latency             14873us     419ms     594ms   26093us   75284us   79660us
Version  1.96       ------Sequential Create------ --------Random Create--------
MonsterUpgraded_4*2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3140  20 +++++ +++ 10402  58  9498  62 +++++ +++  4559  27
Latency              1336ms     147us     356ms     215ms      21us    1495ms
1.96,1.96,MonsterUpgraded_4*2TB_RAID6,1,1262362134,8G,,578,99,180080,44,95911,29,1008,99,239588,34,380.6,6,16,,,,,3140,20,+++++,+++,10402,58,9498,62,+++++,+++,4559,27,14873us,419ms,594ms,26093us,75284us,79660us,1336ms,147us,356ms,215ms,21us,1495ms

7) Shutdown computer, fifth drive as a passthrough

8) mount -o noatime,logbsize=256k,inode64,nobarrier /dev/sdc /mnt/monstertrans/

mount -o noatime /dev/sdc /mnt/monstertrans/ ****** Does not work because you need inode64 otherwise it wont be able to access some files
Copy the files over


9) Reboot server and delete that passthrough disk

10) Add that disk to the Raidset. It will start migrating the volume sets.
64GB OS Volumeset took 42min
3936GB Monster Volumeset took VERY LONG 50hrs


11) In the mean time, I did a software raid 0, 4 x 300 7200.9 Seagate Array. I had a lot of time in between, 50 hours worth.

Code:
tv@media:~$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 54.2265 s, 193 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 48.4513 s, 216 MB/s

Delete files in random order...done.
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
media            8G   293  99 180602  66 84715  25  1181  97 220277  37 411.4  13
Latency             34864us     505ms     203ms   73881us   37896us     162ms
Version  1.96       ------Sequential Create------ --------Random Create--------
media               -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 20598  39 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             16628us     970us     992us     886us     807us     832us
1.96,1.96,media,1,1262495500,8G,,293,99,180602,66,84715,25,1181,97,220277,37,411.4,13,16,,,,,20598,39,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,34864us,505ms,203ms,73881us,37896us,162ms,16628us,970us,992us,886us,807us,832us


12) And on a software raid 0, 3 x 320 7200.10 Seagate Array

Code:
fibersnet@supernova:~$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 47.9861 s, 219 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 48.3318 s, 217 MB/s
Writing a byte at a time...done
Writing intelligently...done


Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
supernova        8G  1200  99 203071  40 82486  24  2772  98 213621  29 378.8  11
Latency             24507us    3516ms     195ms   23108us   68809us     154ms
Version  1.96       ------Sequential Create------ --------Random Create--------
supernova           -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3209  41 +++++ +++  4355  41  5704  67 +++++ +++  4609  37
Latency             65248us     240us    4546us   21776us    1957us    4157us
1.96,1.96,supernova,1,1262504813,8G,,1200,99,203071,40,82486,24,2772,98,213621,29,378.8,11,16,,,,,3209,41,+++++,+++,4355,41,5704,67,+++++,+++,4609,37,24507us,3516ms,195ms,23108us,68809us,154ms,65248us,240us,4546us,21776us,1957us,4157us
13) After 50 hours, the raid set expansion was done from 4 disk raid6 to 5 disk raid6 (2TB drives)
I believe there is some bug/ambiguity with the Areca utility. On the web interface/CLI AND the set of menus accessible from after post by hitting F10:

Initial scenario:
1 Raidset compromising of 4 x 2TB drives
1 Raid6 Volume set that is 64GB
1 Raid6 Volume Set that is 3936GB
Free Capacity: 0


After putting in the fifth 2TB drive and expanding the raid set
1 Raid6 Volume set that is 64GB
1 Raid6 Volumeset that is 3936GB
Free Capacity: 3333GB ????? <----Im not sure what this means or what it refers to, I was expecting it to say 2000GB since I put in an additional 2TB drive

Luckily, when I go to do a volume set expansion, it shows that I can make the 3936GB Volume set into a 5936GB volume set which is correct (3936 + 2000)GB

After 2 hours, the Volume Set expansion is completed:
1 Raid6 Volume set that is 64GB
1 Raid6 Volumeset that is 5936GB
Free Capacity: 0

So after the expansion, the Free Capacity Reads 0 as it should. I was a bit worried that it would let me expand the 3936GB volume set by 3333GB which would probably end up destroying the volume
but luckily it seems, it uses a different set of logic to determine how much I am able to expand the volume set by.



14) Do the volume set expansion of the larger (3936GB Volumeset) to 5936GB. This took only 2 hours.

15) Do some benchmarks:

on an ext3 that I did not get to set the stride settings for (this is on the smaller 64GB volume set)
Code:
fibersnet@Monster:~$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 92.9091 s, 113 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 43.911 s, 239 MB/s



Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Monster          8G   258  99 107813  52 71607  25  1000  98 287419  41 375.8   9
Latency             54068us    1358ms    3972ms   38089us     640ms     503ms
Version  1.96       ------Sequential Create------ --------Random Create--------
Monster             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 24204  88 +++++ +++ 21001  53 15844  58 +++++ +++ 19922  53
Latency             10381us    1005us    1048us     441us      37us      61us
1.96,1.96,Monster,1,1262654721,8G,,258,99,107813,52,71607,25,1000,98,287419,41,375.8,9,16,,,,,24204,88,+++++,+++,21001,53,15844,58,+++++,+++,19922,53,54068us,1358ms,3972ms,38089us,640ms,503ms,10381us,1005us,1048us,441us,37us,61us




Now an XFS that is mounted correctly: (mount -o noatime,logbsize=256k,inode64,sunit=128,swidth=384,nobarrier /dev/sdb /monster) swidth showing the fact that you have 384/128=3 data disks

Code:
fibersnet@Monster:/monster/fibersnet$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 48.5816 s, 216 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 32.0402 s, 327 MB/s


Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Monster          8G   578  99 197863  46 113270  34   841  99 310330  44 363.0   6
Latency             14931us     918ms     278ms   26520us     113ms     104ms
Version  1.96       ------Sequential Create------ --------Random Create--------
Monster             -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3089  22 +++++ +++ 11416  63  8636  61 +++++ +++  5279  31
Latency              1978ms      72us     240ms     415ms      31us     775ms
1.96,1.96,Monster,1,1262654371,8G,,578,99,197863,46,113270,34,841,99,310330,44,363.0,6,16,,,,,3089,22,+++++,+++,11416,63,8636,61,+++++,+++,5279,31,14931us,918ms,278ms,26520us,113ms,104ms,1978ms,72us,240ms,415ms,31us,775ms


16) And now for the final XFS expansion: (All of these commands happened instaneously, figuratively)

Code:
fibersnet@Monster:/monster/fibersnet$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              57G   12G   42G  22% /
tmpfs                 502M     0  502M   0% /lib/init/rw
udev                   10M  208K  9.8M   3% /dev
tmpfs                 502M     0  502M   0% /dev/shm
/dev/sdb              3.6T  1.4T  2.3T  37% /monster

fibersnet@Monster:/monster/fibersnet$ sudo xfs_growfs -n /monster
meta-data=/dev/sdb               isize=1024   agcount=4, agsize=240234320 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=960937280, imaxpct=5
         =                       sunit=16     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

fibersnet@Monster:/monster/fibersnet$ sudo xfs_growfs -d /monster
meta-data=/dev/sdb               isize=1024   agcount=4, agsize=240234320 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=960937280, imaxpct=5
         =                       sunit=16     swidth=64 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 960937280 to 1449218496

fibersnet@Monster:/monster/fibersnet$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              57G   12G   42G  22% /
tmpfs                 502M     0  502M   0% /lib/init/rw
udev                   10M  208K  9.8M   3% /dev
tmpfs                 502M     0  502M   0% /dev/shm
/dev/sdb              5.4T  1.4T  4.1T  25% /monster
fibersnet@Monster:/monster/fibersnet$


Done :)
 
You're in good hands with the Hitachi's. I needed lots of them for a hardware RAID6 array and was weary too at first- but tested everything else - Seagate, WD and friends (which turned out horribly) and a few friends with big arrays recommended Hitachi. Been beating up 20 of them in R6 over a week and they're flawless.

Keep your eyes peeled on the Hitachi owner's thread: http://www.hardforum.com/showthread.php?t=1482622
 
I'll need to build new array with 2tb drives soon, plan was to use wd green but now that they disabled tler that's a no go. I'll keep an eye on those hitachis might have to go with those cuz Seagate drives are bit scarry lately :D Would be nice if you could post an update on how are they holding out after a month or so. What kinda load do they see anyway?
 
Those init times seem to be a bit on the long side, was write cache set to "Enabled" rather than "Auto"? More often than not "Auto" defaults to "Disabled" and operations that do heavy writes, such as an init or expansion, take way longer.

If I recall, expanding from 8 to 11 drives in RAID6 took around 1 day when write cache was enabled. However, since write cache seems to cause problems w/ interleaved read/writes on the 2TB LP Seagate drives I'm using (e.g. audio dropouts) I disable write cache after init or expansion is finished. Maybe the Hitachi exhibits better behavior in this regard. My theory with the seagates is that the disk cache gets filled with write data and ends up in an I/O wait state if an interleaved read command is issued. Thus, e.g. if you try doing some disk intensive interleaved read/write commands like uncompressing archives, then try to stream mp3s at the same time, the mp3 stream will break up because it can't service the read IOs.
 
Those init times seem to be a bit on the long side, was write cache set to "Enabled" rather than "Auto"? More often than not "Auto" defaults to "Disabled" and operations that do heavy writes, such as an init or expansion, take way longer.

If I recall, expanding from 8 to 11 drives in RAID6 took around 1 day when write cache was enabled. However, since write cache seems to cause problems w/ interleaved read/writes on the 2TB LP Seagate drives I'm using (e.g. audio dropouts) I disable write cache after init or expansion is finished. Maybe the Hitachi exhibits better behavior in this regard. My theory with the seagates is that the disk cache gets filled with write data and ends up in an I/O wait state if an interleaved read command is issued. Thus, e.g. if you try doing some disk intensive interleaved read/write commands like uncompressing archives, then try to stream mp3s at the same time, the mp3 stream will break up because it can't service the read IOs.

Hvae you deactivated NCQ? Most drives don't do it very well anyway, and for linux raid my experiance was that disabling NCQ help performance quite a bit, esp if you have relatively memory allocation to read and write buffers in the kernel. I think you wouldn't see that kind of blocking if you shut off NCQ and let the OS effectively manage the access buffering.
 
I'm using my card w/ a Windows7 setup so I can't speak for software raid, but using the Areca hardware raid functionality, if I disable TCQ or NCQ I get some pretty poor performance so I leave those on and just live without the write cache. I can still do interleaved copy from the same array (e.g. duplicating a folder) at around 100MB/sec so performance is acceptable but if write cache was enabled I'd expect to get more like 200-300MB/sec. I'm just chalking it up to crappy NCQ implementation on Seagate.
 
I will have a report in a month on the status of the hard drives. They don't see too much load (mostly stores files) so I'm not sure how good of an indication the report will be but I will give it anyway.


The expansion time surprised me as well. Since my Arc1220 does not support a battery, I forced the Disk Write Cache Mode to Enabled:


CLI> sys showcfg
The System Configuration
=====================================================
System Beeper Setting : Enabled
Background Task Priority : High(80%)
JBOD/RAID Configuration : RAID
Max SATA Mode Supported : SATA300+NCQ
HDD Read Ahead Cache : Enabled
Volume Data Read Ahead : Normal
Stagger Power On Control : 0.7
Spin Down Idle HDD (Minutes) : Disabled
HDD SMART Status Polling : Enabled
Empty HDD Slot LED : ON
Auto Activate Incomplete Raid : Disabled
Disk Write Cache Mode : Enabled
Disk Capacity Truncation Mode : Multiples Of 10G
=====================================================
 
Back
Top