fibersnet2
n00b
- Joined
- Dec 28, 2009
- Messages
- 13
Hello,
Thank you all for the guidance and help with upgrading my file server. As promised, here is the writeup that I made while going through it. (http://www.hardforum.com/showthread.php?t=1481766)
I had 7 x 300GB 7200.9 Segates that performed flawlessly for 4.5 years and it was time to upgrade. I chose the 2TB Hitachi drives because of the issues with the other drives. The Hitachis didn't have so many reviews so decided to take a chance and buy them.
Background:
Copied over 7x300GB drives to one 2TB drive (it wasn't full)
Initialize 4 x 2TB drives in 1 raidset, 2 volume sets.
One for the OS (64GB) and one for the files (3936GB)
Copy over the data from the lone 2TB drive onto the array of 4 x 2TB drives
Make the last 2TB drive part of the array, so in total 5 x 2TB in the raidset
This allowed me to measure how long it takes to expand the raidset, volume set and also expand the xfs filesytem was well
1) Instant to create Raidset
2) 35 Hours to Initialize a Volume Set with 4 Hard drives in Raid6
3) Instant to execute:
**Note, for sunit and swidth, xfs always displays it in terms of its own block size
sunit=16 is actually 16 xfs blocks which is 16 * 4096 = 64kB
swidth=48 blks = 48 xfs blocks, and 48/16 = 3 (3 data disks)
4) mount -o noatime,logbsize=256k,inode64,sunit=128,swidth=384,nobarrier /dev/sdb /monster
5) Quick Performance Tests with 4 Drives
6) /usr/sbin/bonnie++ -d /monster/fibersnet -m MonsterUpgraded_4*2TB_RAID6 -s 8g
7) Shutdown computer, fifth drive as a passthrough
8) mount -o noatime,logbsize=256k,inode64,nobarrier /dev/sdc /mnt/monstertrans/
mount -o noatime /dev/sdc /mnt/monstertrans/ ****** Does not work because you need inode64 otherwise it wont be able to access some files
Copy the files over
9) Reboot server and delete that passthrough disk
10) Add that disk to the Raidset. It will start migrating the volume sets.
64GB OS Volumeset took 42min
3936GB Monster Volumeset took VERY LONG 50hrs
11) In the mean time, I did a software raid 0, 4 x 300 7200.9 Seagate Array. I had a lot of time in between, 50 hours worth.
12) And on a software raid 0, 3 x 320 7200.10 Seagate Array
13) After 50 hours, the raid set expansion was done from 4 disk raid6 to 5 disk raid6 (2TB drives)
I believe there is some bug/ambiguity with the Areca utility. On the web interface/CLI AND the set of menus accessible from after post by hitting F10:
Initial scenario:
1 Raidset compromising of 4 x 2TB drives
1 Raid6 Volume set that is 64GB
1 Raid6 Volume Set that is 3936GB
Free Capacity: 0
After putting in the fifth 2TB drive and expanding the raid set
1 Raid6 Volume set that is 64GB
1 Raid6 Volumeset that is 3936GB
Free Capacity: 3333GB ????? <----Im not sure what this means or what it refers to, I was expecting it to say 2000GB since I put in an additional 2TB drive
Luckily, when I go to do a volume set expansion, it shows that I can make the 3936GB Volume set into a 5936GB volume set which is correct (3936 + 2000)GB
After 2 hours, the Volume Set expansion is completed:
1 Raid6 Volume set that is 64GB
1 Raid6 Volumeset that is 5936GB
Free Capacity: 0
So after the expansion, the Free Capacity Reads 0 as it should. I was a bit worried that it would let me expand the 3936GB volume set by 3333GB which would probably end up destroying the volume
but luckily it seems, it uses a different set of logic to determine how much I am able to expand the volume set by.
14) Do the volume set expansion of the larger (3936GB Volumeset) to 5936GB. This took only 2 hours.
15) Do some benchmarks:
on an ext3 that I did not get to set the stride settings for (this is on the smaller 64GB volume set)
Now an XFS that is mounted correctly: (mount -o noatime,logbsize=256k,inode64,sunit=128,swidth=384,nobarrier /dev/sdb /monster) swidth showing the fact that you have 384/128=3 data disks
16) And now for the final XFS expansion: (All of these commands happened instaneously, figuratively)
Done
Thank you all for the guidance and help with upgrading my file server. As promised, here is the writeup that I made while going through it. (http://www.hardforum.com/showthread.php?t=1481766)
I had 7 x 300GB 7200.9 Segates that performed flawlessly for 4.5 years and it was time to upgrade. I chose the 2TB Hitachi drives because of the issues with the other drives. The Hitachis didn't have so many reviews so decided to take a chance and buy them.
Background:
Copied over 7x300GB drives to one 2TB drive (it wasn't full)
Initialize 4 x 2TB drives in 1 raidset, 2 volume sets.
One for the OS (64GB) and one for the files (3936GB)
Copy over the data from the lone 2TB drive onto the array of 4 x 2TB drives
Make the last 2TB drive part of the array, so in total 5 x 2TB in the raidset
This allowed me to measure how long it takes to expand the raidset, volume set and also expand the xfs filesytem was well
1) Instant to create Raidset
2) 35 Hours to Initialize a Volume Set with 4 Hard drives in Raid6
3) Instant to execute:
Code:
Monster:~# mkfs.xfs -d sunit=128,swidth=384 -i size=1k -l version=2,su=64k,lazy-count=1,size=128m /dev/sdb
meta-data=/dev/sdb isize=1024 agcount=4, agsize=240234320 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=960937280, imaxpct=5
= sunit=16 swidth=48 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
**Note, for sunit and swidth, xfs always displays it in terms of its own block size
sunit=16 is actually 16 xfs blocks which is 16 * 4096 = 64kB
swidth=48 blks = 48 xfs blocks, and 48/16 = 3 (3 data disks)
4) mount -o noatime,logbsize=256k,inode64,sunit=128,swidth=384,nobarrier /dev/sdb /monster
5) Quick Performance Tests with 4 Drives
Code:
Monster:/monster# dd if=/dev/zero of=10GB bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 55.4964 s, 189 MB/s
Monster:/monster# dd if=10GB of=/dev/null bs=1M count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 42.7833 s, 245 MB/s
Monster:/monster#
6) /usr/sbin/bonnie++ -d /monster/fibersnet -m MonsterUpgraded_4*2TB_RAID6 -s 8g
Code:
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
MonsterUpgraded_ 8G 578 99 180080 44 95911 29 1008 99 239588 34 380.6 6
Latency 14873us 419ms 594ms 26093us 75284us 79660us
Version 1.96 ------Sequential Create------ --------Random Create--------
MonsterUpgraded_4*2 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 3140 20 +++++ +++ 10402 58 9498 62 +++++ +++ 4559 27
Latency 1336ms 147us 356ms 215ms 21us 1495ms
1.96,1.96,MonsterUpgraded_4*2TB_RAID6,1,1262362134,8G,,578,99,180080,44,95911,29,1008,99,239588,34,380.6,6,16,,,,,3140,20,+++++,+++,10402,58,9498,62,+++++,+++,4559,27,14873us,419ms,594ms,26093us,75284us,79660us,1336ms,147us,356ms,215ms,21us,1495ms
7) Shutdown computer, fifth drive as a passthrough
8) mount -o noatime,logbsize=256k,inode64,nobarrier /dev/sdc /mnt/monstertrans/
mount -o noatime /dev/sdc /mnt/monstertrans/ ****** Does not work because you need inode64 otherwise it wont be able to access some files
Copy the files over
9) Reboot server and delete that passthrough disk
10) Add that disk to the Raidset. It will start migrating the volume sets.
64GB OS Volumeset took 42min
3936GB Monster Volumeset took VERY LONG 50hrs
11) In the mean time, I did a software raid 0, 4 x 300 7200.9 Seagate Array. I had a lot of time in between, 50 hours worth.
Code:
tv@media:~$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 54.2265 s, 193 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 48.4513 s, 216 MB/s
Delete files in random order...done.
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
media 8G 293 99 180602 66 84715 25 1181 97 220277 37 411.4 13
Latency 34864us 505ms 203ms 73881us 37896us 162ms
Version 1.96 ------Sequential Create------ --------Random Create--------
media -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 20598 39 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency 16628us 970us 992us 886us 807us 832us
1.96,1.96,media,1,1262495500,8G,,293,99,180602,66,84715,25,1181,97,220277,37,411.4,13,16,,,,,20598,39,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,34864us,505ms,203ms,73881us,37896us,162ms,16628us,970us,992us,886us,807us,832us
12) And on a software raid 0, 3 x 320 7200.10 Seagate Array
Code:
fibersnet@supernova:~$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 47.9861 s, 219 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 48.3318 s, 217 MB/s
Writing a byte at a time...done
Writing intelligently...done
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
supernova 8G 1200 99 203071 40 82486 24 2772 98 213621 29 378.8 11
Latency 24507us 3516ms 195ms 23108us 68809us 154ms
Version 1.96 ------Sequential Create------ --------Random Create--------
supernova -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 3209 41 +++++ +++ 4355 41 5704 67 +++++ +++ 4609 37
Latency 65248us 240us 4546us 21776us 1957us 4157us
1.96,1.96,supernova,1,1262504813,8G,,1200,99,203071,40,82486,24,2772,98,213621,29,378.8,11,16,,,,,3209,41,+++++,+++,4355,41,5704,67,+++++,+++,4609,37,24507us,3516ms,195ms,23108us,68809us,154ms,65248us,240us,4546us,21776us,1957us,4157us
I believe there is some bug/ambiguity with the Areca utility. On the web interface/CLI AND the set of menus accessible from after post by hitting F10:
Initial scenario:
1 Raidset compromising of 4 x 2TB drives
1 Raid6 Volume set that is 64GB
1 Raid6 Volume Set that is 3936GB
Free Capacity: 0
After putting in the fifth 2TB drive and expanding the raid set
1 Raid6 Volume set that is 64GB
1 Raid6 Volumeset that is 3936GB
Free Capacity: 3333GB ????? <----Im not sure what this means or what it refers to, I was expecting it to say 2000GB since I put in an additional 2TB drive
Luckily, when I go to do a volume set expansion, it shows that I can make the 3936GB Volume set into a 5936GB volume set which is correct (3936 + 2000)GB
After 2 hours, the Volume Set expansion is completed:
1 Raid6 Volume set that is 64GB
1 Raid6 Volumeset that is 5936GB
Free Capacity: 0
So after the expansion, the Free Capacity Reads 0 as it should. I was a bit worried that it would let me expand the 3936GB volume set by 3333GB which would probably end up destroying the volume
but luckily it seems, it uses a different set of logic to determine how much I am able to expand the volume set by.
14) Do the volume set expansion of the larger (3936GB Volumeset) to 5936GB. This took only 2 hours.
15) Do some benchmarks:
on an ext3 that I did not get to set the stride settings for (this is on the smaller 64GB volume set)
Code:
fibersnet@Monster:~$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 92.9091 s, 113 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 43.911 s, 239 MB/s
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Monster 8G 258 99 107813 52 71607 25 1000 98 287419 41 375.8 9
Latency 54068us 1358ms 3972ms 38089us 640ms 503ms
Version 1.96 ------Sequential Create------ --------Random Create--------
Monster -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 24204 88 +++++ +++ 21001 53 15844 58 +++++ +++ 19922 53
Latency 10381us 1005us 1048us 441us 37us 61us
1.96,1.96,Monster,1,1262654721,8G,,258,99,107813,52,71607,25,1000,98,287419,41,375.8,9,16,,,,,24204,88,+++++,+++,21001,53,15844,58,+++++,+++,19922,53,54068us,1358ms,3972ms,38089us,640ms,503ms,10381us,1005us,1048us,441us,37us,61us
Now an XFS that is mounted correctly: (mount -o noatime,logbsize=256k,inode64,sunit=128,swidth=384,nobarrier /dev/sdb /monster) swidth showing the fact that you have 384/128=3 data disks
Code:
fibersnet@Monster:/monster/fibersnet$ dd if=/dev/zero of=10GB bs=1M count=10000; dd if=10GB of=/dev/null bs=1M count=10000; /usr/sbin/bonnie++ -d . -s 8g
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 48.5816 s, 216 MB/s
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 32.0402 s, 327 MB/s
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Monster 8G 578 99 197863 46 113270 34 841 99 310330 44 363.0 6
Latency 14931us 918ms 278ms 26520us 113ms 104ms
Version 1.96 ------Sequential Create------ --------Random Create--------
Monster -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 3089 22 +++++ +++ 11416 63 8636 61 +++++ +++ 5279 31
Latency 1978ms 72us 240ms 415ms 31us 775ms
1.96,1.96,Monster,1,1262654371,8G,,578,99,197863,46,113270,34,841,99,310330,44,363.0,6,16,,,,,3089,22,+++++,+++,11416,63,8636,61,+++++,+++,5279,31,14931us,918ms,278ms,26520us,113ms,104ms,1978ms,72us,240ms,415ms,31us,775ms
16) And now for the final XFS expansion: (All of these commands happened instaneously, figuratively)
Code:
fibersnet@Monster:/monster/fibersnet$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 57G 12G 42G 22% /
tmpfs 502M 0 502M 0% /lib/init/rw
udev 10M 208K 9.8M 3% /dev
tmpfs 502M 0 502M 0% /dev/shm
/dev/sdb 3.6T 1.4T 2.3T 37% /monster
fibersnet@Monster:/monster/fibersnet$ sudo xfs_growfs -n /monster
meta-data=/dev/sdb isize=1024 agcount=4, agsize=240234320 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=960937280, imaxpct=5
= sunit=16 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
fibersnet@Monster:/monster/fibersnet$ sudo xfs_growfs -d /monster
meta-data=/dev/sdb isize=1024 agcount=4, agsize=240234320 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=960937280, imaxpct=5
= sunit=16 swidth=64 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=16 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 960937280 to 1449218496
fibersnet@Monster:/monster/fibersnet$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 57G 12G 42G 22% /
tmpfs 502M 0 502M 0% /lib/init/rw
udev 10M 208K 9.8M 3% /dev
tmpfs 502M 0 502M 0% /dev/shm
/dev/sdb 5.4T 1.4T 4.1T 25% /monster
fibersnet@Monster:/monster/fibersnet$
Done