Does anyone know why my zpool iostat looks like this? http://pastebin.com/gpApGknB
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
Help with zfs
- Thread starter xtkxhom3r
- Start date
well what im asking is why is the alloc/free blank for my 4tb drives? for example i just added 2 3tb drives and they show data
and what is going on with my space? tank shows 19.9 the raidz1 shows 14.4 and the gui shows 17.2 lol....
Code:
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 88.0G 19.9T 0 1.20K 987 27.5M
raidz1 62.5G 14.4T 0 1.08K 61 17.0M
sda - - 0 110 643 6.42M
sdb - - 0 103 443 6.42M
sdd - - 0 78 445 6.42M
sde - - 0 80 466 6.42M
sdc 8.68G 2.71T 0 60 463 3.56M
sdf 16.8G 2.70T 0 64 462 6.90M
---------- ----- ----- ----- ----- ----- -----and what is going on with my space? tank shows 19.9 the raidz1 shows 14.4 and the gui shows 17.2 lol....
That's just the way iostat displays the info.
iostat is showing how raw data is dispersed across your vdevs.
There is some discrepancy between human readable values and internal values because of the way free space is calculated and file system overhead. If you want human readable value you use the command df -h
iostat is showing how raw data is dispersed across your vdevs.
There is some discrepancy between human readable values and internal values because of the way free space is calculated and file system overhead. If you want human readable value you use the command df -h
How did you create the pool? You said you added "two 3TB drives"...
On my raidZ1 pools they look like this http://pastebin.com/tSuYdPaA
you can do a zpool history and it should give you the commands used to create it.
zpool history z01
History for 'z01':
2012-08-04.15:58:58 zpool create z01 raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf -f
NOTE: the pool z01 was created on a linux box using zfsonlinux and subsequently moved to a solaris machine. This explains why the device names dont match.
Its almost like you have a raidz1 pool and a mirror which might explain the storage reports as well (the raidz1 pool is 14TB and the mirror is 2.7TB (the size zfs typically reports for a 3tb drive).
SDC and SDF look weird in your report, almost like they are a separate mirror under "tank".
On my raidZ1 pools they look like this http://pastebin.com/tSuYdPaA
you can do a zpool history and it should give you the commands used to create it.
zpool history z01
History for 'z01':
2012-08-04.15:58:58 zpool create z01 raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf -f
NOTE: the pool z01 was created on a linux box using zfsonlinux and subsequently moved to a solaris machine. This explains why the device names dont match.
Its almost like you have a raidz1 pool and a mirror which might explain the storage reports as well (the raidz1 pool is 14TB and the mirror is 2.7TB (the size zfs typically reports for a 3tb drive).
SDC and SDF look weird in your report, almost like they are a separate mirror under "tank".
Last edited:
here is the history
thanks for the help guys its really appreciated
Code:
[~]$ sudo zpool history -- INSERT --
History for 'tank':
2013-09-01.15:23:48 zpool create tank raidz /dev/sda /dev/sdb /dev/sdd /dev/sde
2013-09-01.15:23:53 zfs set compression=lzjb tank
2013-09-01.15:23:57 zfs set checksum=off tank
2013-09-01.15:24:02 zfs set atime=off tank
2013-09-01.15:24:05 zfs set primarycache=metadata tank
2013-09-01.15:24:10 zfs set recordsize=16k tank
2013-09-01.15:47:56 zfs set compression=off tank
2013-09-01.15:48:16 zfs set compression=lzjb tank
2013-09-01.16:36:14 zpool add -f tank /dev/sdc /dev/sdfthanks for the help guys its really appreciated
Yeah, looks like you have added a mirror vdev to your raiz1 pool. To expand this pool you would need to add another raiz1 vdev.
something like
zpool add tank raidz /dev/sdf /dev/sdg /dev/sdh /dev/sdi
ok so my 4tb drives are sda,sdb,sdd,sde
and my 3tb drives are sdc,sdf
so i have to destroy the tank and make it with the 4tb drives first like this
zpool create tank raidz /dev/sda /dev/sdb /dev/sdd /dev/sde
then add the 3tb drives on another vdev like this
zpool add tank raidz /dev/sdc /dev/sdf
correct?
will it still be one big pool? i dont want to have 2 separate pools
and my 3tb drives are sdc,sdf
so i have to destroy the tank and make it with the 4tb drives first like this
zpool create tank raidz /dev/sda /dev/sdb /dev/sdd /dev/sde
then add the 3tb drives on another vdev like this
zpool add tank raidz /dev/sdc /dev/sdf
correct?
will it still be one big pool? i dont want to have 2 separate pools
raidz needs a minimum of three drives.
that should work (i honestly haven't tried this). My pools consist of a mirror (boot), a pool of 7x2tb drives and another separate pool of 5x1tb drives. This configuration works well for me but i understand why you want to add all the storage to a single p ool.
You know you can always test it using files:
Create a bunch of files using mkfile (file1.zfs, file2.zfs) etc.
do something like create 4 2gb files and raidz1 them. then create three 1gb files and add them to the pool.
you can see the results and make sure it does what you want before you do it to your real disks.
ok so i just created the pool again and added the 2 3tb drives correctly and now i dont have access to all my space....
Code:
[/tank]$ sudo zpool iostat -v -- INSERT --
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
tank 1.70M 19.9T 0 11 911 14.2K
raidz1 1.28M 14.5T 0 7 590 8.73K
sda - - 0 3 9.08K 15.9K
sdb - - 0 3 6.86K 15.9K
sdd - - 0 3 6.50K 15.6K
sde - - 0 3 6.81K 15.6K
raidz1 424K 5.44T 0 4 321 5.46K
sdc - - 0 3 6.83K 20.0K
sdf - - 0 3 7.00K 20.0K
---------- ----- ----- ----- ----- ----- -----
[/tank]$ sudo zpool status -- INSERT --
pool: tank
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdf ONLINE 0 0 0
errors: No known data errors
[/tank]$ sudo zpool list -- INSERT --
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
tank 19.9T 1.70M 19.9T 0% 1.00x ONLINE -
[/tank]$ df -h -- INSERT --
Filesystem Size Used Avail Use% Mounted on
/dev/sdg1 286G 14G 258G 5% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 3.9G 4.0K 3.9G 1% /dev
tmpfs 787M 1.2M 786M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.9G 80K 3.9G 1% /run/shm
none 100M 12K 100M 1% /run/user
tank 14T 176K 14T 1% /tank
[/tank]$ sudo zpool history -- INSERT --
History for 'tank':
2013-09-01.20:43:22 zpool create tank raidz /dev/sda /dev/sdb /dev/sdd /dev/sde
2013-09-01.20:45:16 zpool add tank raidz -f /dev/sdc /dev/sdf
2013-09-01.20:46:11 zfs set compression=lzjb tank
2013-09-01.20:46:20 zfs set checksum=off tank
2013-09-01.20:46:30 zfs set atime=off tank
2013-09-01.20:46:38 zfs set primarycache=metadata tank
2013-09-01.20:46:45 zfs set recordsize=16k tankthe fact the devices are listed correctly (raidz1-0 and raidz1-1) looks a lot better.
How much space are you expecting and why?
i dont think "du" is a good tool to use against zfs (use zpool list).
Still not sure about the two disk (raidz1-1) vdev. Perhaps someone else can chime in here as i always thought raidz1 needs 2 disks (two data one parity)
i was expecting to get around 18-20tb
when i made a pool with just the 4tb drives i was getting 14tb thats why i think its not adding the 3tb drives to the pool
and zpool list shows 19.9tb just like iostat
im making a 16tb file to see if it will fit in the pool...technically it should but the gui only shows 14tb free so we will see... at 2gb/s it should be done by tomorrow
when i made a pool with just the 4tb drives i was getting 14tb thats why i think its not adding the 3tb drives to the pool
and zpool list shows 19.9tb just like iostat
im making a 16tb file to see if it will fit in the pool...technically it should but the gui only shows 14tb free so we will see... at 2gb/s it should be done by tomorrow
You have 4 - 4TB disks and 2 - 3TB disks. Of those, one of each is lost to parity, meaning you would have 15TB raw.
15 TB raw is actually 15000000 MB, When you convert to Bytes (powers of 1024) you get 13,965 GB of space. Round up and you get 14TB
Which is exactly what is reported by df -h
15 TB raw is actually 15000000 MB, When you convert to Bytes (powers of 1024) you get 13,965 GB of space. Round up and you get 14TB
Which is exactly what is reported by df -h
You can reference here
http://www.whatsabyte.com/ and here
http://en.wikipedia.org/wiki/Hard_disk_drive#Units
Yeah it sucks but those are the facts. The more storage you have the more overall space you will loose to discrepancies like these.
http://www.whatsabyte.com/ and here
http://en.wikipedia.org/wiki/Hard_disk_drive#Units
Yeah it sucks but those are the facts. The more storage you have the more overall space you will loose to discrepancies like these.
Also you should reset all the tuning you did except for compression. Virtually all those things you changed are shooting yourself in the foot.
Especially reset the recordsize variable to 128, it is not the same as other raid systems, or windows native systems, setting it to 16 will hurt performance unless you are running vdevs of mirrors. If you are going to be using this system for movies and home usage you had best leave the recordsize at the default value.
All that aside, if you could spring for 3 more 3 TB drives and add them to your second vdev that would be ideal.
Ideally best performance is going to be achieved by using vdevs of equal size and performance. By using vdevs of equal size they will fill up equally. By using different sized vdevs you will be restricted to your slowest vdev. meaning for some data you will get the performance of a single drive in your current setup.
You could remedy this and increase performance in your current setup by eliminating your raidz1-1 of 2 - 3TB drives and using a vdev of mirrors instead.
You add a mirror set by using the following command:
zpool add tank mirror sdc sdf
Especially reset the recordsize variable to 128, it is not the same as other raid systems, or windows native systems, setting it to 16 will hurt performance unless you are running vdevs of mirrors. If you are going to be using this system for movies and home usage you had best leave the recordsize at the default value.
All that aside, if you could spring for 3 more 3 TB drives and add them to your second vdev that would be ideal.
Ideally best performance is going to be achieved by using vdevs of equal size and performance. By using vdevs of equal size they will fill up equally. By using different sized vdevs you will be restricted to your slowest vdev. meaning for some data you will get the performance of a single drive in your current setup.
You could remedy this and increase performance in your current setup by eliminating your raidz1-1 of 2 - 3TB drives and using a vdev of mirrors instead.
You add a mirror set by using the following command:
zpool add tank mirror sdc sdf
omniscence
[H]ard|Gawd
- Joined
- Jun 27, 2010
- Messages
- 1,310
Your original pool consisted of one raidz vdev and two single disk vdevs. If one of those 3 TB disks fails you are facing a complete data loss. Your last configuration seems about right, although I'm not sure what the disadvantage of a 2 disk raidz (I didn't even know that this is possible) is compared to a simple mirror. Remember, you cannot add disks to a RAIDZ, just add a new vdev to the pool.
To get the capacity of the filesystem use 'zfs list', not 'zpool list'. My interpretation is, that the zpool command outputs the total capacity of the disks, without accounting for parity. The 'zpool iostat' command does the same.
While it may not be significant with such a small number of disks, you should stick to 2^n data disks plus parity if you care about space efficiency.
To get the capacity of the filesystem use 'zfs list', not 'zpool list'. My interpretation is, that the zpool command outputs the total capacity of the disks, without accounting for parity. The 'zpool iostat' command does the same.
Code:
root@quantum:~# zpool list storage
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
storage 30T 17.6T 12.4T 58% 1.00x ONLINE -
root@quantum:~# zfs list storage
NAME USED AVAIL REFER MOUNTPOINT
storage 11.7T 7.98T 725K noneWhile it may not be significant with such a small number of disks, you should stick to 2^n data disks plus parity if you care about space efficiency.
Last edited:
thanks for the help guys im just gonna get rid of the 3tb drives.... now heres the thing i already destroyed the pool and made it with just the 4 4tb drives and started transferring all my data back but i can get a hold of 3 more 4tb drives...should i just add a vdev or should i destroy the pool and make it with all of them at once? from a performance perspective....
More vdevs = more performance
2 raidz vdevs is vastly greater performance than a single large vdev. Though it comes at a data safety tradeoff.
From a safety perspective you would be much safer to use an 8 - 4TB drive raidz2 pool, it will give you a much more substantial layer of protection in case you need to rebuild.
Ideally if you wanted a *high* performance system you would do 2 raidz1 vdevs of 4 drives each.
Highest performing systems are pools comprised entirely of mirrors.
Most people would say, if you have alternate backups for your data then do raidz vdevs for performance with a mild amount of safety. But if you are trusting this to be your only copy of large amounts of data then do a raidz2
2 raidz vdevs is vastly greater performance than a single large vdev. Though it comes at a data safety tradeoff.
From a safety perspective you would be much safer to use an 8 - 4TB drive raidz2 pool, it will give you a much more substantial layer of protection in case you need to rebuild.
Ideally if you wanted a *high* performance system you would do 2 raidz1 vdevs of 4 drives each.
Highest performing systems are pools comprised entirely of mirrors.
Most people would say, if you have alternate backups for your data then do raidz vdevs for performance with a mild amount of safety. But if you are trusting this to be your only copy of large amounts of data then do a raidz2
well i can only fit 7 drives on this mobo so its either 7 - 4tb drives in raidz1
or 4 - 4tb raidz1 and 3 - 4tb raidz1 in another vdev
raidz1 is enough protection for me just looking to maximize my space/performance for my home media sever
what would you do?
edit* i guess another option would be to buy a pcie sata card and make a 4 drive vdev when in the future....
or 4 - 4tb raidz1 and 3 - 4tb raidz1 in another vdev
raidz1 is enough protection for me just looking to maximize my space/performance for my home media sever
what would you do?
edit* i guess another option would be to buy a pcie sata card and make a 4 drive vdev when in the future....
Remember that for ideal performance you want your data disks (N) to be in powers of two, so N+1 for raidz, N+2 for raidz-2 and N+3 for raidz-3
Consider as well a 6x4TB raidz-2, that would equate to four data disks and two parity with a raw total of 16TB storage before formatting, overhead, and marketing losses. Add a seventh 4TB disk to that as a hostpare and you have your seven disk capacity, nice and neat, and requires no intervention from you if a disk should fail.
Consider as well a 6x4TB raidz-2, that would equate to four data disks and two parity with a raw total of 16TB storage before formatting, overhead, and marketing losses. Add a seventh 4TB disk to that as a hostpare and you have your seven disk capacity, nice and neat, and requires no intervention from you if a disk should fail.
brutalizer
[H]ard|Gawd
- Joined
- Oct 23, 2010
- Messages
- 1,603
Ideal performance is not important for a home server. I know lot of people (me included) that use non ideal configurations without any problems.
I would suggest 7 disk raidz2.