• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS Pool & Drives

D

dsumike

Weaksauce
Joined
Apr 15, 2010
Messages
106
I have an existing server that I want to repurpose as a test system for zfs. I have a 6Gb SAS Enclosure for storing up to 24 SAS drives, and I picked up an LSI 9200-8e to connect it to a spare Dell PE2950.

My question is, I have about 12-15 300gb 10k 3Gb SAS drives, and another 12-15 300gb 10k 6Gb SAS2 drives. They're all 300gb 10k drives, and I would prefer to try making one large storage pool for my test using a full 24x 300gb drives if possible as I will be testing this for a higher io esxi datastore and would like as high of spindle count as possible, but I am concerned about the mix of 3Gb/6Gb.

Has anyone done this before? Is there any harm in mixing the drive speeds like this in a single pool? Are there any potential issues or will it work fine but at the lowest common speed (10k 3Gbs)?
 
If the enclosure supports SAS multiplexing then they will each run at their native speeds. If it doesn't they will just run at 3gbps. Shouldn't be an issue.
 
Shouldn't be a problem

If it were me I would plonk the 300 6Gb drives in 1st 12 slots, then the 300 3Gb drives in last 12 slots
Make a Zpool of mirrored vdevs

So you end up with (similar to RAID 10 across all the drives)
Later when funds allow slowly replace all the 3Gb drives with 6Gb drives

You will end up with a decently fast 3.6TB pool (ie the speed approximatley of 12 drives striped)

No harm in mixing drives speeds, it will just all run at 3Gb speeds...
 
Stanza33 said:
Shouldn't be a problem

If it were me I would plonk the 300 6Gb drives in 1st 12 slots, then the 300 3Gb drives in last 12 slots
Make a Zpool of mirrored vdevs

So you end up with (similar to RAID 10 across all the drives)
Later when funds allow slowly replace all the 3Gb drives with 6Gb drives

You will end up with a decently fast 3.6TB pool (ie the speed approximatley of 12 drives striped)

No harm in mixing drives speeds, it will just all run at 3Gb speeds...
Click to expand...

As in try to pair a 3Gb drive with a 6Gb drive for each mirrored vdev?

That would certainly do a good job of forcing all the vdevs to be at an equal 3Gb speed, and hopefully with all 10k SAS drives provide decent speeds.

It will be an interesting test to see how many concurrent VMs I can run off of what ends up being roughly 12 spindles (vdevs).
 
dsumike,
How long ago did you buy those 2.5" hard drives?
 
dsumike said:
As in try to pair a 3Gb drive with a 6Gb drive for each mirrored vdev?

That would certainly do a good job of forcing all the vdevs to be at an equal 3Gb speed, and hopefully with all 10k SAS drives provide decent speeds.

It will be an interesting test to see how many concurrent VMs I can run off of what ends up being roughly 12 spindles (vdevs).
Click to expand...

Actually no sorry

Pair them up speed wise,

Bay 1 << 6G = Mirror 1
Bay 2 << 6G = Mirror 1
Bay 3 << 6G = Mirror 2
Bay 4 << 6G = Mirror 2
Bay 5 << 6G = Mirror 3
Bay 6 << 6G = Mirror 3
Bay 7 << 6G = Mirror 4
Bay 8 << 6G = Mirror 4
Bay 9 << 6G = Mirror 5
Bay 10 << 6G = Mirror 5
Bay 11 << 6G = Mirror 6
Bay 12 << 6G = Mirror 6
Bay 13 << 3G = Mirror 7
Bay 14 << 3G = Mirror 7
Bay 15 << 3G = Mirror 8
etc thru to bay 24

Then I guess you know which are your 6gb and Which are your 3Gb drives to swap out later down the track;)

Naturally if you are worried about data safety....
You would throw another Jbod into the mix along with a second controller
Then spread the drives over the 2 x chassis in the Mirror config

so you would have

Controller 1 > Bay 1 Chassis 1 << 6G = Mirror 1
Controller 2 > Bay 1 Chassis 2 << 6G = Mirror 1
Controller 1 > Bay 2 Chassis 1<< 6G = Mirror 2
Controller 2 > Bay 2 Chassis 2<< 6G = Mirror 2
etc

So if a controller or a chassis dies, you are still up and running with the other half of the mirrors...
 
Stanza33 said:
Naturally if you are worried about data safety....
You would throw another Jbod into the mix along with a second controller
Then spread the drives over the 2 x chassis in the Mirror config

So if a controller or a chassis dies, you are still up and running with the other half of the mirrors...
Click to expand...

Absolutely. That's the long-term goal, but this is just a short-term test setup. We will probably run some training VM's on it to see how well it performs.
 
No problems yet.

My original intentions were to use them as a SAN with DataCore or Starwind.

Only recently have I just started to "inquire" about ZFS

The current 2.5" disk pricing is pretty ridiculous!
 
dsumike said:
Nice. Any problems? It seemed to be one of the better ones I found, so I'm hoping for all good news. :)
Click to expand...

Off all the ones I looked at, the SM had the best overall build quality.

LSI 620J I also liked, they were a little more $$
 
dsumike said:
It will be an interesting test to see how many concurrent VMs I can run off of what ends up being roughly 12 spindles (vdevs).
Click to expand...

Should give you approx 12 x 80iops

quote
"If you want to estimate disk capacity, a good guideline is to expect about 100 IOPS/sec for each spindle (this assumes 10,000 rpm). Depending on your disk configuration, you may need to make adjustments.

Code: 
Estimated RAID throughput per spindle
Raid configuration << Estimated IOPS/second per spindle  
Raid0 << 100
Raid1 << 80
Raid0+1 << 80
Raid5 << 57

In a RAID0 configuration, each read and each write generates one I/O operation. In the RAID1 and RAID0+1 configurations, each read generates one I/O operation, but each write requires two I/O operations (a write to each mirrored disk). "
/quote

so you can work out if the iops will be enough
 
its a pity there's no zfs for windows at least yet

femi said:
No problems yet.

My original intentions were to use them as a SAN with DataCore or Starwind.

Only recently have I just started to "inquire" about ZFS

The current 2.5" disk pricing is pretty ridiculous!
Click to expand...
 
do you have any expectations for double parity raid-z2 config?

Stanza33 said:
...

Code: 
Estimated RAID throughput per spindle
Raid configuration << Estimated IOPS/second per spindle  
Raid0 << 100
Raid1 << 80
Raid0+1 << 80
Raid5 << 57

In a RAID0 configuration, each read and each write generates one I/O operation. In the RAID1 and RAID0+1 configurations, each read generates one I/O operation, but each write requires two I/O operations (a write to each mirrored disk). "
/quote

so you can work out if the iops will be enough
Click to expand...
 
Olga-SAN said:
do you have any expectations for double parity raid-z2 config?
Click to expand...

Depend on the number of drives in the vdev

24 drives doesn't leave much room for iops

Raidz2 loses 2 drives in each vdev
So minimum is what 4 drives.... Which leaves you 2 drives worth of space....pointless might as well go mirrors

Raidz2 with 5 drives? You will get 1 drives more space per vdev ... But out of 24 drives you can get only 4 vdevs....(20 drives).... And that will only get you 4 drives worth of iops

Raidz2 with 6 drives uses all your 24 drives nicely.... But you are still limited to 4 x drives of iops and still losing 8 drives to parity

Mirrors are your best for iops

Read iops can be increased with a cache drive or two.... Ie L2ARC and lots of ram

Disclaimer...these figures I quote are rounded....

.

.
 
Read iops can also be increased by going with n-way mirrors instead of 2-way. Also increases redundancy (assuming of course you can sacrifice drives :) )
 
There is a bit of misinformation in this thread. I'll try to clear some things up.

12 mirrored ZFS vdevs does NOT equal the read IOPS of 12 disks, but the read IOPS of 24 disks. ZFS does in fact distribute reads to both drives in the mirror. This is the case for 3-way mirrors as well, you'd get the read performance of 3 disks.

2.5" 10K rpm drives are not limited in speed by a 3 Gbps interface. If there are ANY differences in speed between your two drive models, this is probably more likely attributed to the platter density rather than the speed of the interface.

If this is for esxi storage, go for mirrors. If you absolutely need more space, go for raidz2 vdevs and enough ARC+L2ARC to fit all or as much as economically possible of your work set into it.
 
spazoid said:
There is a bit of misinformation in this thread. I'll try to clear some things up.

12 mirrored ZFS vdevs does NOT equal the read IOPS of 12 disks, but the read IOPS of 24 disks. ZFS does in fact distribute reads to both drives in the mirror. This is the case for 3-way mirrors as well, you'd get the read performance of 3 disks.

2.5" 10K rpm drives are not limited in speed by a 3 Gbps interface. If there are ANY differences in speed between your two drive models, this is probably more likely attributed to the platter density rather than the speed of the interface.

If this is for esxi storage, go for mirrors. If you absolutely need more space, go for raidz2 vdevs and enough ARC+L2ARC to fit all or as much as economically possible of your work set into it.
Click to expand...

Wow, that is really cool. I was always blown away why most [all?] traditional RAID doesn't do this. I really need to play with ZFS some more.
 
spazoid said:
...enough ARC+L2ARC to fit all or as much as economically possible of your work set into it.
Click to expand...

How does one go about discovering how large their working set is?

Also, am I correct in understanding that L2ARC can span multiple SSDs? So I can start with say, 100gb L2ARC, and if it's not enough, add a second 100gb SSD and it will just add it to the cache pool?

Is there any documentation about how that works? Does it automatically stripe them for additional performance, or does it do some other magic behind the scenes?
 
dsumike said:
am I correct in understanding that L2ARC can span multiple SSDs? So I can start with say, 100gb L2ARC, and if it's not enough, add a second 100gb SSD and it will just add it to the cache pool?
Click to expand...

Correct, the stripe part im not sure about tho.
 
I don't think they are stripes, but separate devices that are round-robin'ed. At least multiple ZILs work that way. One way to tell: add 2 L2ARC devices and see if it will let you remove one of them - if they were really striped, you would not be allowed.
 
No, they aren't stripped, but then you really wouldn't want to stripe them anyways.

If you stripe them, reading a block, would hit all the ssd's with iops

Just doing round-robin per block, things get randomized pretty good, and if a ssd dies, only the info on that ssd goes missing (but can still be recovered from the real disks)

My two ssd, normally sit at around 700 read iops, and about 20 write iops per second.

Still, ram is much much faster, and I'm getting around 7000 iops from my arc.
 
danswartz said:
...separate devices that are round-robin'ed...
Click to expand...

That makes sense, and is fairly optimal. I just wanted to make sure that adding additional SSD's makes the work fairly distributed, so it's not heavy on one ssd over another added later.

danswartz said:
At least multiple ZILs work that way...
Click to expand...

How does ZFS handle losing a dedicated SLOG drive for the ZIL, especially if there's more than one? If adding multiple is it best to always add them in mirrored pairs?
 
AFAIK, it only matters if you crash and ZFS has to replay the transactions in the log. Even then, native ZFS data will never be corrupt, just incomplete (e.g. last few writes missing.) OTOH, if you have a different filesystem mounted to ZFS via NFS or somesuch, that FS might experience corruption. The best practice is to add in pairs, but not everyone does, since AFAIK, you need to unrelated failures to experience an issue.
 
dsumike said:
How does one go about discovering how large their working set is?

Also, am I correct in understanding that L2ARC can span multiple SSDs? So I can start with say, 100gb L2ARC, and if it's not enough, add a second 100gb SSD and it will just add it to the cache pool?

Is there any documentation about how that works? Does it automatically stripe them for additional performance, or does it do some other magic behind the scenes?
Click to expand...

Determining your work set size up front can be quite difficult, but there are some scripts out there that'll give you very nice (l2)arc hite rate statistics.

arc_summary.pl is one such script: http://dtrace.org/blogs/brendan/2012/01/09/activity-of-the-zfs-arc/

Keep adding ARC until your wallet or system cannot handle more, and then add L2ARC until your cache hit rate is acceptable.
Obviously a 90+% hit rate would be very desirable.

As others have stated, you can add L2ARC devices one at a time. They are not striped, and more devices are better than larger devices for a number of reasons:
Speed. The more devices, the more bandwidth and IOPS.
Less impact due to failure. If you have 1 large L2ARC device and it dies, you lose ALL of your L2ARC which can cripple your storage system. if you have 10 devices and one dies, you only lose 1/10th of the speed and capacity of your total L2ARC, the performance of which should be relatively easily absorbed by the spinning disks.
Price. You can buy cheaper disks. First of all, your system uptime is not dependant on the L2ARC devices. Second, not a lot of data is written to the L2ARC once its "warm". Third, you can get whatever size is cheapest/bit.
 
spazoid,
But write IOPS would be like 12 disks, right? Read IOPS is like 24 disks, but write IOPS is like 12 disks?

Any information on this? Links?
 
Write IOPS is 12 disks yes. Lots of information on this available for your reading pleasure on the World Wide Web :)
 
And similarly, if you did two raidz2 vdevs with 12 disks in each, then you would have write IOPS as two disks (two vdevs), but read IOPS as 24 disks (the number of individual disks)?
 
Stanza33 said:
Should give you approx 12 x 80iops

quote
"If you want to estimate disk capacity, a good guideline is to expect about 100 IOPS/sec for each spindle (this assumes 10,000 rpm). Depending on your disk configuration, you may need to make adjustments.
Click to expand...

So about ~960 IOPS for a 24-disk configuration (12x vdevs) of 10k SAS drives. It's not bad, but not stellar. I have a 16-disk EqualLogic unit in with 1TB 7.2k disks in a RAID10 configuration also (so 7 disks after hotspares) that's able to pump out over 2,000 IOPS sustained. Granted, the few GB's of local cache impacts that number, but still... zfs will have the ARC and L2ARC if I add one down the road... so hopefully I'll see better numbers the ~1k IOPS in actual usage.

Do you have estimated IOPS numbers for upgrading to 15k SAS drives or downgrading to 7.2k NL-SAS ... is there a big push towards either?

I have a stock-pile of 500gb NL-SAS's also, if it gets me more space with minimal hit, they may be more worth it. However, likewise, if there's a big enough gain in going to 15k SAS drives, I could be persuaded to actually purchase 24 dives versus using up our leftovers.
 
hmm?

10krpm sas drives give around 145iops (seagate 10k 2.5=149, seagate 10k 3.5=147)

That gives 1740iops writes, and double that for reads.

Going nearline 7.2k rpm, would put you at 980iops for writes

Going 15k disks would push it up to 2220iops, writes
 
dsumike said:
So about ~960 IOPS for a 24-disk configuration (12x vdevs) of 10k SAS drives. It's not bad, but not stellar.
Click to expand...
You use 7200rpm disks. That gives 7200rpm / 60 second = 120 IO per second = 120 IOPS. If you have 24 such disks, you get 24 x 120 IOPS = 2880 read IOPS. If you write, you get 12 x 120 IOPS = 1440 write IOPS.

If you use 10k disks, you get 4000 read IOPS and 2000 write IOPS.

Now we are talking about ZFS. And if you add SSD as cache, then you can boost into 100.000 of IOPS.
https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs_iops
 
Stanza33 said:
Naturally if you are worried about data safety....
You would throw another Jbod into the mix along with a second controller
Then spread the drives over the 2 x chassis in the Mirror config
Click to expand...

I see lots of references to growing the pool, which is awesome. But I haven't found a clear answer to what happens behind the scenes when I add additional vdevs. If I had a second JBOD later, and grow the pool by adding 12 more mirrored vdevs (24 drives). Does the existing data automatically rebalance over all of the devices, new and old? Or am I better off destroying the pool and recreating?
 
No, it doesn't, just new data. However, since the FS is copy on write, as you write files over time it will rebalance.
 
brutalizer said:
You use 7200rpm disks. That gives 7200rpm / 60 second = 120 IO per second = 120 IOPS. If you have 24 such disks, you get 24 x 120 IOPS = 2880 read IOPS. If you write, you get 12 x 120 IOPS = 1440 write IOPS.

If you use 10k disks, you get 4000 read IOPS and 2000 write IOPS.

Now we are talking about ZFS. And if you add SSD as cache, then you can boost into 100.000 of IOPS.
https://blogs.oracle.com/brendan/entry/a_quarter_million_nfs_iops
Click to expand...

Those are really high, that only accounts for the rotational latency of the disk, and not any head seeking.

The correct way to calc is 1000 / (rotational latency + head latency)

The numbers to use for those, are posted in the drives datasheets, easy to locate from wd or seagate.
 
You must log in or register to reply here.
Back
Top