I'm in the process of building a 40TB ZFS storage server. My goal is to have as much space in a single filesystem as reasonably possible. I realize ZFS can do this easily, but it seems like most storage admins won't create filesystems that large. I understand it's primarily because of the rebuild time. If I used a zraid3 + hot spare, would that be bad? This server is primarily going to be a CIFS server in an AD environment. It's going to have 13 3TB drives. I realize the resilvering time will take longer but I'm willing to accept that as long as it doesn't put much load on the server. It'll be a powerful server, lots of RAM, & 300GB SSD L2ARC cache.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
New ZFS storage server
- Thread starter Churnd
- Start date
- Joined
- May 7, 2007
- Messages
- 13,442
As long as you have enough RAM and CPU power to handle all of the users over CIFS, then you should be fine.
The reason many admins wouldn't use a single storage pool that size is because the setup would most likely be a classic hardware RAID array with RAID 5 or 6, which would be very a higher risk in a many-user environment with so many disks. They would rather setup many smaller RAID arrays, at least in their thinking.
Since you are using ZFS though, then it totally changes everything.
Which 3TB drives are they?
If this is not out of your pocket, and you have a budget, seriously consider getting some nearline-class drives such as WD RE4 or Seagate Constellation drives.
They are expensive, but they are far more robust than any desktop-class drive and will be able to withstand a high-access, 24/7 environment, as well as higher heat and vibrations.
The reason many admins wouldn't use a single storage pool that size is because the setup would most likely be a classic hardware RAID array with RAID 5 or 6, which would be very a higher risk in a many-user environment with so many disks. They would rather setup many smaller RAID arrays, at least in their thinking.
Since you are using ZFS though, then it totally changes everything.
Which 3TB drives are they?
If this is not out of your pocket, and you have a budget, seriously consider getting some nearline-class drives such as WD RE4 or Seagate Constellation drives.
They are expensive, but they are far more robust than any desktop-class drive and will be able to withstand a high-access, 24/7 environment, as well as higher heat and vibrations.
I'm actually buying the system from Aberdeen. I need to have support agreements. It's for my workplace.
Currently configured with these drives: Hitachi Ultrastar 3TB 7200rpm SATA 6G 64MB
... Make sure you have spares (hot or cold). Even if Hitachi does advanced replacement on the Ultrastar line you don't want to be caught with your pants down.
I'd go raidz3 if availability is the most important factor. How many disks are you doing per vdev? I'm thinking you're thinking 13 wide? Or 12 disk raidz3 with one hotspare? Could go 11 raidz3, 1 hotspare and a cache and/or slog (assuming you don't have more physical space).
By the way, someone correct me if I'm wrong, but the rebuild time is directly related to the disk (size+rotational speed) not how big the zpool is. Eg, you can only theoretically rebuild a disk as fast as a single disk can write.
I think the rule of thumb is 7200RPM drive, ~12 hours per TB?
Last edited:
Ordering 3 cold spares. Will have one hot spare.
Was planning on doing one big raidz3 pool for all 13 drives.
Hmms, doesn't sound too bad. Lower redundancy potential than I personally would care for.
Someone is probably going to mention that the usable disks isn't divisible (sorry factor) into 128 or something. Meh, my performance doesn't seem to suffer from that.
PS I edited my first post
I ninja'd added.By the way, on my server running an E5520, two vCPUs, and 6GB carved (ESXI) to my OpenIndiana/ZFS VM... Two 2TB disks resilvered in roughly 24 hours. Tthough I had no other IO at the time, but that rebuild time is what I experience with my Home Server on other occasions, and it has moderate IO all the time. Nothing like an enterprise level server would get, but still.
CPU % utilization for the zpool was roughly around 20-29% the entire time.
Fyi, Zpool is 2 vdevs comprised of 8x2TB disks
CPU % utilization for the zpool was roughly around 20-29% the entire time.
Fyi, Zpool is 2 vdevs comprised of 8x2TB disks
- Joined
- May 7, 2007
- Messages
- 13,442
Maybe spare controller too (read my thread haha), multi-port/redundant backplanes, and redundant Power Supplies... Assuming, again, enterprise level (Tier 2, Tier 1.5) requirements.Nice, yeah, just make sure to utilize those hot and cold spares.
By the way, are you really going to create one file system (one zpool is fine)? How are you serving out the storage to the clients (Comstar/NFS/CIFS/Netatalk?)?
- Joined
- May 7, 2007
- Messages
- 13,442
The OP stated that he was using CIFS.
Okay, that's the 3rd thing I've missed in the original post... Sorry my reading comprehension is low today.
... Sorry
- Joined
- May 7, 2007
- Messages
- 13,442
lol, it's cool.
Hopefully he will share some results with us once he gets it up and running.
Benchmarks ftw!
Hopefully he will share some results with us once he gets it up and running.
Benchmarks ftw!
Nice, yeah, just make sure to utilize those hot and cold spares.
Utilize a cold spare how? Other than replace a failed drive when needed?
I think he's just joking.
Or, he means don't let the zpool run in degraded mode just because one disk failed, replace right away
... I admittedly have done that for a couple days (early days)... but its a home server.I'm interested in how it works for him in an enterprise environment. I think CIFS still needs some work, but it should work for the average use-case
lol, it's cool.
Hopefully he will share some results with us once he gets it up and running.
Benchmarks ftw!
I will. It will be NexentaStor OS, so not sure which benchmarks I could run without adding too much unsupported software. My main goal is to easily saturate a 2 gigabit link for large file copies (4GB) & keep the sustained write speeds optimal. I'm thinking the 300GB SSD L2ARC will make that possible.
I think he's just joking.
Or, he means don't let the zpool run in degraded mode just because one disk failed, replace right away
I'm interested in how it works for him in an enterprise environment. I think CIFS still needs some work, but it should work for the average use-case
I have an ESXi/ZFS all-in-one in my office I have been testing for the past few months & I'm happy with where it is now, though I have no idea how it will handle multiple users. Probably get anywhere from 10-25 users at once. We're a small lab so I figured it'll be fine.
You will get better performance with more smaller vdevs than one big one. Your IOPS per vdev = IOPS of one disk for Raid-ZX type vdevs. (Ignoring SSD acceleration from ZIL/L2ARC).
You also say 40TB, but then 13 3TB disks. I'm guessing you mean ~40TB raw, so 30TB of usable space in your proposed configuration. Now consider you also are picking up 3 cold spares and 1 hot spare. So you have 20 3TB disks total.
What if you went with 3 Raid-Z2 vdevs of 6 disks each. This has you utilizing 18 disks hot instead of 13. Your rebuild time will be decreased as your vdev is smaller (vdev is what has to rebuild, not the pool). You will have better performance as your now have the effective IOPS of 3 drives. You will also have 12 * 3 = 36GB available. Your redundancy would be better in some cases, worse in others (you can't loose 3 disks from the same vdev, but you can loose 6 disks total depending on where they come from). I suspect, on average, you would have better reliability.
If you want to stick with raid-z3 you need one more disk, 21, and you can do 3 7disk raid-z3 setups.
As for the "make sure you make use of those spares" - think about it this way.
*What is the point of having the spare? (especially hot spares) - redundancy. So isn't it better to actually have the spares running as part of your array already? With the extra parity drives included the slivering, etc. is already done. Since you are at raid-z3 we can't really go to raid-z4, but if we break down the one large vdev to smaller vdevs you can get in additional parity disks, and so make "realtime" use of those spares.
Basically anytime you would have a hot spare especially, but even a cold spare, you would be better off having it spooled up and running in your array. Only when you can't (controller limit or hard drive location limit) is it worth not keeping in the array.
You will probably also want to consider a ZIL to keep your write IOPS up, especially if you have multiple people accessing the system simultaneously (Random sync write IOPS will be murder on performance with one big VDEV and no zil)
You also say 40TB, but then 13 3TB disks. I'm guessing you mean ~40TB raw, so 30TB of usable space in your proposed configuration. Now consider you also are picking up 3 cold spares and 1 hot spare. So you have 20 3TB disks total.
What if you went with 3 Raid-Z2 vdevs of 6 disks each. This has you utilizing 18 disks hot instead of 13. Your rebuild time will be decreased as your vdev is smaller (vdev is what has to rebuild, not the pool). You will have better performance as your now have the effective IOPS of 3 drives. You will also have 12 * 3 = 36GB available. Your redundancy would be better in some cases, worse in others (you can't loose 3 disks from the same vdev, but you can loose 6 disks total depending on where they come from). I suspect, on average, you would have better reliability.
If you want to stick with raid-z3 you need one more disk, 21, and you can do 3 7disk raid-z3 setups.
As for the "make sure you make use of those spares" - think about it this way.
*What is the point of having the spare? (especially hot spares) - redundancy. So isn't it better to actually have the spares running as part of your array already? With the extra parity drives included the slivering, etc. is already done. Since you are at raid-z3 we can't really go to raid-z4, but if we break down the one large vdev to smaller vdevs you can get in additional parity disks, and so make "realtime" use of those spares.
Basically anytime you would have a hot spare especially, but even a cold spare, you would be better off having it spooled up and running in your array. Only when you can't (controller limit or hard drive location limit) is it worth not keeping in the array.
You will probably also want to consider a ZIL to keep your write IOPS up, especially if you have multiple people accessing the system simultaneously (Random sync write IOPS will be murder on performance with one big VDEV and no zil)
Well the point is to allocate as much usable space as possible, while having good redundancy & being able to saturate a gigabit network link. I will also be using snapshots, so part of the space will be used for that, which means even less usable space.
Got it. However, I sorta feel like having a "cold spare" has it's benefits. If a drive isn't spinning, nothing can go wrong with it. I do realize just because it's "cold", doesn't mean it won't fail the same way as if it were "hot", eventually.
Ah, I thought L2ARC handled all the reading/writing caching. I'm getting dual Intel 300GB SSD's for mirrored L2ARC, but seems like I need to factor in a ZIL cache too. Is there any way to use the same SSD pool for both caches?
L2ARC isn't mirrored (no need)
What about ZIL? It's coming with 2 300GB SSD's as is, so I could actually use one for ZIL & one for L2ARC. But what happens if one fails?
ZIL should always be mirrored.
If the read cache ssd fails it reads the data directly from the actual pool.
If the ZIL fails I believe you're a bit screwed.
From the ZFS Best Practices
Mirroring the log device is recommended. Prior to pool version 19, if you have an unmirrored log device that fails, your whole pool might be lost or you might lose several seconds of unplayed writes, depending on the failure scenario.
In current releases, if an unmirrored log device fails during operation, the system reverts to the default behavior, using blocks from the main storage pool for the ZIL, just as if the log device had been gracefully removed via the "zpool remove" command.
If the read cache ssd fails it reads the data directly from the actual pool.
If the ZIL fails I believe you're a bit screwed.
From the ZFS Best Practices
Mirroring the log device is recommended. Prior to pool version 19, if you have an unmirrored log device that fails, your whole pool might be lost or you might lose several seconds of unplayed writes, depending on the failure scenario.
In current releases, if an unmirrored log device fails during operation, the system reverts to the default behavior, using blocks from the main storage pool for the ZIL, just as if the log device had been gracefully removed via the "zpool remove" command.
Yeah, as long as your Zpool version is > 19 it's not a huge deal if the ZIL fails. Basically, worst case, , your zil fails (physically), then your power goes out and you have no UPS you can loose the data that was on your zil and still uncommitted).
While that's a bad thing it's also quite a confluence of events.
300GB is significant overkill for ZIL. You want drives with high sequential & random write speed & iops (starts out sequential, but the more people you have writing simultaneously the more random the workload becomes). How much space you need depends on the "data in" rate, but really even if you were at 10GBit 8-10GByte would still be overkill. That said you could still use the 300GB as the ZIL and with the firmware handling wear levelling, well you would have plenty of space to wear level over. You could add both devices as L2ARC and end up with 600GB of L2ARC - this may be overkill though (and realize that L2ARC occupies some primary system ram - how much depends on the recordize, but 1GB per 200GB of L2ARC would be a reasonable guesstimate (though it's probably slightly less than that))
Re: maxing out Gigabit.
I guess it depends on your workload - maxing out a single gigabit stream will be trivial with your setup - and really doesn't matter how you configure everything. Maxing out multiple gigabit streams is where some of this stuff comes into play - and I guess I don't know how this will be networked? LACP? 10GbE? And this is sequential workload - if you have lots of random IOPS then how you configure the VDEV's can have an impact on what type of speed you get out.
While that's a bad thing it's also quite a confluence of events.
300GB is significant overkill for ZIL. You want drives with high sequential & random write speed & iops (starts out sequential, but the more people you have writing simultaneously the more random the workload becomes). How much space you need depends on the "data in" rate, but really even if you were at 10GBit 8-10GByte would still be overkill. That said you could still use the 300GB as the ZIL and with the firmware handling wear levelling, well you would have plenty of space to wear level over
. You could add both devices as L2ARC and end up with 600GB of L2ARC - this may be overkill though (and realize that L2ARC occupies some primary system ram - how much depends on the recordize, but 1GB per 200GB of L2ARC would be a reasonable guesstimate (though it's probably slightly less than that))Re: maxing out Gigabit.
I guess it depends on your workload - maxing out a single gigabit stream will be trivial with your setup - and really doesn't matter how you configure everything. Maxing out multiple gigabit streams is where some of this stuff comes into play - and I guess I don't know how this will be networked? LACP? 10GbE? And this is sequential workload - if you have lots of random IOPS then how you configure the VDEV's can have an impact on what type of speed you get out.
I'm going to be on at least v28 zpool & the server will be in a datacenter that is well guarded against power failure (3-level UPS redundancy). The datacenter itself is way overkill for us (if there is such a thing), but we get it for free . Basically, I'm not worried about power loss at all there & the server itself will have redundant power supplies. However, I do want to be reasonably protected & don't want a single SSD failure to be catastrophic. Maybe I would be better off going with a smaller SSD (128GB) & getting 3... mirroring the ZIL & single for L2ARC?
. Basically, I'm not worried about power loss at all there & the server itself will have redundant power supplies. However, I do want to be reasonably protected & don't want a single SSD failure to be catastrophic. Maybe I would be better off going with a smaller SSD (128GB) & getting 3... mirroring the ZIL & single for L2ARC?Not a bad idea. I mean the ZIL doesn't have to be large at all. You should be able to do easily 32GB or 64GB (for any decent SSD) ZIL if we're talking about a Gigabit Network link. IIRC, ZIL has no requirements when weighed against the size of the pool, just how much write IO (until it can be committed) you can push. Since the only thing laying down IO is doing it via a Gigabit link..... the size requirement isn't very large. I don't know, I haven't done a ZIL due to the lack of need + limited physical space.I'm going to be on at least v28 zpool & the server will be in a datacenter that is well guarded against power failure (3-level UPS redundancy). The datacenter itself is way overkill for us (if there is such a thing), but we get it for free
Mirroring the l2arc would be cool, especially for failures/reboots so it doesn't have to warm (or... you know it would stay up), though that's more of a concern for an heavy IO use-case (like DB)... and even that's debatable.
ugh don't waste a 300GB ssd on zil when only a tiny fraction of that would get used. use these large drives for l2arc cache and get 2 of the smallest fastest/iops slc based ssd for zil.
ocz enterprise just launched some new deneva drives that can be had down to 30gb which is still overkill but might be worth it. idk about the availability still waiting on them to schedule a call and tell me more about their new enterprise offerings.
I'm going to be on at least v28 zpool & the server will be in a datacenter that is well guarded against power failure (3-level UPS redundancy). The datacenter itself is way overkill for us (if there is such a thing), but we get it for free
just an aside, you never ever stop worrying about power loss. idk where you live but recently almost all of san diego county lost power as well as parts of arizona because some major cables broke.
my DC partner has 7 data centers in san diego, i think 1 or 2 of them lost power to the floor briefly. the DC i'm in didn't thankfully but shit does hit the fan and DCs do lose power to the floor even with all the power redundancy it still happens more than you might think.
sleep well with that in mind
.L2ARC cache:
Basically, the L2ARC will hold all most frequently used blocks from the pool for read cache purposes.
No need to have enterprise *reliable* storage (by that, I mean being concerned about SSD's wear leveling issues -> cheap but fast SSD based on MLC is typically what you're looking for). Basically if it fails, ZFS will get data from the main storage pool, so no big deal.
Sequential reading speed is the most important criteria, it should be > by a large margin, compared to the combined storage pool bandwidth.
Plus, the more you have in terms of storage, the better. You can JBOD multiple drives if you want (no RAIDZ, no mirror, just JBOD à la RAID0) to increase cache size and speed.
Typical example: Intel 320, 300 GB or even 600 GB sizes. Crucial M4 512 GB should be fine as well.
ZIL log/ZLOG:
Purpose is to log synchronised writes on a stable media, if the system is under pressure. They all will be committed to the storage pool at one point however. Note: CIFS does not make use of synchronised writes, if I am correct.
It is more tricky here. You don't need much space, 4 or 8 GB at most. Extra space won't serve any purpose. Writing speed is essential (high write IOPS, mostly random).
Couple of months ago, a crashed ZIL device could make you loss ALL your pool, even if 99.9% of the data was sill consistent and stored on the HDDs (the lost 0.1% one being the recent before-the-crash uncommitted writes that were still stored inside the ZIL log).
Even if Sun/Oracle introduced recovery mode for lost or inconsistent ZIL devices, it is better to:
a/ mirror by using two devices
b/ have power failure protected devices used in ZIL (on-battery RAM drives or SSD with capacitors).
Note: For SSD, there is no TRIM within ZFS, it is important therefore to use SLC chips or drives with very good Garbage Collector algorithms. However, if needed, it is more convenient to regularly secure erase a ZIL log SSD for restoring full performance compared to L2ARC, because the device should be fully operational right away after being detached/secured-erased/reattached. Regarding L2ARC, secure erasing operation will delete all cached files and ZFS will have to build again the cache from scratch.
Typical example: RAM drives are really expensive, but best devices for this use (no wear levering and performance attrition over time, on battery and world class writing IOPS)
At the moment, I guess there is no SSD available that is 100% perfect for ZIL usage, even if the Intel 320 could potentially match. Writes are not so good on small capacities, using bigger ones could be a waste of money however (but good for wear levering, if you can commit a 300 GB for just a ZIL log, go ahead). They are MLC chips based but good Garbage Collector overall.
The Intel 311 could be interesting (small size but consistent performance, correct speed, with SLC), however, it lacks the capacitors from the 320 series. Perhaps future Harvey Creek would include that.
Crucial Marvel are very good drives, but smaller capacities are less write-efficient, and GC not so good. No power-loss preventive capacity additionally.
Sandforce writes performance is consistent across the various sizes, good garbage collection, but I won't use them for ZIL because of reliability concerns. Furthermore, only the non-available enterprise grade drives are power-loss protected.
Regarding partitioning, I won't recommend partitioning devices, in particular sharing common physical device for cache and log purposes. In my opinion, the only acceptable scheme could be hosting the system pool on the L2ARC device, if really needed.
Basically, the L2ARC will hold all most frequently used blocks from the pool for read cache purposes.
No need to have enterprise *reliable* storage (by that, I mean being concerned about SSD's wear leveling issues -> cheap but fast SSD based on MLC is typically what you're looking for). Basically if it fails, ZFS will get data from the main storage pool, so no big deal.
Sequential reading speed is the most important criteria, it should be > by a large margin, compared to the combined storage pool bandwidth.
Plus, the more you have in terms of storage, the better. You can JBOD multiple drives if you want (no RAIDZ, no mirror, just JBOD à la RAID0) to increase cache size and speed.
Typical example: Intel 320, 300 GB or even 600 GB sizes. Crucial M4 512 GB should be fine as well.
ZIL log/ZLOG:
Purpose is to log synchronised writes on a stable media, if the system is under pressure. They all will be committed to the storage pool at one point however. Note: CIFS does not make use of synchronised writes, if I am correct.
It is more tricky here. You don't need much space, 4 or 8 GB at most. Extra space won't serve any purpose. Writing speed is essential (high write IOPS, mostly random).
Couple of months ago, a crashed ZIL device could make you loss ALL your pool, even if 99.9% of the data was sill consistent and stored on the HDDs (the lost 0.1% one being the recent before-the-crash uncommitted writes that were still stored inside the ZIL log).
Even if Sun/Oracle introduced recovery mode for lost or inconsistent ZIL devices, it is better to:
a/ mirror by using two devices
b/ have power failure protected devices used in ZIL (on-battery RAM drives or SSD with capacitors).
Note: For SSD, there is no TRIM within ZFS, it is important therefore to use SLC chips or drives with very good Garbage Collector algorithms. However, if needed, it is more convenient to regularly secure erase a ZIL log SSD for restoring full performance compared to L2ARC, because the device should be fully operational right away after being detached/secured-erased/reattached. Regarding L2ARC, secure erasing operation will delete all cached files and ZFS will have to build again the cache from scratch.
Typical example: RAM drives are really expensive, but best devices for this use (no wear levering and performance attrition over time, on battery and world class writing IOPS)
At the moment, I guess there is no SSD available that is 100% perfect for ZIL usage, even if the Intel 320 could potentially match. Writes are not so good on small capacities, using bigger ones could be a waste of money however (but good for wear levering, if you can commit a 300 GB for just a ZIL log, go ahead). They are MLC chips based but good Garbage Collector overall.
The Intel 311 could be interesting (small size but consistent performance, correct speed, with SLC), however, it lacks the capacitors from the 320 series. Perhaps future Harvey Creek would include that.
Crucial Marvel are very good drives, but smaller capacities are less write-efficient, and GC not so good. No power-loss preventive capacity additionally.
Sandforce writes performance is consistent across the various sizes, good garbage collection, but I won't use them for ZIL because of reliability concerns. Furthermore, only the non-available enterprise grade drives are power-loss protected.
Regarding partitioning, I won't recommend partitioning devices, in particular sharing common physical device for cache and log purposes. In my opinion, the only acceptable scheme could be hosting the system pool on the L2ARC device, if really needed.
Last edited:
wrong
- Joined
- May 7, 2007
- Messages
- 13,442
Care to enlighten us with that answer, noobie?
Seriously though, danswartz is pretty knowledgeable about this subject, why do you say he is wrong?
he is wrong .. sort of.
from nexenta's site
The minimum size of a log device is the same as the minimum size of device in pool, which is 64 Mbytes. The amount of in-play data that might be stored on a log device is relatively small. Log blocks are freed when the log transaction (system call) is committed.
The maximum size of a log device should be approximately 1/2 the size of physical memory because that is the maximum amount of potential in-play data that can be stored. For example, if a system has 16 Gbytes of physical memory, consider a maximum log device size of 8 Gbytes.
For a target throughput of X MB/sec and given that ZFS pushes transaction groups every 5 seconds (and have 2 outstanding), we also expect the ZIL to not grow beyond X MB/sec * 10 sec. So to service 100MB/sec of synchronous writes, 1 GBytes of log device should be sufficient.
the last part here is the most significant IMO. doesn't matter if you have 4 or 40GB for the ZIL if your IO on the network is only gig interfaces then you're never going to see enough inbound writes to fill it. if you have 10gig ok, now you're looking at a lot more potential inbound writing but you're going to be bottlenecked by the ZIL in this case anyway.
also, IMO, its pointless to mirror the ZIL if you're using SSDs. each ZIL is going to wear out at the same rate anyway. run zfs 5/28 and don't worry to much about losing your zil or run an acard or ddrdrive instead.
from nexenta's site
The minimum size of a log device is the same as the minimum size of device in pool, which is 64 Mbytes. The amount of in-play data that might be stored on a log device is relatively small. Log blocks are freed when the log transaction (system call) is committed.
The maximum size of a log device should be approximately 1/2 the size of physical memory because that is the maximum amount of potential in-play data that can be stored. For example, if a system has 16 Gbytes of physical memory, consider a maximum log device size of 8 Gbytes.
For a target throughput of X MB/sec and given that ZFS pushes transaction groups every 5 seconds (and have 2 outstanding), we also expect the ZIL to not grow beyond X MB/sec * 10 sec. So to service 100MB/sec of synchronous writes, 1 GBytes of log device should be sufficient.
the last part here is the most significant IMO. doesn't matter if you have 4 or 40GB for the ZIL if your IO on the network is only gig interfaces then you're never going to see enough inbound writes to fill it. if you have 10gig ok, now you're looking at a lot more potential inbound writing but you're going to be bottlenecked by the ZIL in this case anyway.
also, IMO, its pointless to mirror the ZIL if you're using SSDs. each ZIL is going to wear out at the same rate anyway. run zfs 5/28 and don't worry to much about losing your zil or run an acard or ddrdrive instead.
I am quoting from the zfs best practices guide. Quote "The maximum size of a log device should be approximately 1/2 the size of physical memory because that is the maximum amount of potential in-play data that can be stored. For example, if a system has 16 GB of physical memory, consider a maximum log device size of 8 GB. " I guess I wasn't anal enough The source cites minimum and maximum sizes for the ZIL - seemed simplest to cite the maximum size so no-one will get in trouble (and the ZIL will be so small anyway, that there is little difference between the two.) In the real world, if the mininum size is 64MB and the max is 4GB, who is seriously going to do other than use 4GB to be safe? I really doubt anyone other than an enterprise SAN designer with 64GB of ram and 10gb nics is going to be tuning the ZIL based on network thruput, do you? As opposed to just getting a 4GB SSD?
The source cites minimum and maximum sizes for the ZIL - seemed simplest to cite the maximum size so no-one will get in trouble (and the ZIL will be so small anyway, that there is little difference between the two.) In the real world, if the mininum size is 64MB and the max is 4GB, who is seriously going to do other than use 4GB to be safe? I really doubt anyone other than an enterprise SAN designer with 64GB of ram and 10gb nics is going to be tuning the ZIL based on network thruput, do you? As opposed to just getting a 4GB SSD?
Last edited:
And there other issue is how many 4GB SSD's have decent random writes/iops? For the most part, with consumer drives at least, it's hard to get a drive that small with decent random write performance that's smaller than 40GB anyway. And you don't want to partition/slice your SSD as that essentially removed the drive cache from play.
i do, well, i will be soon. right now i have a few production ZFS boxes but will be purpose building a new one soon for VDI. multiple gig interfaces has a lot of potential in flight writes (and i may have the budget for 10gig ... hopefully). trouble is the only really great solution is openIA or nexenta only (ddrdrive). openIA isn't bad and i may go this route but if i wanted to pay per TB i wouldn't be paying nexenta thats for sure.
guess i'm just not sold on openIA being around or the rate of updates etc. with bsd you know its going to be around but there are no drivers for the ddrdrive.
the acard 9010ba with the supermicro 835 chassis as a head unit is interesting too.
bottom line i do think it is important to size your zil based on at least 50% of your total theoretical max bandwidth otherwise you're just choking your system. IMO.
- Joined
- May 7, 2007
- Messages
- 13,442
Oh come on madrebel, I'm surprised you would use ZFS over RAID0, I know just how much you love it.