• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Cheap ZeusRAM alternative

That's an interesting idea. I have been running with sync=disabled since I do daily backups. If I want to try an HA/failover solution with my dual-head SAS JBOD enclosure, it entails export on the one host and import on the other. That is in the voluntary failover case. If the owning host is hosed, the backup needs to forcibly import the pool. Not acceptable to run sync=disabled that way, since you could have several seconds of disk writes vaporize with no hint to the client(s).
 
THE BEST SSD q1 2014 compared, from BESTEST to very good in MAX IOPS (for pour ZIL ZFS)

HGST UltrastarHUSMH8040ASS204 <- ref avec 4 à la fin = without encryption : 2050 &#8364; in 400 GB
-> http://www.tomshardware.com/news/SAS-HGTC-12-GB-s-SSD,21929.html
-> http://eurocapa.com/hitachi-ultrastar-ssd800mh-400gb-14610.html
-> http://www.span.com/product/Hitachi...MH8040ASS204-0B30067-SAS-12Gb-400GB-SSD~40509

HGST Ultrastars840Z
-> http://www.hgst.com/solid-state-storage/enterprise-ssd/sas-ssd/s840z-zil-sas-ssd

HGST Ultrastar SSD800MM - eMLC SAS3 12 Gb/s
-> http://www.storagereview.com/hgst_ultrastar_ssd800mm_sas3_enterprise_ssd_review

Toshiba PX02SM - eMLC SAS3 12 Gb/s
-> http://www.storagereview.com/toshiba_px02sm_series_enterprise_ssd_review

SMART Optimus TXA2D20400GA6001 - eMLC
-> http://www.storagereview.com/smart_storage_systems_optimus_sas_enterprise_ssd_review

STEC s840 - eMLC
-> http://www.storagereview.com/stec_s840_enterprise_ssd_review

Toshiba MKx001GRZB - SLC
-> http://www.storagereview.com/toshiba_mkx001grzb_enterprise_ssd_review

Hitachi Ultrastar SSD400M - SLC
-> http://www.storagereview.com/hitachi_ultrastar_ssd400m_enterprise_ssd_review

Intel S3700s
-> http://www.storagereview.com/intel_ssd_dc_s3700_series_enterprise_ssd_review

Kingston E100s
-> http://www.storagereview.com/kingston_ssdnow_e100_enterprise_ssd_review


Best Hard drive

HGST Ultrastar C10K900 HUC109090CSS600
-> http://www.hgst.com/hard-drives/enterprise-hard-drives/enterprise-sas-drives/ultrastar-c10k900
 
Thing is, the ZIL doesn't really care about IOPS. What it wants is low latency. Sure, drives with lower latencies are often in the higher end of the IOPS spectrum, but it is two completely different performance metrics.
 
I suppose when QD=1 we have latency=1/IOPS?

Emmm...seems this will not work with HA. My setup is kinda of all in one so didn't notice this. @danswartz do you have SLOG then? If yes is it located in your JBOD?
 
No, at the moment, I run sync=disabled, since I don't care about crashes (backups every day, snapshots every hour, etc...) otherwise write performance blows...
 
What about monitoring your UPS and as soon as there's a power outage, run a script to switch sync from disabled to standard?

As for the hack, the HP P420 supports up to 2GB of cache.
 
Idea. On the other hand, you could have the storage appliance crash for other reasons (freeze, NMI, kernel panic, etc...) and I'd be screwed there too...
 
I think HW RAID cache has really low latency thus good for ZIL usage? 2GB would be good for my use as the load is very light most of the time. Even 512M would be good, as I currently use a commercial SSD not even having capacitors.

I can't try this idea on my own as I've maxed out my PCI-E slots:( theoretically is there any caveat besides not supporting HA?
 
Seems to be some confusion here. For an AIO, the lack of HA has nothing to do with an SLOG device, but rather that the storage in that AIO is not really 'shared storage' (a prerequisite for HA). Vsphere would indeed think it's shared, but if that host fails, the storage is not available.
 
Yes that's why I'm not too concerned with HA for now.

Just reading Intel DC S3700/3500 specs and found that they both have latency 50&#956;s for read and 65&#956;s for write, whereas the latency is defined as "Device measured using Iometer. Latency measured using 4KB (4,096 bytes) transfer size with Queue Depth equal to 1 on a sequential workload." I believe that is the most critical measurement for SLOG performance?

Where can I get the latency figure for HW RAID card (e.g. HP P420)?
 
if the zeusRAM is 'too expensive' then you're in a position where absolute best performance doesn't matter anyway.

sure, you want that performance but can't justify the cost. you then need to be happy with whatever performance a slc/mlc/emlc/tlc drive can deliver.

also the zeusRAMs are down to like $2100 which is fairly reasonable.
 
well I have to say you made a good point...it's all about cost. my intention is that by using a HW RAID card, which is in the similar cost range of a consumer SSD, I can achieve better performance than a consumer SSD.

Intel DC S3500 80GB might be another choice for SLOG if it's better than a consumer SSD in this regard?
 
Or, like I've said more than once: since it is zfs, run hourly snapshot out of cron, and go sync=disabled with a decent UPS.
 
I'm a little late to the party, but a couple of things come to mind.

One, ZFS doesn't like anything behind a RAID controller. It can't see what the drives are doing when it's hidden like that. It'll tolerate it, but it's not ideal. It's an interesting idea, though, to use the RAID controller between the drives and ZFS just for the ZIL.

In either case, the type of SSD you use for the ZIL matters. A ZeusRAM is unique in that it's not just a RAM backed SSD like an Intel S3500/S3700. Both use supercapacitors to power themselves for the 30 seconds or so it takes to flush RAM to flash. Here's the difference:

- A ZeusRAM shows up as a 8GB drive and never goes to the flash unless power is lost. It uses the flash as a non-volatile backup during loss of power, not as a part of it's normal operation. This is why it lasts nearly forever.

- An Intel S3500/S3700 shows up at a large capacity because it'll use it's flash during normal operation, using it's RAM cache to smooth out writes and spare the flash from excessive write wear. These type of SSDs typically use eMLC flash, but the RAM cache gives the cheaper flash write durability like an SLC flash drive, while providing a speed boost.

Since a ZIL is all writes all the time, a standard non-RAM cached SSD doesn't last long. The flash just gets beat to death. A RAM cached SSD survives much better - some people reporting 2 and 3 years before they finally fail. A lot of this will depend on the write pattern - if writes are large enough to overflow the cache's capacity and they end up on flash, life will get shorter fast (ZFS will do this because it sees a drive that is 80GB or more, not knowing that the RAM cache is actually much smaller). The ZeusRAM, since it shows up only as the capacity of it's RAM, never goes to flash, even for large writes - ZFS just works around the smaller capacity.

So, that explains why the ZeusRAM is unique - and expensive. You use them sparingly - usually for ZIL and nothing else. An Intel S3500/S3700 can do a good job, though, in most cases - though I'd definitely set up a ZIL mirror if I was using a drive that is going to flash at any time (and the Intel will, no matter what the RAM cache is doing). Losing a ZIL means losing in-flight writes, which can lead to a big mess for just about any application.

One more thing to consider for those thinking about using a shared JBOD in an HA configuration - you want SAS drives in those, so each of the 2 heads has a path to the drives (SAS drives are dual path). While ZeusRAM is SAS, Intel S3500/S3700 aren't. Thus the Intel makes no sense in a HA configuration.
 
I'm currently testing a RAID controller backed SLOG like that. I wanted to do something like this for quite a while now and got my hands on a cheap 9271-4i a while ago.

Just as a note, you cannot use normal consumer SSDs, as those do not make any guarantees what data actually reaches stable storage and what is lost even on drive cache flushes.
While I have yet to observe any data loss with my SSDs, there exist some reports about it.
It has to be an SSD with capacitors like the S3700. By the way, the same is true for pool SSDs, those also have to have power-loss protection if you really need to make guaratees about your data consistency.

I'm also testing with a massively short-stroked spinning drive as an unlimited-write replacent for the SLOG SSD.
 
@omniscence, let me know how that goes. I am kinda stuck on my HA plans now, as my s3700 SLOG is sata, so it is not visible to the other HBA (on the jbod's second port). sas ssds with good latency are very pricey. I have a 72GB 15K RPM sas drive that I could short-stroke and use if need be. I am only servicing a single gigabit link so that *might* work performance-wise...
 
So far, not looking good. Crystaldiskmark for a virtual disk on a win7 guest. With intel s3700 SLOG, I get 93MB/sec. With a 15K RPM 73GB SAS spinner, it's turning in around 35MB/sec :(
 
danswartz said:
So far, not looking good. Crystaldiskmark for a virtual disk on a win7 guest. With intel s3700 SLOG, I get 93MB/sec. With a 15K RPM 73GB SAS spinner, it's turning in around 35MB/sec :(
Click to expand...

You are using the SLOG on a RAID controller with enabled writeback cache? A plain spinning drive will of course yield bad results.

Does CrystalDiskMark even do sync writes?
 
I'm confused, I guess. You are the guy who was proposing using a short-stroked spinning disk as a replacement for an SLOG SSD, no? No, I'm not using a raid controller wite writeback cache - this is on the JBOD/HBA the data pool disks are on. I didn't expect as good results as the s3700. I was hoping that a 15K disk with only a few GB being used would give me maybe 50MB/sec, since that drive should be able to peak at about 200 IOPS. Crystaldiskmark is not the key factor here, the sync writes happen because this is a virtualized win7 CDM runs in, and vsphere forces NFS writes to be synchronous.
 
No, I proposed (although for sure I am not the first one thinking about that) to use a short stroked spinning disk on a RAID controller with enabled (and battery backed) write cache.
A spinning disk alone would not make much difference even with a 15k rpm drive. The drives latency is limited by the time it takes for a revolution of the medium.
The latency of the RAID controller cache is much lower than any disk connected to SATA or SAS, so even lower than ZeusRAM.
And the time it takes to commit the SLOG write to the spinning disk does not really matter as long as the controller can aggregate the writes well enough.
That way you can have a cheaper unlimited-write SLOG. Or you can use a write-intensive SLC based SAS SSD, which should still be cheaper than ZeusRAM should the spinner be to slow.
 
Last edited:
Ah, okay. Makes sense. Unfortunately for me, that won't work (AFAIK) for a dual-homed solution. I did expect (and confirmed) that the 15K spinner SLOG is better than on-pool ZIL (pretty low bar there...) Yeah, there are some (apparently) decent SAS SSDs running in the mid hundreds (e.g. $500 or so...) pricey but a lot cheaper than a zeusram :)
 
@danswartz: If you only have a 1Gbps link to your storage, a ZIL is almost pointless. You won't be seeing 1Gbps out of the link, given TCP/IP overhead. You might see 100MB/sec in a perfect world. There isn't enough bandwidth on the network link to stress even a small set of 6Gbps spindles. A dozen or so 10,000RPM spindles would deliver well over 100MB/sec.

You could look at MPIO over multiple 1Gbps links to get more speed, but by the time you scale up, you might as well have gone 10Gbps and avoided the rats nest of cables and the cost of a bunch of network ports and NICs.

LACP / EtherChannel is an option, too, but it doesn't scale as well as MPIO, especially if you have only one host talking to the storage - in that case, bonding links won't speed anything up, since a single path will only use one link at a time. All it'll provide is redundant links in case of a failure.
 
I do in fact get about 110MB/sec for reads and 93MB/sec for writes. The problem with your dozen spindles idea is that I don't need that much space, I already have a 3x2 raid10 pool using nearline SAS drives (which work just fine.) I have to ask: have you actually tried what you are proposing? It is pretty much universally understood that on-pool ZIL sucks for writes (for spinning pool drives, anyway) due to having to reposition the heads twice - once to write the ZIL records, then later to write the actual data.
 
I am proposing not using a ZIL at all with a 1Gbps link to get to the storage from your hosts. The ZIL will speed up disk access, but you'll never see the performance increase because you can only access the data so fast over a 1Gbps link like that. If you're already getting 110Mb/sec and 94MB/sec with 6 spindles, then you're pretty much maxing out the 1Gbps link already. Your bottleneck isn't the disk performance, would be my assumption.

In terms of space vs number of drives, when building an array, you need to calculate how much performance you'll need (IOPS), then make sure your drives are small enough to give you the performance you need for the amount of space you require. If you're filling your array with huge drives and stop at a certain amount of total raw storage, then you'll have sacrificed IOPS for cost. The trick is figuring out your IOPS needs, space needs, and how much money you can afford - then finding a balance that you're happy with. One other thing to consider is resilvering time (rebuilding time, in legacy RAID speak). If you have huge drives, recovering from a failure can take ages. With really large drives, I'd recommend at least RAIDZ2 so you get a second shot at the resilver process if a second bad drive is discovered during recovery. Single parity is dangerous at the best of times, but especially with really large drives.
 
Sigh. I am getting the wire speed *because* I am use a high-quality SSD as an SLOG. vsphere does ALL NFS writes as synchronous, so without either disabling sync on the dataset (dangerous) or using a good SSD like the s3700, you use on-pool ZIL with performance of maybe 25MB/sec. This is not speculation, this is a well-known factor of synchronous ZFS writes. Due to the way I/O is ordered for sync mode, this is inevitable. Not meaning to be pissy here, but this is not remotely new or unique to my situation, as 10 minutes with google will confirm. I've been using ZFS for several years, so while I appreciate your attempt to be helpful, I was well aware of the factors you mentioned, and didn't really need a 101 level lecture.
 
Ok. You know it all, so I'll stop bothering to help. Proper storage engineering is pretty basic info, too - for ZFS or legacy RAID. But, you go with whatever works for you.
 
Sigh. I certainly don't know it all (about ZFS or anything else.) I certainly know more than you, pretty clearly (about ZFS). If you knew the least bit about some of the quirks of ZFS you wouldn't have felt compelled to start giving me a storage 101 lecture.
 
Your comment was snarky and completely mislead.

Given I've built storage solutions on top of Nexenta in HA configurations for enterprise clients for a number of years, I'd say I am well versed in the ins and outs of ZFS and HA storage.

But, as I said, go with whatever works for you. Hack it together however you like.
 
You must log in or register to reply here.
Back
Top