• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

SSD for ZIL

O

oraldlight

Limp Gawd
Joined
Feb 12, 2007
Messages
490
Is there any guideline for how large a SSD to purchase for ZIL?

I intend to use 6-2tb drives.
Someday, maybe grow that to 12-4tb's? (right after I win the lottery....)

Intel 311 20gb big enough?
 
Normally, unless your adjusting default params, need enough to cache 5seconds of write data.

So if your limited to gigabit speeds, 100MB * 5seconds = 500MB max would be used, about.

The other thing you want to look out for, would be the MAX transfer speeds and iops of the device.

An Intel 311 caps out at 85MB/sec and 3100iops per second. So you won't even get full gigabit speeds if you use one of them as a zil device.
 
Interesting details you provided.

I realize SLC is favorable to MLC, which is why I listed the Intel.

I hope to team dual Gigabit NICs. Therefore, your math says about1gb/sec. So Raid0 2-SSD's to keep up???

OCZ SLD3-25SAT3-60G?? Fast enough??
 
oraldlight said:
OCZ SLD3-25SAT3-60G?? Fast enough??
Click to expand...

I don't have an answer for you, but this is a scenario I have just started playing with myself. General recommendations seem to say look at getting a STEC ZeusRAM drive, but if you can't work a multi-thousand dollar ZIL log drive into your purchase, what can you do?

I have an existing setup with 20x 300gb 10k SAS drives that I am hoping to run 30+ VM's off of, and expand it to 40x 10k SAS drives soon. The setup offers plenty of IOPS to work with, and I'm able to see over 700MB/s+ speeds locally, so in theory I should be able to max two bonded GbE NIC's.

With sync disabled, I was easily able to saturate the 2x 1GbE NIC's getting approximately 215MB/s total.

Once I enabled sync and accessed via NFS, I found without a dedicated log drive, I was only getting close to ~40MB/s TOTAL.

I initially tested with a spare SSD I had laying around, and saw ~70MB/s or so, which proved enough to me that there was room to improve. I tried picking up a "better" SSD, the Mushkin Enhanced Chronos Deluxe 60GB SSD. The drive boasted the following specs:

  • Sustained Sequential Read: Up to 560MB/s
  • Sustained Sequential Write: Up to 495MB/s
  • 4KB Random Write: Up to 85,000 IOPS

I've only just started playing with the 60GB SSD (some other projects came up that have delayed me), but with my teamed NIC's and sync=always, I am seeing "better" speeds, but even with an SSD that advertises 85k IOPS and ~500MB/s, I can't seem to saturate the two GbE NIC's like I can with sync=disabled.

Maybe the RAM-based SSD's really are all the only way to go... I was hoping for a more affordable MLC option.

I would be interested in hearing if others have found better performance with cheaper MLC options?
 
There are other threads in the forum explaining the issue with SSD ZIL's, and why they have a hard time saturating multi-gigabit. Nex7 has a write up about how they (Nexenta, or some of their employees) found that the limiting factor is write latency, and RAID0 (aka multiple slogs) didn't help at all. Every sync write requires a response that it wrote the given data before the next block of data is sent. The ZeusRAM and similar devices have a very significantly lower write latency, as well as high-IOPS, both of which matter.

I personally have a box that is running OI+NappIT with a pair of Intel 320 300gb SSDs (slog - mirror) under provisioned to 15gb used for VMware storage. I get varied results on the high-end... anywhere from 35mb/sec to 90mb/sec depending on how loaded down they've been (I suspect they work a lot better when they've been able to do decent garbage collection recently). On the plus side, the drives appear to have a very long life span in this use case with fairly high-load, which I was unsure of when I put them in -- I've written 4.4TB to them so far and they're still showing 100% on the media wear indicator. For me, I'm happy with the performance (vs without them) - I get drastically lower read/write latency from VMware, which is what matters a whole lot more than throughput most of the time, and I can read from the array extremely fast.
 
How are you guys measuring speeds?

I ask because I'd like to follow the same procedure so we can compare "apples to apples"
 
femi said:
How are you guys measuring speeds?

I ask because I'd like to follow the same procedure so we can compare "apples to apples"
Click to expand...

if you run disk benchmarks (like hd tune) inside a VM, use a paravirtual scsi adapter
 
femi said:
How are you guys measuring speeds?

I ask because I'd like to follow the same procedure so we can compare "apples to apples"
Click to expand...

Nothing fancy at the moment, I've been monitoring bandwidth usage on my teamed NICs using a simple iftop ("iftop does for network usage what top does for CPU usage").

Code: 
iftop -i aggr0 -B

The above command will launch iftop and will monitor the interface (-i) aggr0, the -B flag with convert to Bytes instead of bits. This works well enough for monitoring bandwidth between a few hosts, so you can see individual traffic on a per host basis, as well as aggregate bandwidth of the interface.

If you are using OpenIndiana, you can install iftop using: pkg install iftop
 
dsumike said:
This works well enough for monitoring bandwidth between a few hosts, so you can see individual traffic on a per host basis, as well as aggregate bandwidth of the interface.
Click to expand...

Here's a small sample, with SSD ZIL and sync=standard:

MgD86.jpg

And here is with multiple connections, and sync=disabled, where you can see ~205MB/s:

zNSsm.jpg

As you can see, with a couple filters you can view specific service traffic, and filter by hosts. I turned off source/destination IPs, hence the *'s, but I figured you could at least see what it looks like. The three columns are throughput in 2, 10 and 40 second averages.

IOSTAT shows some decent real time stats too:

Code: 
# zpool iostat storage 10
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
storage     1.11T  1.61T      0     14  2.23K   935K
storage     1.11T  1.61T      0  1.20K      0   101M
storage     1.11T  1.61T      0  1.31K      0   110M
storage     1.11T  1.61T      0  1.32K  12.8K   110M
storage     1.11T  1.61T      0  2.19K  12.8K   188M
storage     1.09T  1.62T      0  2.26K      0   194M
storage     1.09T  1.62T      0  2.33K      0   204M
 
Last edited:
Child of Wonder said:
How about the Intel 520 series?[/url]
Click to expand...

While not definitive, I saw a post in another forum where the Intel 520 series SSD was mentioned (except ZFS on NextentaStor):

http://nexentastor.org/boards/2/topics/5662

White stated:

I've added a intel 120GB 520 series SSD on it, perfectly able to write 200MB sec plus, also 6x 15k 73GB sas drives in "raid0" , both don't go over 40MB sec doing sequential writes using NFS v3.
Click to expand...

It doesn't look like it will do any better, quoted at ~40MB/s.
 
There may not be a good one. Forget saturate, I have a windows xp guest doing crystal mark to a 'local disk' (esxi virtual disk from a NFS share) and get about 110mb/sec read 30mb/sec write. Disabling sync gets the write speed up to 70mb/sec or so. I have the SAN/NAS on a UPS, so I may just live with this.
 
Child of Wonder said:
Weird.

So without spending $1,000 there may not be any good options for a ZIL that can saturate 2x1Gb links?
Click to expand...

Keep in mind this is only for synched writes. So it will depend on your use too. Normal NFS mounts from other linux machines can easily saturate the 2x1GbE link when multiple streams are going. Also, the link has plenty of bandwidth for reads then too. So you can still saturate the link depending on what kind of traffic you have.

With the Mushkin SSD, I'm able to get synched writes at up to 65-90MB/s it seems (with some random bursting -- I still want to do more testing on this), with additional reads going on simultaneously, I can still take advantage of the 2x1GbE link.

Worst case, I am considering what danswartz mentioned as well, and can always set sync=disabled if I feel I need the extra performance over data reliability. Since it would be a very small possibility of failure with dual power supplies+UPS, I should be fine... but I might try with the SSD SLOG/ZIL for awhile first.
 
danswartz said:
There may not be a good one. Forget saturate, I have a windows xp guest doing crystal mark to a 'local disk' (esxi virtual disk from a NFS share) and get about 110mb/sec read 30mb/sec write. Disabling sync gets the write speed up to 70mb/sec or so. I have the SAN/NAS on a UPS, so I may just live with this.
Click to expand...

That sucks. I just tested mine and was able to get 99MB/s via ATTO and 88MB/s with CrystalMark.

This is over NFS with 8 disks mirrored and sync disabled.
 
You must log in or register to reply here.
Back
Top