• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZeusRAM bottleneck

U

UberMac

n00b
Joined
Jul 10, 2012
Messages
4
I am having a issue that I hope someone can help with. I have been testing various configs using Nappit on Open Indiana and Nexenta. I have a ZeusRAM as my SLOG. I seem to get only get about a maximum of 100GB/s using NFS and 10Gbe to the pool. I have tried 4 600GB 15K mirrored vdevs, 5 3TB 7.2k vdevs. No difference. in either case I would expect the at least a spiking aspect where I get 5 seconds of full bore transfer then if I don't have enough vdevs seeing disk waits. Instead I see non of this. A more real world test, have a DL380G7 with a 16 10k sas RAID0 volume (to exclude the read spead of the volume) that I try copying 10 22GB 7z files to the ZFS share. If I disable sync I get about 4-500MB/s write, about what I would expect given the overhead of the 10 files and network. With sync enabled I get about 100-150GB/s. This seems wrong. I would expect this using cheap SSD's as a zil.

Any ideas?
 
SLOG performance depends on latency -> http://www.nex7.com/node/12

100-150MB/s is actually very good write performance with ZFS and sync NFS. If you need more performance, you have to rethink your setup.
 
ZeusRAM SLOG. As for 100-150 being good, maybe if you are using standard SSD's but not by anything I have read. Keep in mind I paid several thousand dollars for 8GB. The slog is supposed to be sized by taking your trransfer bandwidth times 5 seconds which at 10Gbe is ~ 5GB. What I would expect to see is the SLOG loading up 5GB+ then flushing to disk. what I am seeing is 3-400MB in 5 seconds.I think I have found the problem becasue when I do a DD to the actual raw disk I get about 60-70MB. I am now trying to see if it is the controller or a bad drive.
 
Thank you for the reply, I have read the suggested blog many times and that is why I bought the ZeusRAM SLOG. As for 100-150 being good, maybe if you are using standard SSD's but not by anything I have read. Keep in mind I paid several thousand dollars for 8GB. The slog is supposed to be sized by taking your trransfer bandwidth times 5 seconds which at 10Gbe is ~ 5GB. What I would expect to see is the SLOG loading up 5GB+ then flushing to disk. what I am seeing is 3-400MB in 5 seconds.I think I have found the problem becasue when I do a DD to the actual raw disk I get about 60-70MB. I am now trying to see if it is the controller or a bad drive.
 
If your doing 4k writes, the max you will get is 180MB/sec, with the amount of latency the zeusram has.
16k writes will pretty much max out the zeusram at 700MB/sec.

This only covers local latency, add into that network latency, and 16k writes could easily be slowed down to 100MB/sec.
 
patrickdk said:
If your doing 4k writes, the max you will get is 180MB/sec, with the amount of latency the zeusram has.
Click to expand...

Yes, for a queue depth of 1, which I think the SLOG is fixed to.....could be wrong. Any Nexenta employees out there available to comment?
 
queue depth is set via zfs_vdev_max_pending

do
# echo zfs_vdev_max_pending::print | mdb –kw
0xa
#

and that will give you the decimal output.
 
queue depth is fixed at one, is a slog drive, NOT a vdev.

Now if you have multible write streams, it can combine them and max out the 128k writes, and give you max thoughput, but a single stream won't be that fast by itself.
 
I see 100-170MB/sec at most block sizes from VMware (10gb) to a ZuesRAM.

You are dreaming if you think any SSD can do that... some of the best can get near the low end, but they all have shortfalls, the least of which is almost none of them have supercaps anymore and if they do they're approaching the price of the ZuesRAM.
 
Hey Mac,

I found this link the other day which maybe useful to you!

http://christopher-technicalmusings.blogspot.com.au/2010/09/zfs-and-nfs-performance-with-zil.html

Basically he talks about how nfs does a flush of the zil after each write which slows down ZFS and NFS considerably also there is a function which you can enable nfs to say the data has been written to disk even tho its only been written to the zil.

As I understand the speed issue is related to the NFS client <-> NFS Server -> ZFS ZIL -> ZFS Disk, so basically whats happening is nfs is by passing all of the features of ZFS's ZIL And caching.

So the way around this is to tell the NFS server to acknowledge the write as complete as soon as it gets it and allow zfs to do its magic as it already does iscsi targets. this is not the same as using async for nfs.

I'd be very interested to hear how this goes for you on 10GBE.


Also as a side question for you how did you go about getting AOE/ATAOE support in openindiana?

Also i've been looking at the SRX by coraid lately and i was wondering what kind of CPU's and amount of memory do they put into the box you have there?

Cheers,
Qten
 
That isn't what is going on.

Nfs writes in sync, ZFS gets the sync write and sends it to the zil, it confirms it's written to the zil, then returns to nfs it is safe, and can go on to the next operation.

So disable zil, disable sync writes all will disable all safety you asked for, making it really fast, and not safe.

What that document above does, is disables the flush cache command that is sent to the zil after the write. So basically with that, you write to nfs, then zfs writes it to zil, and returns everything is safe, without actually checking or waiting for the write. So you might be or might not be safe.

ATAOE was always a hack. The offical current thing is FCOE or iscsi. Both are supported.
 
nexenta officially supports AoE. i'm not sure about target support but the initiator is at least.
 
You must log in or register to reply here.
Back
Top