ZFS Snapshots

[Guldan]

Hey Guys,

I'm playing with two freenas servers (using zfs), both big boxes (lots of cpu/ram) and each have an MD1000 w/ 15 disks, one of which im using to boot the OS and the rest I'm using in a raidz3 (two 7 disk vdevs).

Anyways my question is, I enabled ZFS replication from freenas01 -> freenas02 which is working swimmingly but I noticed when I was hammering it the snapshots looked to be lagging behind. They caught up fairly quickly when I stopped my assault but should I be worried about this?

Currently I'm using a 1gig network link but if I put it in production i'll upgrade to 10gig, perhaps segregating the network traffic would help? I've used ZFS for a while but am new to snapshots just wondering if anyone has experience especially in an enterprise environment.

Thanks!

 

[danswartz]

Well, if you are doing more than 100MB/sec or so, there is no way a 1gb link will be able to keep up... Although if the payload is compressible, that might help (also, are you replicating using ssh? if so, that will be dog-slow...)

 

[zrav]

As Dan already mentioned, ssh can be a bottleneck, its encryption/compression seem to be singlethreaded.
You didn't mention what kind of load you are putting on the pool, but you are using raidzX so I would put my money on it becoming IOps limited.

 

[danswartz]

If security isn't a concern, using mbuffer/netcat (?) instead of ssh might work well...

 

[omniscence]

Since I do not know how FreeNAS handles the transfer, I also recommend to look into mbuffer. netcat probably will not solve the issue if it is related to pipe starvation.

If this is not the issue, it depends on what you mean by "hammering".. a highly random workload? Then like it was already mentioned, the pool is possibly IOPS limited. In this case it would probably help to add an properly sized L2ARC, or even better, more RAM. If it is a highly sequential workload, only more network bandwidth or a lower snapshot frequency can help.

You have to remember that everything you get on the pool the replication process has to get off again if it exceeds the ARC size.

The best would be to ask this in the FreeNAS forums, because there are more people around that have insight into the actual implementation of the replication feature.

 

[Guldan]

Well, if you are doing more than 100MB/sec or so, there is no way a 1gb link will be able to keep up... Although if the payload is compressible, that might help (also, are you replicating using ssh? if so, that will be dog-slow...)

Yes Freenas/ZFS uses SSH for snapshotting and it does appear to be dog slow unfortunately.

Since I do not know how FreeNAS handles the transfer, I also recommend to look into mbuffer. netcat probably will not solve the issue if it is related to pipe starvation.

I'll have to look up mbuffer and netcat, Freenas uses Freebsd/ZFS and these snapshots are the default replication method via the GUI but that doesn't mean I have to use it.

If this is not the issue, it depends on what you mean by "hammering".. a highly random workload? Then like it was already mentioned, the pool is possibly IOPS limited. In this case it would probably help to add an properly sized L2ARC, or even better, more RAM. If it is a highly sequential workload, only more network bandwidth or a lower snapshot frequency can help.

You have to remember that everything you get on the pool the replication process has to get off again if it exceeds the ARC size.

The best would be to ask this in the FreeNAS forums, because there are more people around that have insight into the actual implementation of the replication feature.

By hammering I mean I took four of my production servers on 10gig links and used richcopy to do a multithreaded transfer to each of the freenas servers while replication was running. Each server had different type of files (large/small/mixed) then I pulled out 3 disks from each raidz3 and observed how the data transfer and snapshots reacted.

There was a blip in a couple of the network transfers and they had to retry plus the snapshots started to lag behind but other than that it handled it pretty well.

As Dan already mentioned, ssh can be a bottleneck, its encryption/compression seem to be singlethreaded.
You didn't mention what kind of load you are putting on the pool, but you are using raidzX so I would put my money on it becoming IOps limited.

See my response above for load, but yes using triple parity is quite the hit in performance. That in combination with using a 1 gig link is quite limiting im sure.

--

On further research spawned by your questions it does appear SSH is the limiting factor on replication performance. I've found a couple threads on the Freenas forum where people are having the same issue.

I'll look for specific solutions based on that, thanks guys.

 

[zrav]

You can eliminate the SSH bottleneck by using HPN-SSH: http://www.psc.edu/index.php/hpn-ssh
Still, by your description it sounds like an IOps issue (replication catching up when you reduce the load on the pool).

 

[Guldan]

You can eliminate the SSH bottleneck by using HPN-SSH: http://www.psc.edu/index.php/hpn-ssh
Still, by your description it sounds like an IOps issue (replication catching up when you reduce the load on the pool).

RAIDz3 is going to hit me pretty hard with the 3 disk parity, plus im using regular 7200RPM sata disks and passing them through HW raid0 before ZFS gets ahold of them.

I'm using spare hardware to test Freenas, if I like what I see we'll buy custom boxes full of 6TB drives / 10GBe etc,.

 

[omniscence]

Yes Freenas/ZFS uses SSH for snapshotting and it does appear to be dog slow unfortunately.

I did a test a while ago on a E3-1230v3 and it could not do much more than 250 MB/s over SSH even when using mbuffer if I remember correctly. But this was Linux, maybe on FreeBSD it could achieve more (or less). If the stream really need to be encrypted maybe IPSec is the way to go?

RAIDz3 is going to hit me pretty hard with the 3 disk parity

Nah, the parity calculation does almost nothing in terms of performance. It is the general implementation of the parity vdevs in ZFS. Each parity vdev basically has the IOPS of a single disk regardless of the actual number of disks, so your pool has the combined IOPS of two drives. That is one major difference and drawback in comparison to conventional parity RAID. You could get hundreds if not thousands of megabytes per second from your pool in sequential transfers, but as soon as the load gets random, it slows down. That is also why the people here recommend to use mirror vdevs or SSD-only pools if lots of IOPS are required.

 

[danswartz]

Did you really mean 250MB/sec?

 

[omniscence]

Did you really mean 250MB/sec?

Yes. 10gb ethernet with jumbo frames between two directly connected machines and running mbuffer with a gigabyte buffer through it. I have to say that is was a while ago and I do not remember the exact number, but it was significantly exceeding the 100 MB/s so it would have saturated gigabit ethernet. It was not enough as I wanted to see more than 500 MB/s. I finally just used mbuffer and achieved 750 MB/s, which was limited by the fragmentation of the pool, which easily could have delivered 1 GB/s on sequential transfers.

EDIT: quick test:

root@myserver:~# dd if=/dev/sds iflag=direct bs=128k | mbuffer -m 1g -s 128k | ssh localhost "mbuffer -m 1g -s 128k >/dev/null 2>/dev/null"
in @  362 MB/s, out @  361 MB/s, 5699 MB total, buffer 100% full

The buffer is full on the source side, so the bottleneck is SSH. I could not manage a network test so quickly, but mbuffer should mask any network related lags.
I used an actual drive as source because I wanted to avoid some internal compression of /dev/zero, but a second test with /dev/zero yielded the same result.
It peaked at over 400 MB/s. AFAIK this is only possible with a recent AES-NI enabled system, as SSH is single-threaded.

EDIT2: finished to transfer my 830 256 GB SSD:

summary:  238 GByte in 11 min 15.4 sec - average of  362 MB/s

The SSD could not keep up with SSH at all times, but I did not have a buffer underrun.
I remember that my second machine back the was a E3-1230v2, which does have a less recent AES-NI implementation and resulted in a lower throughput.

 

[patrickdk]

Or use HPA enabled ssh.

though solaris/illumos use sunssh, so that isn't compatable.