Shadow migration speed

[tocket]

I got a new ZFS pool that I wanted to transfer data to from my old pool. After some attempts at zfs send/recv and not getting it to behave properly I discovered this service in Solaris called Shadow Migration (http://www.c0t0d0s0.org/archives/7432-Less-known-Solaris-11-features-Shadow-Migration.html). It seemed very flexible and capable so I decided to give it a try.

I'm currently transferring a 4TB folder, which is taking a good while. Two hours into the migration only 70GB has been transferred. That is too slow for my liking, especially considering that the zfs send did >100 MB/s.

Now, my question is if it's possible to speed this up. There are two properties that may be relevant to the speed, but I'm not sure how to tune these.

config_params/shadow_threads integer 8
config_params/shadow_throttle integer 1000000

I'm pretty sure the service is somehow CPU throttled, as my load average is hovering near 1.0.

Any ideas?

 

[staticlag]

cp command not good enough?

 

[hotcrandel]

config_params/shadow_throttle integer 1000000

sounds like it might limit you to ... 10MB/s ?

10MB/s * 3600 seconds = 36000MB/hr * 2 = ~70GB

Maybe double that and see if you get 140GB in 2 hours.

 

[tocket]

I tried changing the shadow_throttle and it seems to have no effect. Upon further investigation it seems that the shadow_throttle is only used when there are errors in the file system. I found comments like this inside of the source code:

/*
* If we have filesystems with persistent errors, we want to
* throttle migrations so we don't spend time spinning on CPU.
* To do this, we set the 'swe_next' field of the shadow
* migration when we encounter a filesystem with only
* persistent errors. We make sure we pick the filesystem with
* the next time interval.
*/

I did manage to speed up the transfer by increasing shadow_threads. With 64 threads it is running at about 45 MB/s. Although only 50% of the CPU is used, the source pool seems to be at its limit now.

After some more reading I came across this:

A design goal of shadow migration was to make the performance of the transfer unimportant to clients, the important factor being that clients continue to enjoy fast access to the data they require. However, if raw performance of the transfer between old and new media itself is important, you might not want to use shadow migration, because it is designed to be seamless and "invisible," but not fast itself.