• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS De-Dupe existing data

P

Paladin21

Gawd
Joined
Jun 22, 2004
Messages
529
I've got a pool on my SAN that is getting quite full of data. As a large percentage of this data is duplicate OS stuff (~120 clones of a VM) with a small amount of student work on top, I should be able to get a very nice de-dupe ratio on the pool to fit more VMs on it. My question is: is there some way other than moving everything out of and back onto the pool to de-dupe the data already written? I can schedule that sort of thing across the weekend if I need to, but I'd really rather not have to babysit the process across a holiday weekend. Thanks for the help.
 
You don't need to move out of the pool. You can do like:

cp xxx xxx.temp
rm xxx
mv xxx.temp xxx

lather, rinse, repeat
 
How about rsync with --delete-source-files

to a different folder. I can't remember if that deletes during or after so you may need an option to change when the delete happens.
 
You should avoid ZFS dedupe, as the dedupe engine is not that good. You need at least 1GB RAM for each TB disk. The ZFS dedupe engine is immature. (Tegile and Greenbyte has rewritten the ZFS dedupe engine for excellent performance, but they are not open sourced)

Instead, ZFS compression works very fine and should be turned on by default.
 
You must log in or register to reply here.
Back
Top