Dedupe worth it?

[5loth]

Currently running OI + NappIT All-in-One on ESXi 5.

Host has Xeon E3-1230, 16GB ECC, M1015 passed through to OI VM, and using 2 x 1TB Mirrored for NFS Datastore for other VMs and 3 x 2TB in RAIDZ for storage. I only get about 3.2TB usable for storage, and as I'm unable to simply add another drive to expand the array I started looking at using deduplication. I've got a Intel 320 Series 40GB SSD I can use for L2ARC if that is going to help. Currently have 8GB RAM allocated to the OI VM. The NFS Datastore is only used by test VMs/not performance critical stuff.

Storage is mostly filled with video files sized from 150MB to 15TB, couple of GB worth of pictures and some music/applications etc.

Am I likely to gain much usable disk space if I enable dedupe, and would my current setup perform OK with it enabled? Would adding the SSD be of benefit?

Also, can dedupe be easily disabled/reversed out of if the performance isn't satisfactory without losing any data/having to start from scratch?

Any other suggestions appreciated.

 

[brutalizer]

No, it is not worth the effort to dedupe. You need tons of RAM, and SSD is beneficial. Still it takes day to delete a snapshot which is deduping lot of data. Better and simpler to get more disks.

 

[Tiporaro]

Unless I'm missing something in your post, it doesn't sound like you have particularly repetitive data where you'd have a high dup ratio. That being the case, the downsides/costs of running DDP probably make it nowhere worth running for you, although I'm sure someone wiser than I could chime in if I'm incorrect.

 

[apnar]

Storage is mostly filled with video files sized from 150MB to 15TB, couple of GB worth of pictures and some music/applications etc.

Nothing in this list would benefit from dedupe, unless you have lots of copies of the same or nearly the same videos.

There is a very narrow range of uses where dedupe makes sense. The two that I can think of off the top of my head are a backing store for lots of similar virtual machines or a backup destination where the backup software doesn't handle deupe itself.

 

[thedge]

If a lot of your VMs are very similar youd likely get some benefit there. But zero benefit on the videos unless you have a lot of copies of them.

 

[odditory]

As others have said dedupe is pointless for home media archival. Unless you're in the habit of making multiple copies of the same files for some reason.

On top of that, enabling dedupe won't work against existing duplicate files -- it'll only start working against files copied to the pool from the point of enabling dedupe forward. One gotcha to keep in mind. To get around that you'd have to back everything up, delete everything and copy it back. At least this was the case the last time I messed with dedupe on Nexenta + NappIt about 6 mo's ago.