SSD and NTFS Compression Speed Increase?

Upon researching SSDs I came across a couple of threads that suggest that compressing folders can actually improve SSD performance.

So I went ahead and compressed my Program files folders (both X86 and X64), and I'm happy to report that in the very least I don't "feel" there's a performance penalty.

The thinking was that modern CPUs are fast enough to handle all of the overhead needed to decompress the compressed folders, and overall less data is actually being read and written to by the drive. Also, the access time is so small, that all of the seeks that used to slow down compressed hard disks are a non issue.

It was hard to quantify, and AS SSD benchmark strangely reported lower numbers, even though it was using clean free space for the benchmarking. I went ahead and decompressed the AS SSD folder, and results went back to normal.

Otherwise, games load up just the same as they would normally.

Any opinions?

Generally not a good idea since for your system disk you want low latencies, but encryption or compression would add to the latency because it first has to be decrypted/decompressed. That adds latency to each I/O would could add up with things like application loading (basically random I/O).

So i don't think it will particularly help with speeds; it may yield more space to store stuff on SSDs however.

sub.mesa said:

Generally not a good idea since for your system disk you want low latencies, but encryption or compression would add to the latency because it first has to be decrypted/decompressed. That adds latency to each I/O would could add up with things like application loading (basically random I/O).

So i don't think it will particularly help with speeds; it may yield more space to store stuff on SSDs however.

Click to expand...

I completely follow you. Thought the same way and was screwing around with my 2 30GB Vertex's because the 60GB I get from Raid-0 was starting to cramp my style.

I just picked up two 100GB Vertex LEs, So with the larger capacity, I won't be screwing with compression.

But there are posts at various forums claiming that compressing folders could enhance SSD perfomance. The sandforce chipset does this already interanlly (compressing and decompressiong by hardware built into the drive itself,

Hmm.. This is unexplored territory...

The theory is simple, I'll try to make it brief (which means I'll explain it with way too many words, as usual)

Say you have a 100MB file you need to load into an app, the app itself is already open and waiting. Say the 100MB file is uncompressed presently, but the file is made up of highly compressible, and we'll use the oh-so-common ratio of 2:1 meaning if you did use NTFS compression, you'd end up with a compressed version of the file on the platters that's 50MB in size.

You'd verify that by right-clicking on the file icon or name and choosing Properties where Explorer would tell you "Size" which is the native uncompressed actual file size, and "Size on disk" which would tell you the size of the file as it exists in the now compressed form, rounded up to the nearest 4KB boundary (given the 4KB cluster size of NTFS unless you altered that, bad idea).

Ok, now the math:

Let's say it takes 1 second, precisely 1 second to load the original uncompressed 100MB file into the app, 1 second exactly. Logic would dictate that to load a file exactly half that size should take exactly half the time, and let's say on your example machine it does. If you were loading a 50MB uncompressed file it would load in .5 seconds.

Now the fun part:

On today's modern computers doing mathematical operations is a cakewalk. NTFS uses a form of compression known as LZ77 which is incredibly efficient and very fast in operation. So fast, in fact, that this is where our potential speed benefits come from.

Original file uncompressed load time = 1 second flat.
Compressed version of same file load time = .5 seconds

Time to decompress the compressed version and feed it to the application = .003 seconds, or less... as the theory holds but it depends on two variables: CPU computing power and RAM speed/timings.

So more simple math would say "Hey, it takes me a full second to load the compressed file which ain't so bad, but it doesn't even take 6/10ths of a second to read, decompress, and make the uncompressed file available to the app. Wow... not exactly twice as fast (half the time) but, that's a big difference. If that happens on every single read operation that could translate into even greater overall performance, especially on very compressible data like run-time libraries, executables, the OS and the applications themselves, even a lot of different types of data like documents, etc."

The decompression is happening in real-time, it's based on how fast the CPU and RAM working together is, but these days that process is so fast it's not even an afterthought, barely a blip in terms of CPU usage -typically under 1% to decompress so I'd seriously doubt anybody would ever notice, compression takes a tiny bit more but again, real-time and only done once as required.

Now, for all those hardcore people wanting the absolute maximum levels of performance, compressing the file system (NTFS) on an SSD can have potentially massive payoffs in terms of performance. Last example:

Say you have a 240MB uncompressed file (yet a candidate for 2:1 compression) and it takes your SSD - which can read data sequentially at 240MB/s, how convenient - 1 second to load it, and you compress the file to 120MB and now that file loads in half the time, you just effectively created a situation that gives the same performance of an SSD that does sequential reads of 480MB/s, give or take a few milliseconds for the decompression process.

Even in spite of the decompression "penalty" you're still going to see a wicked boost in speed - you're not actually getting 480MB/s of sustained read throughput, but the act of loading the compressed file in half the time basically works out to that kind of benefit.

Now translate such a benefit across every file on your hard drive or SSD that's compressible, even to small degrees and you get a boost in read performance.

With hard drives and SSDs you're already taking a tiny penalty on write performance as it is more often than not, so, adding a tiny bit more of a hit to create the compressed version of a file that first and probably only time (unless the file changes frequently), and with all forms of data storage getting faster all the time on top of the CPUs and RAM speeding up as well, there are no negatives to this simple change of adding NTFS compression to your SSD-powered box. Applies to other OSes with file system level compression as well, of course...

So, for your OS/system drive/partition, I highly recommend using NTFS compression - only on the OS/system partition alone, however. Most people that use multiple drives or partitions nowadays do it for better organization, but this type of compression-based "boost" is only going to be effective at loading small compressible files like those that make up the OS itself on the drive, and the applications that people typically use (the actual application code).

If you have tons of mp3 files (or some other format of music content), video files, DVD rips, Blu-ray rips, etc - anything that is NOT highly compressible data, and takes up huge amounts of drive space, this type of file system compression is not going to suddenly double things across the board; in fact it might actually hurt performance.

People get SSDs to make their OS load and function faster and their applications follow the same pattern; don't waste a lot of SSD space on big storage items, it's too inefficient unless you just can't help it. NTFS compression can give you an even better level of performance when it's properly implemented as explained above. It's not doubling the speed of your storage but it is making things load a lot more efficiently (from that time perspective which translates into that effective difference in performance.

That should spell it out succinctly. If for some reason you're not happy with the increased performance on most every single read request your system makes off a compressed file system, you can reverse the entire setup with a few mouse clicks, piece of cake.

Bahamut said:

snip

Click to expand...

Wow, I just tried this out after reading your response and it's crazy. Copy and pasted a couple of folders (500MB to 3GB) to test before and after, ended up just like you said, it's actually writing faster with the compression. All the while saving me 6 or 7GBs on my SSD array.

Thanks!

Must be some insanely compressible content in those folders...

Bahamut said:

Must be some insanely compressible content in those folders...

Click to expand...

Just the basic windows 7 installation stuff and a few Apps, just migrated my Steam folder over too.

Well you do have a rather monstrous machine by pre-Core generation standards so... hey, whatever helps.

I think I'll try enabling compression in win7 w/ssd, I was surprised at how little disk encrytpion slowed down my linux system. Compression should be even more trival plus my win7 system has a much faster cpu.

Bahamut said:

Snip.

Click to expand...

Limpgawd in 26 days huh?

Welcome to the [H], the theory sound and I've done this with my as many folders as I could think of without messing with system files. Saved about 4-5GB of space.

Don't feel any slowdowns. Day 3 enabling NTFS compression on SSDs and things going smoothly. I still won't go so far to say it enhances, but I don't see any decrease either.

Time to decompress the compressed version and feed it to the application = .003 seconds, or less

Click to expand...

That would correspond with 100 / 0.003 = 33GB per second.

That indeed has to be an extremely fast algoritm; even simpler than XOR. ;-)

It's too bad you can't see anything about the I/O framework in Windows; in gstat you can see directly what encryption or compression layers do to your latencies; how latencies are higher when using these GEOM layers.

Ok, ok, I was a little overly generous on the actual decompress time but, you all seem to get the picture.

The decompression time is still going to cause the increase in performance (theoretically, of course, but this kind of stuff is potentially verifiable) since hard drives and even SSDs are still weak links in the entire data processing chain.

Years ago when "drive compression" first came out the processors couldn't really do it nearly fast enough to make a difference. The primary reason it was created was space savings on those tiny < 1GB hard drives, etc.

But these days, given the incredible performance of modern CPUs when it comes to doing that math crunching required at barely a blip of CPU necessary, it's now turned into a down-and-dirty way of improving performance simply because you're reading and writing less actual data to and from the storage devices. That alone should help things all by itself.

Obviously, all this only really matters on high compressible data - that's the rub. Most folks have games with huge data files (typically already compressed), movies or video files of all kinds (already compressed), music (already compressed), that sort of thing. Compressing the system drive with the actual OS installed on it is still the best candidate for this type of tweak and the only recommended drive/partition to enable NTFS compression on. If necessary, compress the Windows folder only and things should still work slightly better.

Hey, if people notice it, and it helps, it's a positive thing, right?

Well the point is: compression does not hurt on todays CPUs because they could compress 300MB/s or so while the drive does 100MB/s max. But the catch is: higher access times. That won't hurt reading or writing large files, but random read access (booting, loading applications, many system disk related tasks) would have latency added which directly impacts on speed.

So compression may not be useful in a situation where an application will not respond until it has that data; such as the firefox profile; i wouldn't compress that. But many more things.

How big the actual impact is? Well likely not all that high; but given the SSD access times are very low, the added latency could negate some of the SSD strongpoints; handling blocking read I/O very well; which relies heavily on access times.

But, if it makes you able to store more things on your SSD, that's of course another argument to throw at the table.

Nice when you bump into people that actually understand this stuff, ain't it?

simple question, how do i compress my ssd?

Right click on the drive, choose Properties, check the box that says "Compress this drive to save disk space" and then click Apply, then OK. It'll pop up a dialogue asking if you want to apply the compression to the root folder only or the entire drive including all subfolders and files (obviously that's what you want and that selection should already be made), so just click OK and then wait for it to finish.

As mentioned above, some items will get compressed well, some won't, but overall it'll work fine. It doesn't automagically "double the space" you have available so, don't go into it expecting that and you'll be fine.

ar09 said:

simple question, how do i compress my ssd?

Click to expand...

find a folder, right click, properties, general tab, advanced, enable compression, apply to all folders and subfolders.
or find a drive, right click, properties, general tab, "enable compression to save space".
I think thats what these guys are talking about.

I never knew this feature existed --Its bloody brilliant! Compression at the file system level, meaning its invisible to the application layer. That's... thats really clever!

How does it work for a bootable partition?

I'm definitely going to do some extensive looking into this. I'll figure out time complexities and see if I can run any legitimate benchmarks for you guys...

right after I figure out how to configure grub in virtualPC (I... it... if anyone wants to give me some foresite I'm lost, Virtual PC spits me out into the ether after I get Ubuntu installed... Ive never seen this before...)

thanks.
i did it.
i now have the same free disk space i had before lol.

I never knew this feature existed --Its bloody brilliant! Compression at the file system level, meaning its invisible to the application layer. That's... thats really clever!

Click to expand...

This has been in windows since at least NT 4.0. Before that even dos had filesystem transparent compression with double space and other compression.

I still do not see this improving SSD speed especially in writes. Reads, I can buy that at some level, although I doubt even a high end i7 can decompress a file at over 200MB/s. I have seen 10s of MB/s decompression in 7zip but never 100MB/s. Although I have not benchmarked the simpler LZ77. I would like to see benchmarks.

I tend to think this will add latency for small file operations (that are 90% of the real world and main reason for an ssd) but improve large file operations on compressible data.

ar09 said:

thanks.
i did it.
i now have the same free disk space i had before lol.

Click to expand...

I compressed my steam folder (~70GB) and I got about 8GB back.

my games might be loading a little bit quicker, I cant tell --this is coming from the guy who calls the upgrade between a single SSD and a RAID-0 SSD array a barely noticable speed upgrade.

I'll write up some code over the week to see if get some actual HDD benchmarks. I'm not sure how I'm going to go about it...

drescherjm said:

I still do not see this improving SSD speed especially in writes. Reads, I can buy that at some level, although I doubt even a high end i7 can decompress a file at over 200MB/s. I have seen 10s of MB/s decompression in 7zip but never 100MB/s. Although I have not benchmarked the simpler LZ77. I would like to see benchmarks.

Click to expand...

its a massively simpler algorithm.

i gave it a try and i gained 2.8gb from 23.6gb

if we use the same calculations as before the compression is 12%

100mb becomes 88mb, if the CPU can decompress at 200mbs and it will take 0.5s to decompress

if the HDD can transfer at 100mbs then 1s becomes 0.88s + 0.5s

does not sound very good to me ,even if the cpu can decompress at 500mbs you are still looking at 0.2s + 0.88s which is more than 1s

I am not sure of the threshold but the compression algorithm may just store files that have that low of compression ratio. Graphics and sound for games will have a low compression ratio because they are already compressed so in a lot of cases these would be just stored. The executable however will compress.

MrWizard6600 said:

...my games might be loading a little bit quicker, I cant tell --this is coming from the guy who calls the upgrade between a single SSD and a RAID-0 SSD array a barely noticable speed upgrade.

Click to expand...

I'm feeling the same way.

It really doesn't seem to make my SSD array any slower, so I've kept the program folders and anything not system related compressed on my desktop and also on my laptop.

On the SSD side of things I don't think that different controllers would have an impact other than raw throughput.

drescherjm said:

I am not sure of the threshold but the compression algorithm may just store files that have that low of compression ratio. Graphics and sound for games will have a low compression ratio because they are already compressed so in a lot of cases these would be just stored. The executable however will compress.

Click to expand...

Yes, NTFS compression will just not compress files that do not compress well.

Zip archivers compress/decompress from/to disk. The decompression we're talking about here decompresses from disk into memory. So it doesn't have to write to disk again.
You read from disk, then decompress into memory where the code is executed, or data is processed.

Looks like there may be some exceptions. http://support.microsoft.com/kb/251186
More specific seems like your profiles shouldn't be compressed. So I just go to advanced properties on my Users folder and uncheck compress contents to save disc space.


I'll try it out for fun. Already on my 3rd install of Win8. So far I LOST FREE SPACE 1.2GB (went from 20gb before to 21.2 after) but haven't rebooted yet. This should be fun

Bahamut said:

theory

Click to expand...

You are 100% correct. The 'speed increase' is even bigger when you have sufficient cpu/ram and a regular spinning hard drive.

In the older days when cpus went from 233mhz to 2ghz seemingly overnight, this was a good trick. Especially b/c people were often using older slow ide drives. The compression greatly increased throughput.

The same thing can happen with encryption! Trucecrypt has been shown, in some cases, to greatly increase hard drive performance. This happens when your cpu has AES instructions, which means it literally take almost 0% cpu to handle the encryption. Truecrypt has a parallel disk scheme and parallel write/read caching. So multicore with AES and a somewhat slow hard drive can see better perf.

Wow, high level thread necromancy, buddy!

Old thread but it is still a relevant topic, my experience says dont do it. Especially with SSD's.. I know readyboost stores data compressed but when just talking about response, compression or anything else will slow things down noticeably. Might be different witha hard drive because of latency issues so when the drive is seeking to another files you have free time to decompress the file. Maybe people still today try to compress their data because of space issues on SSD's not realizing that it is not the case in many situations and maybe moving or deleted files works better.

Hmm well I have a few old (but hardly used) 40GB Intel SSDs which work fine for quick and dirty builds and older laptops that just need light use. I might give it a try with them.