A seemingly big point of differentiation between Intel CPUs (i5 to i7 to workstation CPUs) is cache size. All else being equal, what is the effect of cache on speed? I'm primarily interested in scientific and statistical computing, not gaming. So, tasks like simulations, fitting regressions, etc. Thank you.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Effect of Cache size?
- Thread starter DrinkTea
- Start date
http://arstechnica.com/gadgets/2002/07/caching/
Basically, you have different levels of memory. You can think of it as a pyramid. At the very top (also the type with the smallest amount of memory) are the CPU registers. These are the fastest. Next you have your various levels of in CPU cache (level 1, level 2, perhaps level 3). This memory sits on the actual CPU, so if the CPU requires something from memory quickly, if it can get it from the cache, it's going to process faster.
Outside of the CPU, you have what is normally referred to as RAM. This sits on a bus connected to the CPU. Various RAM architectures require a certain number of CPU clocks to actually deliver data. They are much slower than CPU level cache.
Next you have media memory like hard-drives (SSD, magnetic, etc.) If you're depending on using these as RAM, you're going to have a bad time.
For scientific applications, it never hurts to have as much CPU cache as possible. Some programs are better than others at utilizing this cache.
A lot of the time, a CPU sits around waiting for data from memory if it is outside of its cache. If the CPU is waiting for that data, it can't compute and thus a program will run slower.
EDIT: Increasing the CPU cache usually brings exponentially lower returns. For instance, moving from 512k of cache to 1 megabyte might improve performance by 30%. Moving from 1 megabyte of cache to 2 megabytes might further improve performance by 10%. Eventually you get to the point where adding more cache is not worth the cost that it would incur by taking up space on the actual CPU die.
Basically, you have different levels of memory. You can think of it as a pyramid. At the very top (also the type with the smallest amount of memory) are the CPU registers. These are the fastest. Next you have your various levels of in CPU cache (level 1, level 2, perhaps level 3). This memory sits on the actual CPU, so if the CPU requires something from memory quickly, if it can get it from the cache, it's going to process faster.
Outside of the CPU, you have what is normally referred to as RAM. This sits on a bus connected to the CPU. Various RAM architectures require a certain number of CPU clocks to actually deliver data. They are much slower than CPU level cache.
Next you have media memory like hard-drives (SSD, magnetic, etc.) If you're depending on using these as RAM, you're going to have a bad time.
For scientific applications, it never hurts to have as much CPU cache as possible. Some programs are better than others at utilizing this cache.
A lot of the time, a CPU sits around waiting for data from memory if it is outside of its cache. If the CPU is waiting for that data, it can't compute and thus a program will run slower.
EDIT: Increasing the CPU cache usually brings exponentially lower returns. For instance, moving from 512k of cache to 1 megabyte might improve performance by 30%. Moving from 1 megabyte of cache to 2 megabytes might further improve performance by 10%. Eventually you get to the point where adding more cache is not worth the cost that it would incur by taking up space on the actual CPU die.
Why do we have cache on the CPU? Because it is faster to access than RAM
Why do we not put huge amounts of memory on the CPU? Because the bigger the cache, the slower it becomes*
You have to figure out what is slower, having a bigger cache or hitting ram? The high cache ones target workstations because their benchmark show that it is faster for applications working with a large amount of data, and the hit it takes in being slower is still better than always having constant cache misses and having to access the slower RAM.
So if you're working with big data, big cache helps more.
* the bit lines become longer with more mosfets, thereby increasing the line capacitance, taking longer for them to charge initially and for their value to change before the sense amps are able to detect the value being read
Why do we not put huge amounts of memory on the CPU? Because the bigger the cache, the slower it becomes*
You have to figure out what is slower, having a bigger cache or hitting ram? The high cache ones target workstations because their benchmark show that it is faster for applications working with a large amount of data, and the hit it takes in being slower is still better than always having constant cache misses and having to access the slower RAM.
So if you're working with big data, big cache helps more.
* the bit lines become longer with more mosfets, thereby increasing the line capacitance, taking longer for them to charge initially and for their value to change before the sense amps are able to detect the value being read
But are there any benchmarks to quantify the effect? For example, if I'm doing anything (SVD, regression, whatever) on a large matrix, how much does an increase in cache size buy me, all else held constant? I'm guessing it would be more in situations where you can perform some computations on part of your data and be done rather than a long sequence of operations where the next result depends on new data AND the previous result. But, has anyone ever benchmarked this?
- Joined
- Sep 7, 2011
- Messages
- 10,219
The best thing to test with this is to get an i5 3570 and an i7 3770, clock them both to 3.5ghz, disable HT on the i7 and test in a slew of benches. So long as the systems are identical besides the CPU, you'll have a quantifiable amount of performance difference with cache sizes.
It depends on the type of application and how well it uses the cache to hide main memory latency. Something with purely random access patterns won't benefit from larger cache sizes by much. Applications with working data sets that fit within the caches will do much better.
Your example suggests a type of application which is often done via SIMD if the CPU is targeted. There are both hardware and software prefetch mechanisms that can be used to ensure that main memory accesses can be reduced. It's also worth noting that the CPU doesn't deal in 1,2,4,8 or other data byte sizes: memory is moved into cache a whole cache line at a time, which is 64-bytes on current Intel processors. So if you're accessing sequential data, that works out well after the first access.
There's an art to optimization, and the layout of the data structures, and even preloading cache lines can have a significant effect on application performance.
The basic answer for applications which can benefit from the caches is that it depends on the what's holding back CPU resource utilization. If it's caused by flushing and reloading caches often because the ones in use are too small, a larger cache will help. If it's compute bound, the cache size may have a smaller significance.
Your example suggests a type of application which is often done via SIMD if the CPU is targeted. There are both hardware and software prefetch mechanisms that can be used to ensure that main memory accesses can be reduced. It's also worth noting that the CPU doesn't deal in 1,2,4,8 or other data byte sizes: memory is moved into cache a whole cache line at a time, which is 64-bytes on current Intel processors. So if you're accessing sequential data, that works out well after the first access.
There's an art to optimization, and the layout of the data structures, and even preloading cache lines can have a significant effect on application performance.
The basic answer for applications which can benefit from the caches is that it depends on the what's holding back CPU resource utilization. If it's caused by flushing and reloading caches often because the ones in use are too small, a larger cache will help. If it's compute bound, the cache size may have a smaller significance.