Theory About Hard Drive Speed

[gerbiaNem]

Is it possible that CPU and/or hardware makers are holding back hard drive advancement?

The hard drive is the largest bottleneck on any modern system, it is the reason we need a boot up and shutdown time and the reason it takes so long for programs to start.

Now image a drive (not necessarily a disk drive) appears on the maket with a capacity and pricepoint similar to disk drives with characteristics of pc2100(or even less, doubtful many would notice) ram in [transfer rate/access time/no need to defrag].

The common individual would be amazed at their computer starting and stopping in 5 seconds and programs starting almost instantaneously. If their main computer usage were office and web-browsing, they might think, "Wow, this is a much speed as I could possibly need!"

Now image back at a Dell/CPU company headquarters:
Bill: Our main userbase just disappeared with that drive.

John: Damn, maybe we can still increase ratings?

Bill: Nope, avg Joe feels he's fast enough. Even though
our ratings made him feel like he was getting a big upgrade
for the miniscule software he used in the past.

John: Does this actually mean we have to focus on usability and innovation now?

Note the problem. Could this be the reason why hard drive speeds haven't increased nearly as much as other components even though the technology is obviously there in lower enterprise drives?

 

[DougLite]

I know that Hitachi GST, and possibly others, are working feverishly on solid state storage. The main problems with it are that density and sustained transfer speed are simply not there to make it a viable competitor to the hard drive, not yet. However, anyone can see that the hard drive is doomed, eventually. It's a mechanical device with a logical solid state sucessor, which will dramatically increase reliability and lower power consumption. Anybody who makes solid state storage work, with any sort of density, at a reasonable cost, is going to have the 'hard drive' market all to itself - it's the killer app. However, they've only got 1/3 of those factors so far, and conventional mechanical magnetic still rules the other 2.

 

[boycee]

you can get like 2gb solid state cards, could you possibly install an os on one, and test the performance ? usb2 is pretty quick, but what are the read/write times like ?

Cheers

 

[unhappy_mage]

IMO the biggest bottleneck is the BIOS. It takes several seconds to go through it, and most of the things it does are for DOS or similar era operation systems. Especially with the newer hard disk controllers taking a few seconds to do their thing, it can take 30 seconds to even start booting windows. I have a file server with 3 things that slow it down: ecc, a pci ide controller, and a pci scsi controller. Boot times are about 45 seconds (!) to start booting linux. The Linuxbios project is attempting to address this, but it's still in beta (so to speak, they've got working bioses for several tyan server boards and others).

When Joe Sixpack can turn on his computer and be checking his email within five seconds, then it will be a commodity device. Not until then.

 

[Lazn_Work]

you can get like 2gb solid state cards, could you possibly install an os on one, and test the performance ? usb2 is pretty quick, but what are the read/write times like ?

Cheers

Sure you can, it's slow. Flash is the limiting factor, Solid state will have to either move away from Flash, or Flash will have to improve (both performance, and read write cycles)

==>Lazn

 

[uzor]

Here ya go: http://www.cenatek.com/product_rocketdrive.cfm

I know there are others, but that was the first I found. Probably a little spendy, tho.

 

[Nikola]

Damn, somebody beat me to this in posting about the rocketdrive! Yeah, people have been working on solutions to the HDD problem for years but the problem is finding another viable, cheap, dependable solution. In addition to the rocket drive, there are RAM controller cards you can buy and simply insert RAM and bam! you have a new windows drive, but the problem is that those drives are never very big and unless you are running a whole battery of UPS in the background, eventually that drive is gonna lose power and then your zippy new OS drive is gone.

 

[gerbiaNem]

When the power goes out on that rocket drive...

 

[boycee]

Could you not possibly use one for swap files etc, would that not increase performance?

And when system is off swap file is flushed anyway, ?

 

[Lazn_Work]

Could you not possibly use one for swap files etc, would that not increase performance?

And when system is off swap file is flushed anyway, ?

Yep, but you would get more out of putting all that RAM into main system RAM and having no swap file, or creating a RAMdrive in main RAM to use for swap.

(PCI = 133mb/s while RAM=3200+mb/s)

==>Lazn

 

[boycee]

ram drive how do you do that ? and what sort of size is needed for the swap file ?

ie - total ram = system ram + ram for swap

?

Quite interested in this idea, is the performance increase particually big ?

Cheers

Chris

 

[Herulach]

I dont think you really get what the swap file is, it only ever gets used if you run out of main memory for things. So removing main memory to make a swapfile there is just well, stupid frankly. Ram is cheap, adding more is a good thing, and will stop your pc hitting the swapfile anything like as often.

 

[Lazn_Work]

ram drive how do you do that ? and what sort of size is needed for the swap file ?
ie - total ram = system ram + ram for swap

Quite interested in this idea, is the performance increase particually big ?

Cheers
Chris

Ramdrive: http://www.cyberwizardpit.net/article3.htm

As for swap file size, that is a interesting question. If you have enough RAM you can theoretically disable it, and have no swap file. It is one of those things that people always argue about. Anywho, It got slashdotted today, but it should be back up soon, a good artcle on swap file optimizing is here: http://www.rojakpot.com/default.aspx?location=3&var1=143&var2=0

Really putting your swapfile into a RamDrive would be stupid... You would be better off getting enough RAM to be able to disable your swap file instead.

==>Lazn

 

[gerbiaNem]

The swapfile is not accessed nearly as much when you have more ram. And yes, you can setup a ramdrive, but you everytime you restart windows it would be reset.

 

[J-Mag]

Having your browser's cache stored in a ram drive might help out a tad...

 

[boycee]

oh i see. i, like you said dont know much about swap files, i always assumed they were created ragardless of ram availability, guess not lol. I have 512 Ram atm, and never have less than 100Mb available, so guess I'm doing ok.

Cheers anyway.

Chris

 

[gerbiaNem]

Leaving the entire system kernel in ram really helps.

 

[mikeblas]

Yep, but you would get more out of putting all that RAM into main system RAM and having no swap file, or creating a RAMdrive in main RAM to use for swap.

I'm confused. You're saying you want to start out with a pile of memory, and then reduce the amount of memory you can use for applications and data (which will cause you to swap more), so that you can swap to RAM?

How is that an optimization?

 

[klowngoblin]

solid stage HDD;s exceeding 60Gb are already out, they also cost you 7 of your children and 2 limbs. they are hella fast but just way too expensive, these aren't ramdrives either

also just get 1-1.5GB of ram and DISABLE your swap if you are running XP, you will notice your system go WAY faster. swapping thashes the hell out of your drive.

 

[Tha_Bomb]

well, the way I understand it, under NT you can't "disable" your swap, as you always need 2MB minumum of it. Even the "disable" option on XP simply puts a 2MB swap in your %windir% directory. The way it works is any little-accessed portion of the loaded app(s) goes to swap, and can be "swapped" to RAM if it's needed, but the system's betting on the fact you won't need it. The reason why most of the "swapping" goes very very fast on a system with a lot of RAM is it gets cached as well, and since the system can afford a large cache, it won't get flushed any time soon, and is readily available to be swapped from cache to RAM.

Don't believe me? Watch the RAM usage of say IE if you minimize it. Then restore it. The RAM usage will change, since more will be less likely to be accessed if it's minimized, than once restored it comes back to up as more likely to be used and is swapped back to RAM. This is the main advantage of the NT memory manager vs. 9x/DOS where swap isn't touched until all the RAM is used, and even then, inefficiently. That and the fact that it doesn't know when to limit or how to free disk cache from RAM.

That's why it would be beneficial, even if minimally to swap to a RAM disk. And system stability would be greatly improved from no "ghosting" effect a static paging file would have on a magnetic medium, like a hard drive, during R/W cycles, since we all know even "wiped" info on a hard drive is really still there. I noticed a great improvement in stability on my server when I enabled the "erase swap on shutdown" or somesuch switch in the Registry.

 

[mikeblas]

also just get 1-1.5GB of ram and DISABLE your swap if you are running XP, you will notice your system go WAY faster. swapping thashes the hell out of your drive.

I think that you'll have a very hard time proving what you're saying. I certainly can't reproduce your results, and none of my machines have less than 2 gigs of physical memory.

If you have a method for benchmarking the claim that you're making such that other people can actually reproduce it and measure the results accurately, please post your instructions over in this thread. The bar is set pretty high -- it's not a very easy thing to measure, and your claim of "WAY faster" is very dubious.

 

[mikeblas]

The reason why most of the "swapping" goes very very fast on a system with a lot of RAM is it gets cached as well, and since the system can afford a large cache, it won't get flushed any time soon, and is readily available to be swapped from cache to RAM.

Are you perfectly certain the file system cache is used for the page file? I don't think it does.

OTOH, paging operations also includes pages that are not stored in the page file, such as read-only executable pages from images. The file system certainly does cache these accesses.

Don't believe me?

I'm sorry, but I don't. What's the name of this two megabyte file?

Watch the RAM usage of say IE if you minimize it. Then restore it. The RAM usage will change, since more will be less likely to be accessed if it's minimized, than once restored it comes back to up as more likely to be used and is swapped back to RAM.

The working set of the observed application changes because Windows calls SetProcessWorkingSetSize() on the process as it is being minimized. That's a documented fact.

The size increases when the application is restored because that memory actually has been accessed. When you restore IE, it touches a lot of memory in order to render the page again. This isn't done preemptively because the access is "more likely". The page faults to bring that data and code back into memory are done reactively, as the pages are touched.

Repeat your experiment watching the "Page Faults" or "Delta Page Faults" counts for the process and you'll see that the memory is faulted back in.

But you've offered this anecdote as proof about this two-megabyte file that helps with frequently-accessed portions of files. I don't understand what the SetProcessSetWorkingSize() behaviour proves anything about this two-megabye, always-required swap file you're talking about.

This is the main advantage of the NT memory manager vs. 9x/DOS where swap isn't touched until all the RAM is used, and even then, inefficiently. That and the fact that it doesn't know when to limit or how to free disk cache from RAM.

What was specifically inefficient about Windows 9X for swap file access?

Meanwhile, Windows 9X uses its page file before running out of RAM. Certain APIs require it: creating a PF-backed memory-mapped file, for example.

That's why it would be beneficial, even if minimally to swap to a RAM disk.

Sorry to be so dense, but could you make it a little clearer, please? Why do you think it's even minimally beneficial to reduce system memory in favor of an in-memory volume that holds the page file? Why wouldn't it be better to maximize the system file cache, or make as much memory as possible directly available to applications without forcing them to do any swapping at all?

And system stability would be greatly improved from no "ghosting" effect a static paging file would have on a magnetic medium, like a hard drive, during R/W cycles, since we all know even "wiped" info on a hard drive is really still there. I noticed a great improvement in stability on my server when I enabled the "erase swap on shutdown" or somesuch switch in the Registry.

You've really lost me on this last one. You say that there's a "great improvement in stability" when you "ClearPageFileAtShutdown". What is a "great improvement"? Does that mean you've gone from crashing all the time to not crashing at all? What did you specifically observe after changing the setting? What is your explanation for those observations?

 

[gerbiaNem]

Don't be so harsh on the guy, he's just trying to make a point..
And making a ram drive to page would be a bad idea, it would be more beneficial to just disable paging with that much ram.

 

[Vertigo Acid]

Don't be so harsh on the guy, he's just trying to make a point..

Heh, but Mikeblas's point is that he is 100% wrong; this has been rehashed over and over in OSes, just like the "disabling services" debate.

 

[mikeblas]

Don't be so harsh on the guy, he's just trying to make a point..

I'm sorry if you found my note "harsh"; I'm not sure what part of it you didn't like.

Thing is, I'm trying to make a point or two, too. This is a forum, and that's what it's for, after all!

One of my ponts is very subtle. Lots of people post on subjects, especially complicated ones like this, and give advice. They state what they think are facts. It's unfortunate that they do so because they often don't know what they're talking about.

I'm not sure Tha_Bomb is "100% wrong", but I don't think I understand most of his post. It doesn't match with the things I know are true, and it the things I don't understand can't fit with what I'm sure is correct and what can be empirically demonstrated.

I'd like to learn why that is. Maybe there's something I think I know that's wrong, or maybe there's some alternative explanation to what's observable.

it would be more beneficial to just disable paging with that much ram.

You can't disable paging altogether, so you must mean to remove the paging file. If I do that, how much RAM must I have to see benefits? That is, how much is "that much"?

And what's the benefit you would expect me to see when I do that? Certainly, my system won't be able to load more data or programs into memory.

Since you know that it is more beneficial to disable the page file, you must know a good way to measure the performance increase. How did you measure that increase? Can you explain your methodology? Did you not measure it yourself and read about measurments that someone else made? Can you provide links to the description of those tests, please?

 

[gerbiaNem]

Ok Mr. no common sense. When a computer (thing that usually looks like a box with grooves, I can't show you any tests to prove this however) with a Windows operating system uses a paging file, the hard drive is used in something called paging to lower the amount of ram used.

What I said in my post above was that instead of using a virtual hard drive on his ram and creating unnecessary transfers between the virtual hard drive and ram. It would be smarter to just disable paging (AKA remove the paging file from the hard drive altogether) so that no swapping between different portions of the ram is done.

I wasn't inferring that there were any performance gains through disabling the paging file with a large enough amount of ram (4GB or whatever), since I don't know there are. What I said, and you would have known had you read through my post and the posts beforehand, was that disabling the paging file would be a performance increase over using ram as virtual hard drive containing the paging/swap file.

And for the record, the things "you are sure are correct" might not be. So go be sarcastic somewhere else.

 

[mikeblas]

It would be smarter to just disable paging (AKA remove the paging file from the hard drive altogether)

Paging still happens when there's no paging file because Windows can use files other than the paging file to back reserved memory. Executable files, for example, directly back code and resources that might get swapped out, or that are not yet loaded but are mapped.

What I said in my post above was that instead of using a virtual hard drive on his ram and creating unnecessary transfers between the virtual hard drive and ram. It would be smarter to just disable paging (AKA remove the paging file from the hard drive altogether) so that no swapping between different portions of the ram is done.

I know. I did indeed read that.

What I'm wondering is why you didn't recommend keeping the page file isntead of turning it off entirely. I'm sorry if I misread your note, but to me that implies you think disabling the page file results in better performance than leaving the page file settings at their defaults.

 

[gerbiaNem]

I was implying that if he's already going to use ram as a drive, he might as well do it differently. And I think paging is great, it saves system memory for processes that are more used.

I did find that keeping the system kernel in memory gave a measurable performance boost to the responsiveness of the system. The start menu never hitches anymore and system settings pop up faster.

 

[mikeblas]

I was implying that if he's already going to use ram as a drive, he might as well do it differently.

Thanks for the clarification!

 

[Ruffy]

how did this turn into a page file debate. Back on topic

No I dont think HD makers are purposely keeping us for fast drives.
I DO think that they focuse on Storage Density over Speed though. It's just easier to make a drive with 300gb then it is to create a 20gb drive with a 1ms seek time and 400mb transfer rate.

Alot of companies are working on solid state solutions. And there are many currently out there. Besides ramdrive based ones(rocketdrive) they are INSANELY expensive.

 

[gerbiaNem]

But is a ramdrive really where the future is headed? It doesn't seem like some people want to tolerate having to backup their data to external hard drives just in case the battery dies.

Does anyone know of any technologies trying to make volatile storage.. well, less volatile?

 

[Lazn_Work]

But is a ramdrive really where the future is headed? It doesn't seem like some people want to tolerate having to backup their data to external hard drives just in case the battery dies.

Does anyone know of any technologies trying to make volatile storage.. well, less volatile?

IBM has been working on MRAM for years now.. it might pay off some day.

==>Lazn

 

[mikeblas]

No I dont think HD makers are purposely keeping us for fast drives.

Of course they're not. If a company had a substantial advancement available, why wouldn't they productize it? Particularly in such a competitive market?