![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Paging file
- Thread starter Glyphic
- Start date
Phoenix86 said:
Why do you think you can't make it happen on command?
If you wrote a program -- your own, yourself -- that allocated a ton of memory and started touching it, wouldn't that cause the paging that you're after?
GreNME said:
I'm going the other way; I'm forcing a pages to be swapped out, not swapped in. Maybe you want to write two applications so you have full control over everything.
1) Run application #1.
2) Run application #2.
Application #1 allocates a ton of memory. It touches each page to do the zero demand page fault, and has committed the memory so that it really, pyhscially exists. That is, it calls VirtualAlloc() with MEM_COMMIT and MEM_PHYSICAL. It allocates, say, all of virtual space the machine claims to have available. Then, it sleeps until you ask it to free the memory and exit.
But you don't, not right away. Instead, you load and run application #2, and it tries to allocate a ton of memory. It's calling VirtualAlloc() for committed memory, too, and touching all the pages it gets to immediately zero-demand fault them.
Is the problem with this approach simply that it won't work when there's no page file available? That you'll run out of pyhsical memory before (or, just when) you run out of virtual memory?
Hmm. If you didn't use MEM_PHYSICAL, what would you prove?
Just another voice to agree that Windows XP does not need a pagefile, and that virtual memory and pagefiles are not the same thing, and of course that there are a lot of confusing information out there.
I do recall hearing more than one instance of anecdotal evidence that there are certain applications (or have been in the past) that experience instability without a pagefile, but I have no evidence to present. Those stories could be anything from myth to math. I wouldn't be one to argue the point if someone were to say "Adobe Premier Pro is unstable without a pagefile even if you have 4GB of RAM." I'd cut the folks in the "must have a pagefile" camp some slack based on that slight possibility, but reiterate that XP alone doesn't have this requirement and there's no logical reason why an application 'should' care since it's not the application's job to manage virtual memory.
Just my 1.5 cents.
I do recall hearing more than one instance of anecdotal evidence that there are certain applications (or have been in the past) that experience instability without a pagefile, but I have no evidence to present. Those stories could be anything from myth to math. I wouldn't be one to argue the point if someone were to say "Adobe Premier Pro is unstable without a pagefile even if you have 4GB of RAM." I'd cut the folks in the "must have a pagefile" camp some slack based on that slight possibility, but reiterate that XP alone doesn't have this requirement and there's no logical reason why an application 'should' care since it's not the application's job to manage virtual memory.
Just my 1.5 cents.
GreNME said:
Performance counters might work.
Or, you can use the tracing APIs; StartTrace() with EVENT_TRACE_FLAG_MEMORY_PAGE_FAULTS, then watch for EVENT_TRACE_TYPE_MM_HPF ... assuming it was physical page faults (and not any other kind) you were interested. Maybe you want EVEN_TRACE_TYPE_MM_COW, too.
GreNME said:
Force the process's what? To do what? Huh?
rcolbert said:
An application can lock pages into physical memory. In this way, it is "managing virtual memory". It can also, if written to do so, ask for memory in very specific ways that put strain on the memory manager.
Say you're playing an MP3 file. The codec has decompressed the MP3 into playable waveform data. It would be pretty crummy if the OS decided to swap those pages out just before they were played, right? So the application (or the driver) will lock them into physical memory.
Need another reason? Read the docs for the CreateFile() API, for example. If you're after performance, you're going to use FILE_FLAG_NO_BUFFERING. To get that to work the fastest, on all disk subsystems with all drivers, you need to make sure the memory you show the WriteFile() API is aligned.
How do you align it? One way is to request the allocation so that it is aligned at a specific logical address. This isn't "management", but it's darn close -- you're coaxing the manager to do something in a very intentional way.
It's easy to provide code that falls over on a machine that has no page file: just ask for all the physical memory, or ask for physical memory in a pattern that prevents other logical memory requests from being satisfiable.
With this information, I hope that you see there are reasons, and good ones -- quite common ones, why an application does have a reason to care about carefully managing, or at the very least strongly influencing, virtual memory.
mikeblas said:
'Should' being the key word. Most of what you are saying are examples of why memory would NOT be swapped to the pagefile. In those cases, the application again shouldn't care that no pagefile exists. What you are describing with application calls isn't the same as the job the VMM does, but I see your point. The examples where the application would fail on the other hand are cases where applications probably "should" not be written to do such things, but you're right in that it's fairly easy to conceive of code that would do what you say. In all cases it is not really a Windows issue so much as it is an application specific issue, and that was the main distinction that I'm trying to draw.
rcolbert said:
I don't agree. Virtual Memory is there to be used. Without using virtual memory, you're not able to keep as much information in memory. If you can't keep as much information in memory, you either can't do the job you wanted to do (because it's too big) or you end up partitioning your working set or virtualizing your working set yourself.
If your application needs three gigabytes of memory and can't get it because you only have one gigabyte and no page file, then it actually must fail. Otheriwse, it can't run!
A robust application might try to do 3 GB of work 1 GB at a time. Maybe its is coded to do so, maybe it isn't -- partitioning the work and merging the results may not be practical depending on the task. It's certainly harder than not worrying about partitioning the work, and that's why lots of programs don't do it.
VirtualLock, sure. Aligning page allocations? Nope -- that's got nothing to do with swapping, but it does end up meaning you're looking for a particular hole in the virtual address space and that probably causes swapping for adjacent ranges.
Is one out of two really "most"?
Let's tip the scales to one out of three. Say you have a couple of programs (or threads) which want to commincate with eachother. They have a lot of data to exchange, so they use a memory-mapped file. Since there's no reason to have the file actually stored on disk, they call CreateFileMapping() and pass INVALID_HANDLE_VALUE for the backing file. This causes the OS to back the memory mappe file with the page file.
But you have no page file. So the memory mapped file can't be created. Certainly, you're not telling us that applications shouldn't try to perform IPC through PF-backed MMFs, are you?
I'll always use a page file. My reasons are pretty simple:
First, say something goes wrong with my machine. There are too many processes running, or some combination of processes are using up all the memory. I want to fire up a diagnostic tool to figure out what happend, or even just start task manager and kill apps.
Without a pagefile, I can't. When I want to start new processes, I'll need to swap pages from the existing processes out. I won't be able to; I'll just get an out of memory error. I've lost control of my machine at this point.
Next, you end up using more physical memory when there's no page file. The reason is that memory might be reserved but not committed. Every single thread does this! When the thread starts, the operating system reservers some amount of memory for the stack on that thread. The thread initializes and starts using its stack space. It's quite uncommon to need a megabyte of stack space, and the thread is probaly only using a few dozen kilobytes, at most, of that space.
The OS only commits pages from the reserve for the stack space when the pages are actually needed. Since the majority of those stack pages are rarely needed, why back them with pyhsical memory?
Using no page file guarantees you'll use physical memory for reserved but uncommitted memory. (As far as I know, anyway. "Guarantee" is a pretty strong word, so maybe I shouldn't use it in this case. But I really can't think of a time when you'd end up not reserving physical memory for uncommitted reserved memory when you don't have a page file).
If you can totally predict the workload your machine sees, then I guess you can get away without running a page file. Measuring the difference between running a page file and not is pretty challening. I think you could make a synthetic benchmark to show that either choice was faster because you can write the benchmark to favor what you know is happening.
In general use, I'd really rather let my machine swap out data that's never being used in order to free physical memory for more important things -- like the file system cache.
Ahh, someone found the right thread to discuss PFs in, as opposed to 50 1-2 page threads. This one has most of the best ideas, just got to dig a bit through the longer thread.
Anyways, a bit has happened since this thread was active. I figured out the best way to monitor the PF. This would be pretty tricky for any other file other than the PF, however perfmon has the perfect counters for this. First here's a good link explaining the counters in perfmon.
Counters by Object.
Try this: Open Perfmon and right click the graph-add counters. Select the memory object, then add pages per second.
Pages/sec Shows the rate, in incidents per second, at which pages were read from or written to disk to resolve hard page faults. This counter is a primary indicator for the kinds of faults that cause system-wide delays. It is the sum of Pages Input/sec and Pages Output/sec. It is counted in numbers of pages, so it can be directly compared to other counts of pages such as Page Faults/sec. It includes pages retrieved to satisfy faults in the file system cache (usually requested by applications) and noncached mapped memory files.
This is both reads and writes to disk (not disk+RAM) you can further break it down with Page Reads/sec (pagefile reads) and Page Writes/sec (pagefile writes).
I monitored a couple of systems with 1GB and 512MB RAM and various memory usages. So what I found was systems with plenty of RAM would not *write* to the PF once booted. They would read from it, fairly regularly (I assumed this was system cache, wasn't real sure). Reads don't kill performance like read/writes in a swap. My rough conclusion is that systems with *enough* memory simply don't write to the PF, so disabling it wouldn't benefit much (this explains why disabling it doesn't yeild measurable performace).
This is kosher up to a point. When the commit charge neared (but never surpassed) physical RAM, the system would page quite a bit (say 440 used out of 512). Now if I know I'm not going over that limit, and do see performance hits, you could disable to PF and see gains. However, your so close to your total limit, you also likely to see "out of memory" errors. At this point it becomes dual edged, pick your poision.
In summary I don't think disabling the PF helps many people unless they happen to fall in that area where system usage is so very close to total RAM, yet they never exceed total RAM in normal usage. That's such a specific case, I barely think it's worth mentioning unless we find the threashold for paging is wider than I thought (which I haven't seen in practice).
Does this mean some people don't see benefits from disabling the PF? No. Does it mean everyone with plenty of RAM should disable the PF? Not from what I'm seeing so far.
Anyways, a bit has happened since this thread was active. I figured out the best way to monitor the PF. This would be pretty tricky for any other file other than the PF, however perfmon has the perfect counters for this. First here's a good link explaining the counters in perfmon.
Counters by Object.
Try this: Open Perfmon and right click the graph-add counters. Select the memory object, then add pages per second.
Pages/sec Shows the rate, in incidents per second, at which pages were read from or written to disk to resolve hard page faults. This counter is a primary indicator for the kinds of faults that cause system-wide delays. It is the sum of Pages Input/sec and Pages Output/sec. It is counted in numbers of pages, so it can be directly compared to other counts of pages such as Page Faults/sec. It includes pages retrieved to satisfy faults in the file system cache (usually requested by applications) and noncached mapped memory files.
This is both reads and writes to disk (not disk+RAM) you can further break it down with Page Reads/sec (pagefile reads) and Page Writes/sec (pagefile writes).
I monitored a couple of systems with 1GB and 512MB RAM and various memory usages. So what I found was systems with plenty of RAM would not *write* to the PF once booted. They would read from it, fairly regularly (I assumed this was system cache, wasn't real sure). Reads don't kill performance like read/writes in a swap. My rough conclusion is that systems with *enough* memory simply don't write to the PF, so disabling it wouldn't benefit much (this explains why disabling it doesn't yeild measurable performace).
This is kosher up to a point. When the commit charge neared (but never surpassed) physical RAM, the system would page quite a bit (say 440 used out of 512). Now if I know I'm not going over that limit, and do see performance hits, you could disable to PF and see gains. However, your so close to your total limit, you also likely to see "out of memory" errors. At this point it becomes dual edged, pick your poision.
In summary I don't think disabling the PF helps many people unless they happen to fall in that area where system usage is so very close to total RAM, yet they never exceed total RAM in normal usage. That's such a specific case, I barely think it's worth mentioning unless we find the threashold for paging is wider than I thought (which I haven't seen in practice).
Does this mean some people don't see benefits from disabling the PF? No. Does it mean everyone with plenty of RAM should disable the PF? Not from what I'm seeing so far.
It's most ironic that I have had to spend a bit of time this past week working on what appears to be a memory issue with a CRM (Onyx) application server. My one conclusion: 3GB of physical RAM is a lot better than 2GB of physical RAM if you have one process that wants to grab all the memory it possibly can.
I'm not making any argument here. Just an anecdotal coincidence.
I'm not making any argument here. Just an anecdotal coincidence.
Phoenix86 said:
PerfMon has been brought up a few times in this thread. I'm glad you've discovered it, as well.
Thing is, the counter you mention monitors paging, not the page file. Paging can happen against files other than the paging file.
You don't describe what you did to the systems specifically to stimulate them while monitoring their paging, or what "various memory usages" could possibly mean. But I would expect the in-page operations you're seeing are pages faulting in from executable images.
When you ask Windows to load a program, it maps the image to memory. This means that the size of the image is reserved in memory, then the reserved memory is backed by the executable image. Windows loads the page with the image's entry point on it, then resolves all the fixups and so on. This leaves an amount of memory equal to the size of the image reserved, but with only a few pages actually loaded from the image and committed in memory.
Windows then calls the executable's entry point. As the code inside the image starts running, it ends up touching pages for read-only data, resources, and code in that image. The first time each page is referenced, it will cause a page fault. The OS will load the page from the image, and that action causes a read fault to show up on the performance counter even though the read isn't from the page file.
There are interesting optimziations to minimize program startup time that are available because of this behaviour.
Phoenix86 said:
I would agree. I think another way to think about it is this: people who are saying that disabling the page file when there's lots of RAM are assuming that Windows swaps to the page file when it doesn't actually need to. Disabling the page file causes Windows to be unable to sawp, eliminating the preceived unnecessary swaps. Thing is, there really aren't any unnecessary page file hits, so the suggestion of disabling the PF to avoid those hits is a fallacy.
The counter is limited to hard faults, and until someone shows me another file that recieves pages, that describes only pagefile.sys.mikeblas said:
I have heard the "claim" that windows will always use a page file, even if pagefile is disabled. I don't recongonize anything you have said about this, so here's my standard response.
I realize there are other files involved in paging (process), but I'm not aware of any other files that store paged data. What are the other file names used to store paged data? Please provide some link (frequently claimed, never shown) that we could all agree on, prefferable from MS.
Well I wasn't giving specifics (numbers) for reasons. I quickly found what I was looking for in the data I was seeing was countering some of what I have seen. So quick I think I could demonstrate it fairly quickly to anyone who wanted to see the same thing with these counters.
Basically I monitored the counters under different loads. At low to mid-high memory loads (commit charge vs. RAM) the PF was not swapping data to-from RAM at all. This kinda killed my opinion of no PF= good tweak for any system with plenty of RAM.
At higher loads (where commit charge does not exceed RAM, that of course would require a PF) the system would page, in my opinion, too much. This is what I was seeing when on my home machine when I was gaming. I had 768MB RAM, and was running ~700MB commit charge and saw quite a bit of paging (effecting performance quite a bit). In that case I benefited from no PF. I forced all PF reads and writes to happen in RAM, which was good, it stopped the system from paging. However, it only left me a small amount of RAM, if my requirements changed I'd be over my limit.
I see the same thing on my work laptop, it has 512MB RAM and I run constantly ~400MB commit charge. This causes a fair amount of paging when I'm alt-tabbing from app to app.
Conclusion? Windows handles paging very well. Only when you seem to be running out of memory will it page to disk, as it should. If someone was in that middle ground, they might see a benefit from disabling the PF, but only when they are using nearly all their RAM, and they have no room to grow w/o getting an "out of memory" error.
That's basicaly my findings, except in certian cases where it's really not worth it otherwise.
Phoenix86 said:
Any disk file you use to back a memory mapped file will receive pages. That can be any file a developer would like; see the CreateFileMapping() and API documentation. The hFile parameter to this API gives the handle to a file you've created and opened; that file can receive paged data.
Mapped files are created by applications to access data from disk that's anticiapted to be larger than memory, so this paging is initiated at the application's request.
However, Windows maps files for executables that are loading. It internally uses the same CreateFileMapping() API as it opens the process. As it references pages in the image, they're loaded. It references a few pages as it gets the application's address space setup, then leaves the rest of the pages to fault in as the application references them.
Hard Faults can be reads for read-only pages, and those are the pages I'm talking about when I'm explaining how executable loading works.
There are plenty of othe types of page faults: Zero Demand Faults are probably the next most common. Soft page faults are page faults that get resolved without going to disk by picking a page off one of the transition lists. There's two or three less common, other types.
Phoenix86 said:
I don't know what you mean by "recognize" here: refuse to believe it, or haven't heard it before?
I don't think I have a link to any such documentation. You can read about how processes are loaded and executed in Chapter 6 of Windows Internals by Russinovich and Solomon. There, the process of loading a new executable image and setting up its address space is discussed in great detail, though the authors repatedly use the term "mapping" without discussing what they really mean. It's not until Chapter 7, where memory mapped files and their role in shared memory is discussed do the authors explcitily explain that "the image loader uses section objects" which are created by the CreateFileMapping() API "to map executable images, DLLs, and device drivers into memor, and the cache manager uses them to access data in cached files."
Please understand that I'm not claiming Windows will always use a page file, even if the user has disabled the page file. To the contrary, in this thread, I've pointed out that Windows is unable to swap out changed pages if you don't have a swap file. Executable images are different because they involve read-only data. When read-only is loaded from a memory-mapped file, and the page is never changed as is usually the case with the data in an executable file, the memory is not backed by the page file. It is backed by the executable file itself.
The page file is normally used for modified pages. If I allocate some memory and touch it, I cause a zero demand fault. Then, I've modified the page by writing some data to it. If that page needs to get swapped out, it is written to the page file. Windows can't know what data is there.
On the other hand, executable data isn't modifiable. My program loads its bits from some .EXE file. That data comes into memory and isn't changed. If the page isn't dirty, and Windows knows that it came from the executable file, there's no reason to ever copy it to the page file. It's just discarded and not written out. If it is ever needed again, it can come from the executable image.
This is why the executable file usually can't be deleted when the process is running. And it's also the reason why the linker has the /SWAPRUN option. Say you put an executable on a CD. The user starts the executable, and Windows does the mapping that I've described here. But all the pages aren't in memory yet. The user then ejects the CD and continues exercising the application. The application goes to execute code or find image data that it doesn't already have, and tries to read from the CD.
But the file can't be read because the disk has been ejected and isn't readable. The application fails because the page fault to bring that part of the image in fails.
Similarly, an application image loaded from a network share might be interrupted when the network goes down. Even if the network stays up, it would have been more efficient to copy the image to the page file, then let the page file locally back the memory reserved by the loader when the processes demands it, than to go over the network and to the remote drive for the pages as they're needed.
For completeness, I'll try to answer your question: the only files I think Windows uses for dynamic paging (that is, writing and reading pages) are either files you've used to back memory-mapped files your application creates, or the files listed at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagingFiles.
Windows will read pages, but never write to, executable images. This includes DLLs, EXEs, and anything else -- drivers, for instance -- that it is loading as an executable file.
Phoenix86 said:
Unless you have very intimate knowledge of the access patterns for the pages invovled, I'm not sure how you could make a judgement about the frequency or quantity of pages being swapped. Maybe you can use the system debugger or PFMON to monitor the paging and get some insight into what's happening. If the game is randomly accessing a wide range of memory (over several pages) then Windows will naturally be required to try and swap data in and out to satisfy the requests.
Windows doesn't move pages to and from the swap file just because it thinks it might be a good idea -- it does so in order to try to satisfy the demands of the application.
As an Amazon Associate, HardForum may earn from qualifying purchases.
Quick post to answer you Qs.
I don't recogonize anything you have said about the issue in the past, sorry your name just hasn't stuck in my head. I need to re-read your posts in the thread. I have seen some pretty wild claims about they way the OS works, and wasn't sure exactlly what you were claiming...
I think it's "too much" because I still have available RAM, yet the system is paging. If it's only satisfying my demands why doesn't it always use RAM space first? It should only page to disk when more memory is required than available in RAM, but that's not the case.
I'll go through the post in earnest when time permits.
I don't recogonize anything you have said about the issue in the past, sorry your name just hasn't stuck in my head. I need to re-read your posts in the thread. I have seen some pretty wild claims about they way the OS works, and wasn't sure exactlly what you were claiming...
I think it's "too much" because I still have available RAM, yet the system is paging. If it's only satisfying my demands why doesn't it always use RAM space first? It should only page to disk when more memory is required than available in RAM, but that's not the case.
I'll go through the post in earnest when time permits.