Help me break the laws of Kernel VMA physics

C

cmpsr2000

Weaksauce
Joined
Mar 6, 2005
Messages
72
First off a quick hello to everyone, I've been [H] since the celleron 300a, just never managed to post on the forums until now.

I have a problem that I think even the l33t hax0rz among us may not even be able to solve.

All that follows is on a p4 2.4c machine with 2gb of ram running windows XP pro SP2.

I have a program that runs on the kernel level, using up those memory addresses. I need to *expand* the kernel memory address area in order to allow the program to envelope all the physical ram I have in the machine(2gb currently) minus what's needed for the OS (500meg or so). This is opposite of what 4GT does. Rather than reducing the addressable memory range to 1GB I need to increase it to about 2.5gb at the moment but as close to 4gb as possible for the future..

I know, it's impossible, but I thought I'd see if anyone here can come up with an "outside the box" solution to it.

good luck :eek:
 
This sound kind of relevant, even if it only mentions 2000 and 2003, not XP. I assume you know about it and it really doesn't work in XP. :)

You might perhaps consider moving to a 64-bit OS, if possible. (Though limiting yourself to windows 64-bit will right now greatly reduce the number of potential users.)
 
http://www.tascamgiga.com is the program.

I've tried /PAE already, unfortunatly it assigns the extra memory addresses to the user page area rather than the kernel, just like /3gb, and even then only on demand. I've tweaked out most of the session memory related registry entires(pagedpoolsize, etc) Which has allowed me access to a few hundred more MB, but there is still about 500mb phyiscal memory left in the system not being used by anything, there's just not enough virtual addresses to get to it :(

I've looked at switching to a 64bit OS, but gigastudio doesn't currenty support 64bit. And on top of that my current hardware isn't 64bit compatible at the moment. When I build my next gigaPC I willl deffinitly have 64bit extensions, and I have access to the 64bit beta from microsoft so I may try it out. But in the mean time I'd love nothing more than to find a way to use all my memory, as it would benefit a whole comunity of people who use the program.
 
I'm confused. They have a driver that needs that much memory? How do you know this? (Trying to figure out why anyone would write an app in kernel space instead of user space.)
 
Gigastudio is a sampler/playback program, being able to access memory on the kernel level allows it to stream samples dirrectly from the hard drive by loading only a small portion into memory. So instead of being limited to sounds at a few megs in size you can use ones that are gigs in size, since only a small portion of each is loaded into memory at any given time. This has the effect of making instruments sound much more realistic(see my website for examples: http://www.cmpsr2000.com). You can see how a full symphony of samples loaded into one machine would very quickly overrun 1gb, even with only the 64k of each sample loaded into memory at any given moment. The problem with this, is that the kernel only has 2gb of addressable memory, and about half that is usually used by the OS. Meaning that I could have a pc with 2gb of memory or 4gb and still get the same max load in gigastudio (1gig at the moment)

However if I could break the OS's VMA split point ... :rolleyes:

Someone suggested to me that one(or all) of the versions of server 2k3 have expanded kernel memory areas, or management thereof. Does anyone know if this is true?
 
But, User mode is fine for that too. But obviously you didn't write the program your just using it. I'll take a look at it at work.
 
Let me see if I've gotten this right:
It maps the (huge) files into virtual memory, but only small parts of them actually need to be in physical memory at any given time?

Sounds like a poster case for 64bit addressing, really, even with the associated issues.
 
HHunt said:
Let me see if I've gotten this right:
It maps the (huge) files into virtual memory, but only small parts of them actually need to be in physical memory at any given time?

Sounds like a poster case for 64bit addressing, really, even with the associated issues.
Click to expand...

It maps the 64k chunks of each sample into VMA, those pages are used for the actual streaming process sort of as a buffer between the hard drive and the sound card. So yeah, only 64k of the sample is active at any given time, the rest of the sample remains on the hard drive. The reason I end up with so many addresses used is because I'm loading an entire orchestra, which is about 30 virtual instruments with an average range of 4 octaves each which = 48 samples per instrument x 64k per sample. Plus on top of that some instruments are keyswitched, so you have 4 octaves(48 samples) x 12 keyswitches(articulations)= 12x48x64k So yeah, it runs out pretty fast.

(contextualize the above with my limited understanding of what the software is actually doing)

You are correct though, 64bit addressing would solve all my problems but unfortunatly this most recent upgrade to the software is NOT 64bit compatible >.<
 
So the only way around it today is to switch quickly between what's loaded and not (unmapping something not currently in use and mapping in something else you need whenever neccesary) ?

For me, the most obvious thing seems to be to do some very speculative preloading of only the interesting parts, stuffing things into a limited number of buffers within available physical memory, and dumping things when they're not needed. I guess this is either
1) What it already does that you want to get away from,
2) Too slow,
3) A mess to implement,
4) No at all feasible, for some reason I've overlooked,
5) Too ugly to suffer
or some combination of the above. :)

(It's also eerily similar to what happens when you run out of physical memory.)
 
So let me take a step back here. How do you know you are running out of addressable kernel memory? That the driver is allocating it all? Do you have a kernel debug dump to prove this, or is this just conjecture based on another problem?

They don't have a demo downloadable so I can't profile it here at work. =( So how do you know there's a problem?
 
Short of talking Microsoft into giving you the full kernel source code, there is no way other than the boot switches already mentioned to adjust how the system uses or addresses memory.

I have never heard of anyone running out of kernel address space, but if that's truly the case then yes, go with a 64 bit OS and loads of RAM.
 
ThomasE66 said:
Short of talking Microsoft into giving you the full kernel source code, there is no way other than the boot switches already mentioned to adjust how the system uses or addresses memory.

I have never heard of anyone running out of kernel address space, but if that's truly the case then yes, go with a 64 bit OS and loads of RAM.
Click to expand...

He shouldn't actually need more RAM than he already has, but let's not spoil a good excuse for adding more. :D
 
hehehe I like the way you guys think :D

hhunt: great ideas, unfortunatly I'm a musician not a developer :cool: I can tell you that fast switching between patches is impossible. The load time for the entire orchestra is about 15 minutes, loading an individual patch takes 15 seconds to 2 minutes depending on size(I'm guessing it has to parse all the sample files which are already compressed in the native format of the file and determine which data needs to be paged to memory, then decompress and move said portions).

Ranma_Sao: as far as proof goes, there is a memory tester included with giga3 that has been my primary testing tool, and every number it spits out is right in line with the number I get when actually running the program and loading up some patches. In addition a wonderful bunch of people over at www.vsl.co.at have done boatloads of testing, but again they are all musicians not IT peoples. For my evidence I did a series of tests which I won't detail so as to not bore everyone too badly, but basically the short and sweet:

The Available tested memory readout goes something like this:

x,x,x,x numbers are (nopagedpoolquota, nopagedpoolsize, pagedpoolquota, pagedpoolsize)

Normal Boot, no Paging modifications 730mb
Normal Boot, 256,256,256,256: 956mb
Normal Boot, 2, 230000, 2, 230000: 1010mb
Normal Boot, 2, 8000000, 2, 8000000: 1008 <----increasing pool size too great uses more kernel memory addresses, hence I can load less samples. This trend increases as I move the number higher.
/3g /userva=3030 Boot, 2, 230000, 2, 230000: 22mb <----3gb mode reduces the kernel to 1gb addressable memory, which is about what my os is using when you take into account all the paging tables, drivers etc...
/3gb /userva=2900 Boot, 2, 230000, 2, 230000: 147mb <---increasing the alotment to the kernel by 130mb of addresses increases the accessable memory in gigastudio by basically the same amount
/3gb /userva=2800 Boot 2, 230000, 2, 230000: 250mb
at that point I started taking jumps, the /userva switch only works for adjusting the kernel between 1gb-2gb, which means I couldn't just do a /userva=500 and all of a suden be working the way I wanted. In any case the trend continued until I hit about 950megs loadable into gigastudio, which is what I expected.

on top of that the developers of gigastudio have said several times that they" implement everything at the kernel level for fast streaming etc etc blah blah"

anyway the docs on microsoft.com seam to confirm everything, plus everyone elses experience with the software...I just want to find a way to beat it! >:|

I can find screenshots of some of this stuff if anyone thinks it would help/
 
Sounds like a poorly written program to me if it is accessing kernel memory instead of userspace memory :(
 
Fryguy8 said:
Sounds like a poorly written program to me if it is accessing kernel memory instead of userspace memory :(
Click to expand...
I imagine it simplifies and/or speeds up feeding the soundcard drivers, or perhaps it's related (in some obscure way) to a timing issue.

Realtime audio mixing isn't ... normal. These kinds of things have been done on all sorts of unusual platforms, in all sorts of unusual ways because of it. (BeOS was supposed to be very nice.)
 
HHunt said:
I imagine it simplifies and/or speeds up feeding the soundcard drivers, or perhaps it's related (in some obscure way) to a timing issue.
Click to expand...

quite right, they even had to deveolope a hardware chip to go onto sound cards (GSIF) so they would be able to handle the streaming samples. It's simply not possible to ooperate the way it does without being in the kernel. If I spent 30min I could probably dig up some articles on why in google >.>
 
I'll dig when I have spare time, but I believe the answer is move to 64 bits. This is a poster child of why we need more address space. However, I do belive the developers could have written the program to use user space and not have had the huge perf hit they are talking about.
 
Ranma_Sao said:
I'll dig when I have spare time, but I believe the answer is move to 64 bits. This is a poster child of why we need more address space. However, I do belive the developers could have written the program to use user space and not have had the huge perf hit they are talking about.
Click to expand...

Possibly, I'm not aquainted with the intricacies of why it was written the way it was.
 
You must log in or register to reply here.
Back
Top