Setup question - bigadv and GPU not playing nice

[musky]

So here is the system:

I7 920 @ 3.33
275 GTX
8800 GTS

For the last several days, the video cards have been crunching P59xx units, turning about 7800 ppd on the 275 and 3200 ppd on the 8800, which is about normal for those units. The CPU running bigadv units turns 35 - 36 minute frames, for about 22K - 23K ppd. Overnight, I picked up a couple of P57xx GPU units. This boosted ppd on the video cards (8300 and 3900 respectively), but it kills my CPU, down to over 46 minutes per frame and under 17K ppd. I have GKrellm monitoring usage on each core, and it looks like the GPUs randomly hit various cores on the CPU, dropping them momentarily to 0% usage once they are finished. The core will then jump back up to 100%. Overall, the GPUs are only using about 4% CPU, but I believe that the way they are using the CPU is killing its production.

Any thoughts on a better way to do this? Giving up 5K ppd CPU to gain 1.2K GPU is not exactly optimal. This is a Linux machine (Ubuntu 9.04), so Windows tools are not going to help me. I am seeing this on my two I7 rigs with video cards. The other one has a single 260 GTX, and I am seeing a 3K ppd drop in CPU with a 600 ppd gain in GPU, so it looks like it scales decently with the number of video cards. I see it also with an AMD rig, but the CPU drop is only 400 ppd, while the GPU gain is about 600 ppd. That is probably because the AMD rig only turns 4100 ppd from the CPU on a good day.

 

[Tobit]

Unfortunately, this is about normal and there is little that can be done. I also see this on my Windows rig that folds on three GPUs and a Q6600. There are several new GPU work units out there that simply kill CPU performance regardless of platform. When these rogue units hit my Windows box, I drop about 2.5K in PpD. When I ran the GPUs on my bigadv box, I experienced the same thing.

 

[capreppy]

If you can find affinity tools for linux like WinAFC, you might be able to do something about it.

I fold with 3 GTX285's in my sig rig so can't actually do what i am about to recommend:

use 7 cores SMP and tie the GPUs to one core. you would have 12.5% CPU allocated to the GPUs and the remainder will go SMP (87.5%). This would (in theory) focus your GPUs on one core and prevent any impact on high CPU usage GPU WUs.

 

[Tigerbiten]

All my boxen fold with a bigadv VM and multi (2 or 3) GPU clients.
I lock my GPU's to one core and set them to normal piority.
I have my bigadv VM useing all 8 cores but set to low piority.
I think this setup maximizes my PpD.
YMMV.

Luck ...........

 

[Kendrak]

The Tiger has spoken.

 

[SmokeRngs]

While I don't have proof of this, it has been my experience that the A3 core work units generally do not like even a single thread impinging on the the resources needed for the work units. Even some basic tasks which use up a small amount of CPU resources on a single core seem to disproportionally affect the overall PPD of the work unit. I believe it has to do with how the A3 core is optimized to more efficiently use multiple cores. This probably applies to -bigadv as well. It also has to do with the bonus points system used for -bigadv and A3 core work units.

My observations are based on A1, A2 and A3 core work units over the last couple of years. A1 and A2 core work units don't seem to be affected nearly as much when something else is pulling cycles away. This is only a guess on my part but on my Linux machine when something is fully using a single core while running an A3 unit I will have a second core drop to 0% usage while the other two cores of my quad will continue crunching away. Again, it's only a guess on my part but I believe the A3 units and probably -bigadv as well are optimized to be used in a configuration with an even number of cores. When a single core is bogged down with something else, the "matching" core of the pair may have to idle down in order to wait for the loaded core to produce its own results for the "matching" core to continue its work. I have no clue if the SMP clients or cores work this way or not but I do know some SMP aware programs do have limitations like this since the work in one thread is required for the work in another thread to start.

If I'm way off base here, please say so as I would prefer to know the truth so I don't mislead anyone else with my ramblings and delusions.

 

[musky]

All my boxen fold with a bigadv VM and multi (2 or 3) GPU clients.
I lock my GPU's to one core and set them to normal piority.
I have my bigadv VM useing all 8 cores but set to low piority.
I think this setup maximizes my PpD.
YMMV.

Luck ...........

That is more or less what i was thinking. Any ideas on how to do this in Linux? I used taskset this morning to bind the GUI process to core 7, but I am still seeing the same behavior of different cores shutting down at different times. Also, i assume core 7 is a virtual HT core. Would a physical core like core 0 be better?

If I only had BigOS available...all my problems would be solved...

 

[musky]

Another problem with taskset is that for each new work unit, the core gets a new process. Is there a way to start the client to force all processes spawned by that process to use a specific processor core?

 

[Tobit]

Other than setting nice levels, I've really never had to do any specific affinity management on any of my Linux boxen. I've run upwards of three G200-based video cards via WINE along side of a bigadv and, although, bigadv ran a little slower, it was never as bad as what you are experiencing. However, all of my Linux boxen do not run X11 so I am sure you have a lot more overhead than I do.

You could try running bigadv with -smp 7 but it may not give you the PpD you are looking for or are used to. I ended up moving my GPUs into dedicated boxen to maximize PpD of bigadv. With the new bonus system, I am being extremely anal these days and doing everything i can to not run any GPUs along side A3 or bigadv work units.

 

[musky]

Binding to a single core has bumped my TPF on the bigadv up to just under an hour. I am shutting down the 8800 to see if that help frame times significantly. My I7 rig with a single 260 GTX at the same clock speed is still turning 38 minute frames, which is not too bad (down by about 3K PPD.) Something is seriously wrong with this thing.

Tobit, I would really like to try your OS on this thing to see if it has the same problem in its current configuration. I'll be gone for the next few days, but I should have some time next week to make the change.