Points going nuts on DUAL SMP

[Sunin]

SMP1 - 2665 - 2058.16 PPD - 12hrs completion
SMP2 - 3062 - 184.43 PPD - 8d 21h +

I've had this combination on 2 machines. Every time I get this combo the 3062 project bogs to hell... anyone have this scenario?

I'll post this at F@H after we discuss a bit here.

Stats on the machine:

Q6600 @ 3.6Ghz + 1 GPU 8800GT

[01:51:35] Thank you for your contribution to Folding@Home.
[01:51:35] + Number of Units Completed: 84

[01:53:39] - Preparing to get new work unit...
[01:53:39] + Attempting to get work packet
[01:53:39] - Connecting to assignment server
[01:53:42] - Successful: assigned to (171.64.65.63).
[01:53:42] + News From Folding@Home: Welcome to Folding@Home
[01:53:42] Loaded queue successfully.
[01:53:56] + Closed connections
[01:53:56] 
[01:53:56] + Processing work unit
[01:53:56] Core required: FahCore_a1.exe
[01:53:56] Core found.
[01:53:56] Working on Unit 05 [July 20 01:53:56]
[01:53:56] + Working ...
[01:53:56] 
[01:53:56] *------------------------------*
[01:53:56] Folding@Home Gromacs SMP Core
[01:53:56] Version 1.74 (March 10, 2007)
[01:53:56] 
[01:53:56] Preparing to commence simulation
[01:53:56] - Ensuring status. Please wait.
[01:53:57] - Starting from initial work packet
[01:53:57] 
[01:53:57] Project: 3062 (Run 3, Clone 76, Gen 72)
[01:53:57] 
[01:53:57] Assembly optimizations on if available.
[01:53:57] Entering M.D.
[01:54:14] 3 percent)
[01:54:14] - Starting from initial work packet
[01:54:14] 
[01:54:14] Project: 3Entering M.D.
[01:54:14] one 76, Gen 72)
[01:54:14] 
[01:54:14] Entering M.D.
[01:54:20] Rejecting checkpoint
[01:54:20] a SSE boost OK.
[01:54:20] ambda5_99sbExtra SSE boost OK.
[01:54:20] 
[01:54:20] Extra SSE boost OK.
[01:54:20] Writing local files
[01:54:20] Completed 0 out of 5000000 steps  (0 percent)
[03:16:58] Writing local files
[03:16:58] Completed 50000 out of 5000000 steps  (1 percent)
[05:31:49] Writing local files
[05:31:49] Completed 100000 out of 5000000 steps  (2 percent)
[07:47:09] Writing local files
[07:47:09] Completed 150000 out of 5000000 steps  (3 percent)
[10:02:15] Writing local files
[10:02:16] Completed 200000 out of 5000000 steps  (4 percent)
[12:17:31] Writing local files
[12:17:31] Completed 250000 out of 5000000 steps  (5 percent)

Folding@Home Client Shutdown.

 

[HighYield]

Wow Sunin. Can't say I have had that exact same situation and I am glad about it! How are you setting affinities with this boxen?

 

[Sunin]

SMPs are spread across 0-3 and so is the GPU.. so it should only be slowing down 1 of the 4 threads... I don't get it... for now I removed the 2nd SMP and points jumped up some.

 

[Tigerbiten]

Which OS ??
Are you useing AC ??

It sound like one of the CPU cores is being slightly overloaded.
This is causing one of the SMP cores to stall.
Not badly enough to swop to another CPU core, but just enough to slow to a crawl.
And because one core runs slow, the whole SMP client runs slow.

I've seen a similar effect running the SMP client while gameing.
You can get a better PpD by running the client on only three CPU cores but the client tries to keep running on all four.

What happens if you set affinity so one SMP client uses only CPU cores 0-1 and the other on cores 2-3 ??

Luck ...........

 

[APOLLO]

I had two 2665s and one 3062 on a dual quad a couple of days ago. There were no major slowdowns on any of the clients, but I have noticed that certain combinations of WUs can affect one another in certain ways. However, that machine does not have a GPU client.

What you've experienced is an anomaly, and I'd shut one client down to determine if that affects the other. You are likely seeing weird behavior because of the GPU client running alongside the SMP. On another machine, I had one SMP and one GPU client and the SMP client would freeze or error after several frames. There was no production from that particular client for two days, so I decided to shut it down and replace it with four standard clients. The standard clients working on bonus WUs never see more than 200PPD less than a single SMP client working on P2665s. The net loss is therefore negligible. This measure is only temporary, however, knowing that within a month's time there will no longer be any more bonus WUs for the standard clients, unfortunately.

 

[Sunin]

Which OS ??
Are you useing AC ??

It sound like one of the CPU cores is being slightly overloaded.
This is causing one of the SMP cores to stall.
Not badly enough to swop to another CPU core, but just enough to slow to a crawl.
And because one core runs slow, the whole SMP client runs slow.

I've seen a similar effect running the SMP client while gameing.
You can get a better PpD by running the client on only three CPU cores but the client tries to keep running on all four.

What happens if you set affinity so one SMP client uses only CPU cores 0-1 and the other on cores 2-3 ??

Luck ...........

Sorry XP 4gig, no AC. I had not messed with affinities. The fact is I need to have these setup where I don't touch the boxes. I travel far too much to be messing with them on a daily basis. That is part of the reason I went to single SMPs on some of the boxes. Yes I'm leaving 400-500 PPD on the floor, but it keeps the box happy without much hands on. Also when I add the 2nd GPU to each box it will be able to be done with little effort.

Ideally I'd like to limit the SMP to just 2 cores and leave the remaining two to the dual gpus. But that is where I am heading. I will have my 2nd box with dual GPUs by the end of the week. I even bought heavy duty extension cords 50ft, orange 14gauge 15amp cords and a circuit finder to trace the circuits back to my box in order to run in total 4 seperate 15 amp circuits into my data room. This should provide the power I need and allow enough flexibility to balance loads out.

 

[EvilAlchemist]

3062 3064 3065 need a large amount of cache.

Those are the WU's that are great if they have all 4 cores.

Mine slow down as well, if I get one of those.

 

[Tigerbiten]

Inside the VM's, I'm crunching p2605 98% of the time, the other 2% is p2665.
+3,000 p2605 vs ~60 p2665.
It one of the main reasons I run the SMP client inside VM's.
I get a higher average PpD. Runs at around 330 PpDpGhz for my VM's.

Luck ..............

 

[Xilikon]

Inside the VM's, I'm crunching p2605 98% of the time, the other 2% is p2665.
+3,000 p2605 vs ~60 p2665.
It one of the main reasons I run the SMP client inside VM's.
I get a higher average PpD. Runs at around 330 PpDpGhz for my VM's.

Luck ..............

Same here personnally with my home farm. At work, I don't have much choice but to run the WinSMP and sometime, they suffer from the 30xx WU but most of the time, it's 2665.

 

[nomad8u]

Sorry XP 4gig, no AC. I had not messed with affinities. The fact is I need to have these setup where I don't touch the boxes. I travel far too much to be messing with them on a daily basis.

I know where you're coming from. But with XP, from my experimenting, with running a GPU and 2x SMP clients, the only real successful way to keep the PPD up is using VM's for the SMP clients. VMware Server will allow you to run 2x Notfred SMP clients locked to 3 cores (I used 0-2) and you can isolate the GPU on core 3. I ran a consistent 75-7700 PPD like that since the GPU client came out until I added a 2nd GPU client the other day.

I know you need to have it setup for auto restarting in case of a power outage, and setting the GPU client to core 3 can be done with an environment variable. The VMware server SMP clients (I used Notfred's Folding CD but I guess it would be the same to setup 2 separate Linux VM's) can be set to auto start when the machine boots up. That just leaves the affinity issue for the VM SMP clients. This affinity manager (I have it downloaded, both versions but haven't had a need to try it yet) looks to be "rules based" one which if it is, AND if it works, could be setup to run the 2x "vmware-vmx.exe" Notfred SMP clients when they start on only cores 0-2.

I'm not sure if this would work out for you, but may be worth a shot... Good Luck.

 

[Sunin]

I know where you're coming from. But with XP, from my experimenting, with running a GPU and 2x SMP clients, the only real successful way to keep the PPD up is using VM's for the SMP clients. VMware Server will allow you to run 2x Notfred SMP clients locked to 3 cores (I used 0-2) and you can isolate the GPU on core 3. I ran a consistent 75-7700 PPD like that since the GPU client came out until I added a 2nd GPU client the other day.

I know you need to have it setup for auto restarting in case of a power outage, and setting the GPU client to core 3 can be done with an environment variable. The VMware server SMP clients (I used Notfred's Folding CD but I guess it would be the same to setup 2 separate Linux VM's) can be set to auto start when the machine boots up. That just leaves the affinity issue for the VM SMP clients. This affinity manager (I have it downloaded, both versions but haven't had a need to try it yet) looks to be "rules based" one which if it is, AND if it works, could be setup to run the 2x "vmware-vmx.exe" Notfred SMP clients when they start on only cores 0-2.

I'm not sure if this would work out for you, but may be worth a shot... Good Luck.

Thanks for the information. I might just build a new Quad box just to tinker with this VM stuff which won't make me freak out about not having it up since my main farm is going to be at almost 70k by the end of the week.

On the other hand I'm kinda leaning toward the future with only running 1 SMP with what has been talked about at the F@H forums in the public forums the new A2 core will scale much better and not need dual SMPs to max out a Quad core. I'm kinda torn. decisions decisions... if nothing else I think it would be great for me to have a "Beta" box to play with that has no impact on my farm production!

 

[nomad8u]

Thanks for the information. I might just build a new Quad box just to tinker with this VM stuff which won't make me freak out about not having it up since my main farm is going to be at almost 70k by the end of the week.

On the other hand I'm kinda leaning toward the future with only running 1 SMP with what has been talked about at the F@H forums in the public forums the new A2 core will scale much better and not need dual SMPs to max out a Quad core. I'm kinda torn. decisions decisions... if nothing else I think it would be great for me to have a "Beta" box to play with that has no impact on my farm production!

ANY excuse for m0ar D is a good one!

70kPPD! You ROCK man! I'd have a hard time screwing with those kinda numbers too.