Linux versus Windows - bigadv folding results

I think I'm going to compare schedulers next. BFS was brought up in MIBW's thread on FF. There's a couple of distros that use it in their kernel (Zenwalk 6.4 and PCLinuxOS according to wikipedia), and Zen Kernel uses it as well as a few other custom kernels that are out there. The nice thing is you can install Zen from portage on Gentoo, you just have to unmask it first. I should have it up and running sometime Friday morning.
 
I'd be really interested to hear your results with Zen on Gentoo - I've been looking for an excuse to get off Windows again.
 
What if you start it with -smp 22 rather than just -smp or -smp 24 ? Do you still get the fluctuations?

You'd be losing two cores for folding, but they'd then be available for whatever is going on in the background thats screwing with folding.

H.
 
hspeirs said:
What if you start it with -smp 22 rather than just -smp or -smp 24 ? Do you still get the fluctuations?
Click to expand...

You don't get to fold at all. It will not fold with -smp 22 or -smp 23 - complains about the numbers of threads and restarts the WU. Unfortunately the gromacs error I saw in the terminal window did not make it to the logfile I saved:

Code: 
[15:08:34] Project: 6701 (Run 0, Clone 5, Gen 91)
[15:08:34] 
[15:08:34] Entering M.D.
[15:08:40] Using Gromacs checkpoints
[15:08:40] mdrun returned 255
[15:08:40] Going to send back what have done -- stepsTotalG=2000000
[15:08:40] Work fraction=390842023936.0000 steps=2000000.
[15:08:44] logfile size=142307 infoLength=142307 edr=25 trr=1
[15:08:44] logfile size: 142307 info=142307 bed=25 hdr=1
[15:08:44] - Writing 142845 bytes of core data to disk...
[15:08:44]   ... Done.

hspeirs said:
You'd be losing two cores for folding, but they'd then be available for whatever is going on in the background thats screwing with folding.

H.
Click to expand...

I am afraid I don't get that - how would that help? I am reasonably sure this is not disruption due to excessive background processes. And besides, moot point - no benefit folding on 22 cores when I can do 24 on windows.

As I see it us SR2 users have 3 showstopping issues on Linux right now.

  • Using all cores - consistently/reliably/at all
  • 1 hour to shutdown a unit
  • high risk of borking a unit if stopping/resuming
 
may i be worthy said:
You don't get to fold at all. It will not fold with -smp 22 or -smp 23 - complains about the numbers of threads and restarts the WU. Unfortunately the gromacs error I saw in the terminal window did not make it to the logfile I saved:
Click to expand...

Damn. Wonder what causes that, never heard of it being a problem before.

I am afraid I don't get that - how would that help? I am reasonably sure this is not disruption due to excessive background processes. And besides, moot point - no benefit folding on 22 cores when I can do 24 on windows.
Click to expand...

To compensate for two less cores you'd need about 8% better performance under Linux - I've seen improvements from 12% - 27%. If it would fold with 22 under linux you still should be getting more PPD.

As I see it us SR2 users have 3 showstopping issues on Linux right now.

  • Using all cores - consistently/reliably/at all
  • 1 hour to shutdown a unit
  • high risk of borking a unit if stopping/resuming
Click to expand...

Using all cores is definitely a problem.

The one hour shutdown may be a file system issue. I found a post over on FF talking about 10 minute shutdown with standard SMP when the partition is formatted with EXT4 - which is the default for most distros these days. I've just reformatted mine with EXT3 to see if it resolves it. Another option is to copy the folding folder to a tmp fs (basically a RAM disk) with regular backups to real disk.

My experience is under linux you can generally stop a unit once, twice will sometimes work, more than twice you're asking to bork it.

H.
 
Damn. Wonder what causes that, never heard of it being a problem before.
Click to expand...

I think it is the new code enforcing valid smp numbers - but that explains -smp 23 but not -smp 22.

To compensate for two less cores you'd need about 8% better performance under Linux - I've seen improvements from 12% - 27%. If it would fold with 22 under linux you still should be getting more PPD.
Click to expand...

Oh I see where you are headed. I was presuming a benefit of about 10%...

The one hour shutdown may be a file system issue. I found a post over on FF talking about 10 minute shutdown with standard SMP when the partition is formatted with EXT4 - which is the default for most distros these days. I've just reformatted mine with EXT3 to see if it resolves it. Another option is to copy the folding folder to a tmp fs (basically a RAM disk) with regular backups to real disk.
Click to expand...

Great, that would be awesome if that was the case - thanks for testing and keep us posted.

My experience is under linux you can generally stop a unit once, twice will sometimes work, more than twice you're asking to bork it.
Click to expand...

Well my 6701 that I tried across 2 distros probably had 8 stops before it died, so that wasn't completely unexpected. :D
 
may i be worthy said:
I think it is the new code enforcing valid smp numbers - but that explains -smp 23 but not -smp 22.
Click to expand...

Was that a new WU or a resumed one? It might not handle a resumed unit going from 24 threads down to 22.

Oh I see where you are headed. I was presuming a benefit of about 10%...
Click to expand...

For the times I looked at over on FF, it was 12-20% on Intels. For my AMD box it was 27% - don't know if that's AMD vs Intel, or number of cores. Either way, I'd expect at least 12% on the SR-2's

Great, that would be awesome if that was the case - thanks for testing and keep us posted.
Click to expand...

No problem - will have an answer in about 24 hours ....

Well my 6701 that I tried across 2 distros probably had 8 stops before it died, so that wasn't completely unexpected. :D
Click to expand...

You did damn well then - don't think I've had one survive more than 4.

Debating whether to stop the linux box and cut it over to the [H] - others already done, nervous about killing it .......

H.
 
Someone give me something to try...please! I am seeing the same results with Ubuntu 9.04 - 2300% cpu usage. That is with gdm stopped and nice -0 when starting the client. This is getting beyond annoying....
 
Musky,

Get the pid of the fah_core process - top will show it, then try:

renice -19 pid

That'll push the fah process to almost highest possible priority.

H.
 
One other thing you can try - on the boot screen add to the boot option "noapic" (without quotes). You can also edit the grub.lst and add it as a permanent boot option.

H.
 
I reniced it to -13 and it made no difference. -19 doesn't either.
 
Tell it sudo

I hear that makes things happen
Posted via Mobile Device
 
Try the noapic boot option then, it's something that screwed with me for about 6 months on vmware - caused one core to be 50% utilized doing nothing. Add the noapic option, problem solved.

H.
 
Musky,

What are the frame times like though? My dual hex runs at about 1130% with consistent times. Are yours still all over the place ?
 
Frame times are consistant. They are just significantly slower than Windows.
 
musky said:
Frame times are consistant. They are just significantly slower than Windows.
Click to expand...

well take heart... your just doing it wrong :D
but so is almost everyone else...

lol... so they broke the known fast one... let me see if they are listening to a fix... recompile with flag for compat...
 
Damn. I'm pretty much out of hail mary's. My situation is the opposite - and no idea how to explain why.

Are you using live CD's or actual installs ?
 
These have all been actual installs. I am out of things to try at the moment, so I guess I'll wait until someone figures it out.
 
I was hoping I could reproduce it so I could take a shot at fixing it - but look like my hardware just isn't up to exposing the issue :(

H.
 
Well, if anyone wants SSH access to one of my machines to troubleshoot, just let me know. Tell me which distro you want to start with and i will get it set up. I may need some help with the initial setup, depending on the distro.
 
have you tried using sar to see if disk io is an issue?

sar -dp 5 60

that will give you 60 iterations at 5 second intervals

one thing you'll want to look at is the avg wait time
 
took my slow box 20min from 100% to submitted last a3 and dled a5 and starting 6901

no 40min from 100% to submitting here...
dual 2ghz i7 xeon no ht 36:26 tpf rhel5.6 x64
 
Well I am going to subscribe to this thread. I currently have an SR-2 running Win7 and the A5 core. It is a tad slower than the A3 core which really sucks. I've moved my two i7 920's to the A5 core as well. A tad slower there as well.

The SR-2 & one i7 920 are dedicated and I can move them to Linux when someone finds a good distro and instructions. I tried Linux in VMs and once (just once) as a primary OS and I realize I pretty much SUCK at Linux.

Hopefully someone comes up with a good Linux Distro to use and will graciously provide instructions for us Linux Noob's to follow.
 
I am going to get Skankbox48® on CentOS tonight and then try a Ubuntu desktop variant on the SR-2 by swapping out the HD (leaving the Win7 intact) and see if I can find anything. I was gonna do the 48 core last night but work/life got in the way (booooo work).

Even it if means BIOS changes or different distros I will not give up on the SR-2 . Even if I have to turn on NUMA (no!!!)
 
10e said:
I am going to get Skankbox48® on CentOS tonight and then try a Ubuntu desktop variant on the SR-2 by swapping out the HD (leaving the Win7 intact) and see if I can find anything. I was gonna do the 48 core last night but work/life got in the way (booooo work).

Even it if means BIOS changes or different distros I will not give up on the SR-2 . Even if I have to turn on NUMA (no!!!)
Click to expand...

cool... I am probably gonna swap some ram around...

beasty 24cores @1.7ghz 18:23 tpf in dual channel... both 6901
half the cores c32@ 2.1 29:39

I think I managed to keep the c32 in dual channel (which is its max) and put the G34 in quad...
soo if the c32's tpf dont change... and the g34s do.... :)

G34 17:20 next frame was 16:20 :eek: hope it keeps That trend... rhel5u5 --- it stayed at 16:20... that is much better than 18:23 yeah!
c32 waiting for another few frames but looks like 27:56
 
Last edited:
You may want to play with node-node-interleaving as well. With it on NUMA is off, and with it off NUMA turns on. Tear on the folding forum has his on, which seems to be fine for Linux but not Windows.

The best Windows TPF I've seen was on p6900s where it would hit around 10:30, but with all things being equal on Linux should go down to the 7:45 range (assuming linear scaling to 1.9Ghz from 2.2ghz of Tear's 6174s to my 6168 G34 CPUs).

For the G34 there is also a huge improvement by staying at quad.

At 1.9Ghz with only duals I was getting p6900s at around 22:00 TPF so that is definitely an improvement for your dual 12-cores.

Patriot said:
cool... I am probably gonna swap some ram around...

beasty 24cores @1.7ghz 18:23 tpf in dual channel... both 6901
half the cores c32@ 2.1 29:39

I think I managed to keep the c32 in dual channel (which is its max) and put the G34 in quad...
soo if the c32's tpf dont change... and the g34s do.... :)

G34 17:20 next frame was 16:20 :eek: hope it keeps That trend... rhel5u5 --- it stayed at 16:20... that is much better than 18:23 yeah!
c32 waiting for another few frames but looks like 27:56
Click to expand...
 
10e said:
Even it if means BIOS changes or different distros I will not give up on the SR-2 . Even if I have to turn on NUMA (no!!!)
Click to expand...

Help us Obi-Wan Ken-10e, you are our only hope...

Well not entirely true, it looks like we have a lot of bright folks working on this now, but I have clearly reached the end of my abilities, and must patiently wait for a savior. Thankfully there are some positive dual socket results starting to happen on the FF thread I started. Time will tell.

Hmmm.... OWK10E - I like that. You have to admit, it just fits you - ancient, likely English accent, watching over us from afar, almost certainly a white beard...
 
Its really starting to sound like a scheduler issue with kernels that use CFS for scheduling. bollix47 on FF said hes not seeing the issues that MIBW is on Ubuntu using the linux-ck kernel, which uses BFS for scheduling instead. He gave a link on how to install it. Honestly, this would be what I would recommend trying next. add the repo, install, and reboot using that kernel. Hopefully that solves your issues.
 
my tpf are relatively stable under rhel... I actually have rhel5u4 5u5 and 5u6 installs on my set of folders... though most being rhel5u6 ... from what I understand CFS is default in the current kernel.org
so no matter what distro you pick you have got it... unless your a gentoo freak :p

That would explain why these custom kernels and super customized distros work better...
it seems there is something about the SR-2s that the CFS dislikes... amd based rigs seem fine either way...

average of 15 frames on G32beasty 16:20 i see a second or two flux...

grrr I think I have to rename that machine... need to find some faster 12 cores...

dual intel hexacores 2.8ghz just pulled a 15:43 tpf on first frame... I will wait around for the next one to confirm...
its in dual channel mode... have used up my stash of scavenged ram on my army... 4 x2gb sticks
How much does ram affect intel folders?
 
Last edited:
Vediovus said:
Its really starting to sound like a scheduler issue with kernels that use CFS for scheduling. bollix47 on FF said hes not seeing the issues that MIBW is on Ubuntu using the linux-ck kernel, which uses BFS for scheduling instead. He gave a link on how to install it. Honestly, this would be what I would recommend trying next. add the repo, install, and reboot using that kernel. Hopefully that solves your issues.
Click to expand...
All of my Linux folders over the past two years have ALWAYS run the BFS scheduler so I'd agree with this. Also, avoid exotic file systems such as XFS and ReiserFS and even ext4 in some cases, this can lead to long delays at the end of processing that some of you have already experienced. ext3 has been the most reliable over the years for me and I've tried all the options.

Off topic to all: Sorry for not answering any of the countless PMs and emails I've received over the past couple months. It has been difficult to get online in my current situ.
 
Tobit said:
All of my Linux folders over the past two years have ALWAYS run the BFS scheduler so I'd agree with this. Also, avoid exotic file systems such as XFS and ReiserFS and even ext4 in some cases, this can lead to long delays at the end of processing that some of you have already experienced. ext3 has been the most reliable over the years for me and I've tried all the options.

Off topic to all: Sorry for not answering any of the countless PMs and emails I've received over the past couple months. It has been difficult to get online in my current situ.
Click to expand...

Thank you for the info.... I hope your situation improves...
 
So what does all of this mean in terms of adding the BFS scheduler to an already custom kernel? I think this would just give me another kernel option to boot to. I'll see if I can figure it out - my Linux box has been pulling regular SMP units all day anyway.
 
Tobit said:
All of my Linux folders over the past two years have ALWAYS run the BFS scheduler so I'd agree with this. Also, avoid exotic file systems such as XFS and ReiserFS and even ext4 in some cases, this can lead to long delays at the end of processing that some of you have already experienced. ext3 has been the most reliable over the years for me and I've tried all the options.

Off topic to all: Sorry for not answering any of the countless PMs and emails I've received over the past couple months. It has been difficult to get online in my current situ.
Click to expand...

No need for apologies, just great to hear you are alive and kicking.:cool:

How much of this current hassle do you think is Linux and how much is the FAH client? Is the worth pursuing now or do you suggest we wait?
 
may i be worthy said:
How much of this current hassle do you think is Linux and how much is the FAH client? Is the worth pursuing now or do you suggest we wait?
Click to expand...
I think it's all kernel related issues combined with inexperience on some of you with Linux.
 
Tobit said:
I think it's all kernel related issues combined with inexperience on some of you with Linux.
Click to expand...

all I heard was noobs :D

I would try the BFS (great name) guys... I will do a test run again on tuesday...
 
Patriot said:
all I heard was noobs :D

I would try the BFS (great name) guys... I will do a test run again on tuesday...
Click to expand...

I would, but the guide doesn't work...

dave@cowgirl:~$ sudo apt-get install linux-ck-generic linux-ck-headers-generic Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-ck-generic
E: Unable to locate package linux-ck-headers-generic
 
You must log in or register to reply here.
Back
Top