Linux folding - What is the BFS kernel actually supposed to do?

C

Carbon_Rod

Gawd
Joined
Apr 2, 2012
Messages
1,022
So I just built a couple of 2700K SMP rigs for dedicated folding. I have Ubuntu 11.10 running on both (probably going to switch to 12.04 LTS when it's released because hey, it's LTS). Even though I figured I knew what I was doing (I'm not too shabby at handling Linux, but when it comes to kernels, I know just enough to get myself into trouble), I read the guide posted here in the forums about setting up Linux for folding and it suggested installing BFS. I read up on what BFS was and figured it couldn't hurt, so even after I'd been running the vanilla kernel for a couple days I made the switch.

Well, after all is said and done, what is it supposed to do? Particularly, are there any benefits to installing the BFS kernel? Is it supposed to provide some sort of PPD increase?

For the record, I did try doing a couple searches on the web to see if I could find any documented record of anyone stating what exactly it's good for. I've read other guides which, in not so many words, say "Install it" but none actually say why. Maybe I missed something along the way... but I figured I'd ask here in hopes that one of you Linux folding gurus could provide some of the answers I seek.

I apologize for my noobness.
 
Last edited:
So far, it seems that total CPU throughput is increased a bit for us with 24 threads or less. What you could do to test though, is to install it on one 2700K machine, and leave the other one without BFS and compare the differences (at the same clock speed of course).
 
Actually, to be honest, that's what I've already done which is why I had to ask what the intended benefits were supposed to be.

In all honesty, I didn't run them side by side for very long... only a couple days, but the preliminary results actually don't show any increase in PPD, no decrease in TPF, no real difference between the two machines. In fact, at first glance, it almost appears that my PPD actually decreased on the BFS installed machine even if it is by a very tiny amount (100 or less) though I do admit I don't have a very large data set to make a proper comparison.
 
Honestly, I have never really tested the difference. Even in the benchmarks I have seen, http://www.phoronix.com/scan.php?page=article&item=bfs_two_years&num=1, it does not have much of a performance improvement across the board. In some cases it has a performance decrease.

It has been in the F@H guide here for as long as I have been around so I never really questioned it. Maybe it has a performance improvement from 8-24 threads? It would be nice to find someone who has done a comparison.
 
I would imagine if it was a daily driver box, as in doing other stuff than folding, it might make a difference. It is essentially a different scheduler in the kernel. Some are better for server workloads (higher latency, higher throughput), some are better for client workloads (lower latency nut lower throughput).
 
That's what I was thinking as well... I had read a couple of things on the interwebs about peoples' desktops feeling "snappier" even when a processor intensive process like folding was happening in the background. But other than that, nothing else quantitative like a marginal PPD boost or anything.

So, given the circumstances, I'm moving my rigs back to the stock kernel. I had some issues with things suddenly being incompatible with the BFS kernel (ATI drivers, lm_sensors modules, etc.) and with the stock kernel, stuff just works. No extra effort of recompiling or recombobulating stuff to force it to work.
 
That is interesting...
BFS is a scheduler that specializes in lower core count cpus it scales very well up to 24 threads.

After various testing we run the ubuntu 10.10 (fastest kernel tested to date) installed on EXT3 <important, bfs if <24, kraken ... The guide is up-to-date ...
If you are running with those variables your ppd will most likely be faster...
If you are on ext4 the hour lag at end of wu might be throwing off any ability to notice ppd differences.
 
Well, that's one variable that I didn't account for. I had noticed the guide says to use ext3, but doesn't say why. The BFS section of the guide also doesn't make the earlier choice of ext3 apparent. And to be honest, I just referred to the guide for the BFS installation. I didn't actually use any of the instructions on installing Ubuntu or the FAH client because I already had that covered (the v7 client install is a breeze). I wanted to use the latest Ubuntu mainly because I had read that earlier versions had compatibility issues with some of the current devices found on current motherboards.

Any of the other guides I found on the intertubes about BFS installation also didn't mention to use ext3, though many of those guides were not related to F@H and were just explaining the BFS installation in general.

For the record, I didn't notice any hour long lag at the end of a work unit. But, I also didn't watch it over any period of time so it may have been there for some WU's but not for others. The most I noticed was a couple minutes.

I may revert one machine back to 10.10 just for kicks, just to maybe see what I would be missing by continuing to use the stock kernel. If it's significant (a couple thousand PPD), that would probably be enough reason for me to switch.
 
Carbon_Rod said:
Actually, to be honest, that's what I've already done which is why I had to ask what the intended benefits were supposed to be.

In all honesty, I didn't run them side by side for very long... only a couple days, but the preliminary results actually don't show any increase in PPD, no decrease in TPF, no real difference between the two machines. In fact, at first glance, it almost appears that my PPD actually decreased on the BFS installed machine even if it is by a very tiny amount (100 or less) though I do admit I don't have a very large data set to make a proper comparison.
Click to expand...


I am not so sure your test are not flawed. If you are going to run a test like that you need to use the same WU on both machine in order to get a relibale comparison. Also you should use a bigadv WU, you really will not see the diffrence on smp a coupple of sec per frame. I have never tested it on a MP computer so I really do not know if it makes a diffrencs there or not but I would imagine that it would be the same.

The test that I have run show up to 2 minutes per fram reduction in TPF on a i7 980X running a 6903 WU. I have not tested it in a while and the test were run on 2 diffrent Identical machines set up the same (same motherboard, memory and settings). One running Ubuntu 10.10 with BFS and ext3 file system and the other running Ubuntu 10.10 with ext3 file system.
 
Grandpa_01 said:
....

The test that I have run show up to 2 minutes per fram reduction in TPF on a i7 980X running a 6903 WU. I have not tested it in a while and the test were run on 2 diffrent Identical machines set up the same (same motherboard, memory and settings). One running Ubuntu 10.10 with BFS and ext3 file system and the other running Ubuntu 10.10 with ext3 file system.
Click to expand...

No kidding ? :rolleyes: ... I haven't install BFS on my rigs .. Gotta do it asap. darn.!
 
Carbon_Rod said:
Well, that's one variable that I didn't account for. I had noticed the guide says to use ext3, but doesn't say why. The BFS section of the guide also doesn't make the earlier choice of ext3 apparent. And to be honest, I just referred to the guide for the BFS installation. I didn't actually use any of the instructions on installing Ubuntu or the FAH client because I already had that covered (the v7 client install is a breeze). I wanted to use the latest Ubuntu mainly because I had read that earlier versions had compatibility issues with some of the current devices found on current motherboards.

Any of the other guides I found on the intertubes about BFS installation also didn't mention to use ext3, though many of those guides were not related to F@H and were just explaining the BFS installation in general.

For the record, I didn't notice any hour long lag at the end of a work unit. But, I also didn't watch it over any period of time so it may have been there for some WU's but not for others. The most I noticed was a couple minutes.

I may revert one machine back to 10.10 just for kicks, just to maybe see what I would be missing by continuing to use the stock kernel. If it's significant (a couple thousand PPD), that would probably be enough reason for me to switch.
Click to expand...

The reason we use ext3 is because ext4 on the bigadv work units causes an hour or more delay in the processing time. Since you are on SMP, you probably dont notice it. Just for comparison, the machine that I accidentally ran ext4 on took 2.8 days to finish a 6903, and I have it on SMP now. It completes 8 SMP units per day so you would be looking at that that hour divided by about 22 in my case. So maybe 3 minutes per SMP unit is wasted writing to disk which is a lot less noticeable. With ext3 the same system takes barely a minute to write the same 6903 that had a previous hour duration. SMP takes mere seconds.
 
Grandpa_01 said:
I am not so sure your test are not flawed. If you are going to run a test like that you need to use the same WU on both machine in order to get a relibale comparison. Also you should use a bigadv WU, you really will not see the diffrence on smp a coupple of sec per frame. I have never tested it on a MP computer so I really do not know if it makes a diffrencs there or not but I would imagine that it would be the same.
Click to expand...
The results I was looking at were taken from HFM.net which recorded both machines doing identical WUs. But you're right and like I said I don't have a big enough sample size to draw any firm conclusions. My data is almost certainly flawed to begin with especially when I find out now that I was running BFS wrong in the first place (I wasn't using ext3... I was using ext4 which may make a significant difference).

You're right though, a couple of seconds per frame on a SMP rig is hardly noticeable. IMO, that's all the more reason for me not to deal with BFS on these systems (but I might re-test anyway just to be sure).

The test that I have run show up to 2 minutes per fram reduction in TPF on a i7 980X running a 6903 WU. I have not tested it in a while and the test were run on 2 diffrent Identical machines set up the same (same motherboard, memory and settings). One running Ubuntu 10.10 with BFS and ext3 file system and the other running Ubuntu 10.10 with ext3 file system.
Click to expand...
I'm not sure if I want to get into the whole bigadv thing on these rigs... I may decide differently when I'm done tweaking.
 
Carbon_Rod said:
I'm not sure if I want to get into the whole bigadv thing on these rigs... I may decide differently when I'm done tweaking.
Click to expand...

None of the rigs in your sig are big enough for bigadv.
16 threads for 8101 and 2.4 day deadline. Even at 4.9ghz you would miss it by a day.
 
Carbon_Rod said:
The results I was looking at were taken from HFM.net which recorded both machines doing identical WUs. But you're right and like I said I don't have a big enough sample size to draw any firm conclusions. My data is almost certainly flawed to begin with especially when I find out now that I was running BFS wrong in the first place (I wasn't using ext3... I was using ext4 which may make a significant difference).

You're right though, a couple of seconds per frame on a SMP rig is hardly noticeable. IMO, that's all the more reason for me not to deal with BFS on these systems (but I might re-test anyway just to be sure).


I'm not sure if I want to get into the whole bigadv thing on these rigs... I may decide differently when I'm done tweaking.
Click to expand...

Sorry, let me clarify.

EXT3 VS EXT4 has nothing to do with BFS, you should run EXT3 because of the observed write issues.

Do run EXT3, even though you may not "notice" it, there is quite a bit of waiting. A few seconds off SMP TPF adds up when you are doing multiple units per day. My example was only showing that it is much easier to "notice" the wait when the units are so large and the time is so large. The "TOTAL" amount of time waited averaged out will be similar between SMP and the bigger units. An hour lost production is an hour lost production no matter how many pieces you cut up into.
 
Patriot said:
None of the rigs in your sig are big enough for bigadv.
16 threads for 8101 and 2.4 day deadline. Even at 4.9ghz you would miss it by a day.
Click to expand...

Ya, I sorta figured they wouldn't make bigadv standards... I wasn't really sure to be honest though.
 
Untitledone said:
Sorry, let me clarify.

EXT3 VS EXT4 has nothing to do with BFS, you should run EXT3 because of the observed write issues.

Do run EXT3, even though you may not "notice" it, there is quite a bit of waiting. A few seconds off SMP TPF adds up when you are doing multiple units per day. My example was only showing that it is much easier to "notice" the wait when the units are so large and the time is so large. The "TOTAL" amount of time waited averaged out will be similar between SMP and the bigger units. An hour lost production is an hour lost production no matter how many pieces you cut up into.
Click to expand...

Ah, I see! OK... guess I'll be at least changing over to ext3 then. Thanks for the info!
 
Carbon_Rod said:
Ah, I see! OK... guess I'll be at least changing over to ext3 then. Thanks for the info!
Click to expand...

You can shrink your ext4 with gparted and make a new partition with ext3 and stick the folding directory in there.

Or reinstall 10.10 and get the faster kernel...
 
DWolvin said:
That Kernal is Intel only, correct?
Click to expand...

No, but its happy up to dual 2011 octos and dual G34 dodecas....

24 cores or 36 threads... I could use a verify via pjkenned though.
 
its happy up to dual 2011 octos and dual G34 dodecas....
Click to expand...
Alright, now I'm confused. Did I misunderstand? I'm running 10.10 on my G34 4P, 36 cores. Should I run a newer kernel? What about 48-core G34?
 
Linden said:
Alright, now I'm confused. Did I misunderstand? I'm running 10.10 on my G34 4P, 36 cores. Should I run a newer kernel? What about 48-core G34?
Click to expand...

UP to.... no leave your systems at stock kernel...
It has negative effects over the line.

I think it was even bad on my 2p 24core amd rig....
but severe negative impacts on 32 and 48core amd boxen.

10.10 stock kernel is what you want unless you are running,
Intel: 1366 1p,2p 1155,1156, 2011, 1p and I think 2p

AMD: c32, G34 1 (8,12,16) 2p (8)
 
<OT>
Linden, keep your kernel; make sure you've got Kraken in place and that NUMA is enabled.

Use of Kraken virtually takes any scheduler out of the equation.
</OT>
 
Now we know what has changed since BFS was added to the ubuntu guide. Thank you Tear, that helps this thread out.
 
Has anyone compared the new 3.2 kernel in 12.04LTS against the kernel in 10.10 (2.6.35 if I'm not mistaken)?
 
Untitledone said:
EXT3 VS EXT4 has nothing to do with BFS, you should run EXT3 because of the observed write issues.
Click to expand...
So, just out of curiosity, is ext4 the only "problem child" filesystem when it comes to this issue? Are there any reported issues with any others like ReiserFS? JFS? ZFS?
 
Carbon_Rod said:
So, just out of curiosity, is ext4 the only "problem child" filesystem when it comes to this issue? Are there any reported issues with any others like ReiserFS? JFS? ZFS?
Click to expand...

I have not heard of any reports from other file systems. However most of us do not deviate much from the guide. Many of us do not use linux outside of folding so there are particular benefits from the cookie cutter approach. The only filesystem that his has been observed on in large numbers is ext4 . If you want to do your own tests, go right ahead. I personally do not know of any other file systems that exibit this behavior. I have only used ext3 and briefly ext4.
 
Untitledone said:
I have not heard of any reports from other file systems. However most of us do not deviate much from the guide. Many of us do not use linux outside of folding so there are particular benefits from the cookie cutter approach. The only filesystem that his has been observed on in large numbers is ext4 . If you want to do your own tests, go right ahead. I personally do not know of any other file systems that exibit this behavior. I have only used ext3 and briefly ext4.
Click to expand...

OK, no worries. I'll work with ext3 for now, but this sounds like it would be something that I might experiment with at some point down the road. :p
 
Any journaled filesystem w/enabled barrier demonstrates FahCore's and client's issues.
 
I just want to thank everyone for the info they've provided in this thread so far! This is one reason I once again started to fold for the [H]... the folks here just rock!
 
You must log in or register to reply here.
Back
Top