Bigbeta in a VM

J

jebo_4jc

[H]ard|DCer of the Month - April 2011
Joined
Apr 8, 2005
Messages
14,566
Has anybody else played with this?

I am trying Virtualbox on my SR2, but so far it doesn't seem to want to use 100% CPU utilization, even though I have it set to 24 CPUs.


I have ubuntu loaded in Virtualbox right now, crunching on a 6904. The problem is, Virtualbox is only loading 12 or maybe 13 of the CPUs, despite TOP reporting 2400% CPU is being used from within Linux. We'll see what happens.

In other news, I guess with all that VT tech being used, I had to bump up my VTT voltages from a formerly stable 1.325 to 1.375 in order to keep the PC happy while folding in a VM.

I figured it out, go to post #17 for a guide
 
Last edited:
unsurprisingly, an hour later it has not made 1% progress.

I installed BFS and the Kraken, I wonder if that could be a problem.
 
MIBW did a lot of work on this, and I believe the conclusion was that it does not work worth a darn regardless of which VM software you use. TJ was playing around with VirtualBox on his 980x, but I never heard the results.
 
Musky's Ubuntu guide says to install BFS and Kraken for the dual processor Intel systems. I'll take a look at my VBox install on the 980x later and see if I can be of more help.
 
From the install guide:
Install BFS - for all Intel systems
Click to expand...

The kraken is only for dual processor systems, but I had the best results with BFS, even on 8 and 12 core single proc systems.
 
musky said:
MIBW did a lot of work on this, and I believe the conclusion was that it does not work worth a darn regardless of which VM software you use. TJ was playing around with VirtualBox on his 980x, but I never heard the results.
Click to expand...
If I recall, that was before bigbeta. With the added PPD of bigbeta WU, I wonder if it starts to make sense.

I'm running a 6904 now using the Linuxforge VM image in Virtualbox. Surprisingly, I'm now seeing 90% CPU usage.

Side note: of course, while I'm testing WU, I get two consecutive 6904s. Meanwhile, my actual 4P folding boxen is crunching away on regular bigadv. :/
 
Also, KMac PMed me with some tidbits from a PM conversation he had with MIBW.

MIBW said he tried VSphere but it wouldn't install properly with the SR2 NICs. I have an Intel NIC in my SR2, so I signed up to try VSphere. If this current Virtualbox WU doesn't behave nicely, VSphere will be my next target.
 
jebo_4jc said:
If I recall, that was before bigbeta. With the added PPD of bigbeta WU, I wonder if it starts to make sense.
Click to expand...

No, it was after bigbeta, but before the bigadv change.
 
I just completed a 6904 frame in 51 minutes. That results in PPD of about 120,000 according to the calculator. That's a nice improvement over the 90-100k PPD I've been getting on a 6900 in Windows.

I've been using this box all morning, so it's very possible performance has room to improve.
 
jebo_4jc said:
Also, KMac PMed me with some tidbits from a PM conversation he had with MIBW.

MIBW said he tried VSphere but it wouldn't install properly with the SR2 NICs. I have an Intel NIC in my SR2, so I signed up to try VSphere. If this current Virtualbox WU doesn't behave nicely, VSphere will be my next target.
Click to expand...

Don't bother unless it's the latest version of ESXi, version 5. Version 4.1 has a virtual CPU limit of 8.
 
Do you actually have v5? I'm been looking to download it (the free version) but it seems they have 4.1 still listed under their downloads.
 
musky said:
MIBW did a lot of work on this, and I believe the conclusion was that it does not work worth a darn regardless of which VM software you use. TJ was playing around with VirtualBox on his 980x, but I never heard the results.
Click to expand...

After running a few workunits the typical increase in ppd from Windows to a VM is 5%. I found the increase significant enough in my case to keep folding with the VM for now.
One caveat about the VM is if it is not configured to save the state upon exit, there's goes a workunit.
 
tjmagneto said:
One caveat about the VM is if it is not configured to save the state upon exit, there's goes a workunit.
Click to expand...

You could actually suspend or pause the thing before you shut it down instead of just closing the app...that is kinda like shutting off your PC by pulling the power cord...:)
 
tjmagneto said:
After running a few workunits the typical increase in ppd from Windows to a VM is 5%. I found the increase significant enough in my case to keep folding with the VM for now.
One caveat about the VM is if it is not configured to save the state upon exit, there's goes a workunit.
Click to expand...
that's good to know.

Frame #2 just completed at just under 47 minutes, resulting in 135,000 PPD. So, I am back to the PPD levels that I was seeing before bigadv died.

So, it may not match native linux perfectly, but it appears to at the very least be a 30% improvement in PPD (as long as the bigbeta WU flow) for those of us who can't live without Windows (or maybe, more accurately, can't live *with* linux).

Does anybody know anything about the linuxforge linux image? Should I try to install BFS or the Kraken on it?
 
jebo_4jc said:
that's good to know.

Frame #2 just completed at just under 47 minutes, resulting in 135,000 PPD. So, I am back to the PPD levels that I was seeing before bigadv died.

So, it may not match native linux perfectly, but it appears to at the very least be a 30% improvement in PPD (as long as the bigbeta WU flow) for those of us who can't live without Windows (or maybe, more accurately, can't live *with* linux).

Does anybody know anything about the linuxforge linux image? Should I try to install BFS or the Kraken on it?
Click to expand...
fantastic news jebo... now all you need to do is write up a guide like musky's so we can all do the same ;)
 
Activate: AMD said:
fantastic news jebo... now all you need to do is write up a guide like musky's so we can all do the same ;)
Click to expand...
There's not much to it. (If I did it, it must be simple).

1. Download Virtualbox http://www.virtualbox.org/wiki/Downloads (I am currently using version 4.1.0)

2. Download the linuxforge virtual machine image http://linuxforge.net/docs/crunching/fah-virtualbox.php

3. Install Virtualbox

4. Follow the "Virtualbox configuration" instructions (not the "native ISO" instructions) found here: http://www.linuxforge.net/docs/crunching/fah-install.php
Make sure you give it plenty of RAM (I'm using about 5GB right now) and the correct number of CPU cores. Everything else should be the way it's described on linuxforge
Here are screenshots from my config in case you need them for reference. Everything is grayed out because my VM is currently in use. Don't wanna interrupt the folding :)



5. FOR INTEL BOXEN: Once you've got the VM booted and you are sitting at the linux CLI, install BFS by typing
Code: 
upkernel http://www.linuxforge.net/fah/img/linux-native-39-bfs-404.tar.xz

6. Start FAH by typing:
Code: 
./fah6 -smp -bigbeta

You'll need to make sure you have all the virtualization options turned on in your BIOS (VTd and all that). In my testing it's best to have NUMA off.

If you see lower performance rather than higher once you are using the VM, double check on your CPU clock speed. According to MIBW, turbo doesn't always engage, resulting in a lower CPU clock speed. He was able to get around that problem with a workaround he posted. I did not have the same problem. I am using BIOS version A50 (dated 08/20/10). I have my BIOS settings posted here
 
Last edited:
musky said:
You could actually suspend or pause the thing before you shut it down instead of just closing the app...that is kinda like shutting off your PC by pulling the power cord...:)
Click to expand...
I forgot the verbiage when I made my post but that's what I did when I used VMs the first time around. It still scares me. :eek:

jebo_4jc said:
There's not much to it. (If I did it, it must be simple).

1. Download Virtualbox http://www.virtualbox.org/wiki/Downloads

2. Download the linuxforge virtual machine image http://linuxforge.net/docs/crunching/fah-virtualbox.php

3. Install Virtualbox

4. Follow the "Virtualbox Installation" instructions found here: http://www.linuxforge.net/docs/crunching/fah-install.php
Make sure you give it plenty of RAM (I'm using about 5GB right now) and the correct number of CPU cores.
Everything else should be the way it's described on linuxforge

5. Once you've got the VM booted and you are sitting at the linux CLI, type
Code: 
./fah6 -smp -bigbeta

You'll need to make sure you have all the virtualization options turned on in your BIOS (VTd and all that). At this point I still have NUMA off. I'm still trying to figure out if I should try BFS or the Kraken. Ubuntu clearly didn't like that combination when running in a VM. I don't know if this distro would be any different.
Click to expand...

There is an option to use a web based tool to configure the client. I'd skip using that route and go straight to the CLI method that jebo mentions. If somebody wants to try the web based tool I'd recommend not having the "SMP Auto-startup" and "enable bigadv" boxes check-marked.
 
Update: Linuxforge's image is pretty damn flexible, and has a built in option to install BFS. Doing this netted me an additional 30K PPD or so, so I'm up to around 42:00 TPF on a 6904 or 160K PPD. This is a huge improvement over the 90-100K that I was getting this morning on a 6900 in regular windows.

I edited Post #17 above with the BFS tidbit.
 
jebo_4jc said:
Update: Linuxforge's image is pretty damn flexible, and has a built in option to install BFS. Doing this netted me an additional 30K PPD or so, so I'm up to around 42:00 TPF on a 6904 or 160K PPD. This is a huge improvement over the 90-100K that I was getting this morning on a 6900 in regular windows.

I edited Post #17 above with the BFS tidbit.
Click to expand...

BFS gave you a really nice boost.

Meanwhile, my 980x landed a 6903. My average TPF is currently running at 43:40 which is around 91k ppd.
 
Cool. If I were you I would install BFS asap.

Linuxforge says his kernel is NUMA aware, but enabling NUMA actually decreased performance. I suppose the Virtualbox application is not optimized for NUMA, so I'm going to reboot and disable NUMA again.
 
This reminds me of when we were doing testing with NotFred's distro to run SMP WUs within a VM. All the comforts of windows with a linux performance edge.

 
yep, then we did it again when bigadv first hit the scene and we were all dumbfounded by the fact that we could get 25k PPD from our i7 920s at 4ghz running linux in a VMWare 3.0 instance.

I've been playing around with it a bit (too much), trying to get it to auto run FAH with the bigbeta flag set. I can't figure out how though. I tried the auto config web setup, and I wasn't happy with the results. I also tried to set up langouste, but that apparently didn't go well either.
 
This brings back memories of when I first played with VMs on the 980x. I configured 2 vms with both of them set up to report 8 cores to Stanford but used the -smp 6 flag. The ppd looked ugly until the points from both vms were added up.
 
I decided to give this a go on one of my i7-970s last night. It picked up a 6904 right away. :)
 
Nice! Post some ppd numbers when you get a chance.

My current status with the 6903 running in a VM on my 980x is 63% complete and running around 92k ppd. :)
 
I'll post the numbers tonight after work. I'll probably get this running on my other i7-970 as well.
I saw there were two choices at bootup that were for i7.
There was plain i7 and then i7-???
I can't remember what the ??? was off the top of my head.
Which one of those should I be using?
 
I figured that it was for the sandy bridge so I picked the right one. :)
 
90K for a 980, if I can get numbers similar to that on a 970, it would definitely induce me to shutdown the 3 GTX cards that I have folding.... I get about 75KPPD (combined bigadv + GPU) on my 970 rig as it is. Yes I know it's in-efficient, but it's more than bigadv alone and my electricity is free.

Dozer... mine is clocked at 4.15, what's yours?

If these numbers continue to look promising, I'll join the testing on Wednesday or Thursday over my "shiftwork weekend" between mids and days.
 
Anyone willing to try an Ubuntu Server VM with BFS to run against linuxrouter's image? I doubt it would make any difference, but it would be worth a try. Do I need to use VirtualBox to build the image, or can it run a VMWare image with the core count changed in the config file? The reason I ask is that I do not have VB instlled anywhere ATM.
 
How many cores can you use in VMWare these days?
I think that I remember VB allowing for you to use the vmdk images so I am going to assume that means that you can use them
 
Linuxrouter supplies separate downloads for vmware and vb so I guess there are some incompatibility issues there.

Can't you just download vb using the link above?
 
jebo_4jc said:
Linuxrouter supplies separate downloads for vmware and vb so I guess there are some incompatibility issues there.

Can't you just download vb using the link above?
Click to expand...

The main issue I ran into was the difference in timer frequency support between VB and VMware. VMware seemed to work best with a tickless kernel. VB would intermittently freeze with a tickless kernel although that was with an older VB version so this issue may no longer be present. I'll have to play around with the latest version of VB. Also, the VB image has the VB additions which was mainly added for clock drift.

Also, if BFS scheduler works best with VB, then I can default to that scheduler as well.

Running The Kraken may help performance also. I had messaged back and forth with tear a while back about integrating his program in to the folding images but there was some concern with compatibility and wrapping of future folding cores. Hopefully one day I can get that added in as well while in the short term it can be manually installed.
 
Cool. Thanks for the input.

linuxfah said:
Also, if BFS scheduler works best with VB, then I can default to that scheduler as well.
Click to expand...
I think we've found BFS works great for Intel systems, but not for AMD. At least, that was the case in native linux.
Running The Kraken may help performance also. I had messaged back and forth with tear a while back about integrating his program in to the folding images but there was some concern with compatibility and wrapping of future folding cores. Hopefully one day I can get that added in as well while in the short term it can be manually installed.
Click to expand...
I am pretty helpless once I'm in linux (hence the whole reason I'm trying to run FAH in a VM in the first place) so I wasn't able to install the Kraken. To able to fully take advantage of the Kraken, wouldn't VB need to be NUMA compatible?
 
Hey all, been too busy lately, but I finally had ago again at this last night. I wish I had posted about my testing, but then I would probably have put people off trying VB, which I had given up on as too buggy.

Haven't been able to duplicate Jebos success just yet, but having said that I have done better than any previous effort. Virtualbox has been updated to a new version since last time I tried, and has more options there now. I am not going to use the snapshots feature, as last time simply taking regular snapshots (not restoring from) borked the vm bootup.

SR2#1 - Win7, x5650 using Linuxrouter/BFS, (no krakken) NUMA off, cstates enabled I have been able to get TPF that are16% worse than windows and 28% worse than native linux with Krakken. (prior to this I have not gotten better VM frame times than 2 to 3 times worse than native performance on anything)

CPU as measured on windows hovers around the 90-95% mark.

So there must be something I am missing in either the BIOS or VB setup. Fark. This is where I take back all the horrible things I like to say about Jebo:p, and beg him to screenie every damn bios and VB setting.
 
Well sir that crack about my mom really hurt, but but I will try to forgive you :)

Do you have all the virtualuzation option turned on?

I will post screens in a couple hours
 
Ok thanks for that - I have identical settings in VB, which makes me think I have something in the BIOS different.

What does windows taskmanager say your CPU usage is?

Are there virtualisation settings in the BIOS other than in the CPU configuration menu?

I have:

MAX CPUID Value Limit - [Disabled]
Intel (R) Virtualization Tech - [Enabled]
Intel VT-d - [Enabled]
Coherency Support - [Enabled]
Execute-Disable Bit - [Enabled]
Intel (R) HT Technology - [Enabled]
A20M -- [Disabled]
Turbo and c state enabled fully
ACPI T State - [Enabled]

Right now the only thing I can conclude is that this is a personal vendetta against those of us who spell virtualisation without a z in it. :p
 
You must log in or register to reply here.
Back
Top