***Official Multi-GPU support available***

[Xilikon]

Now that Scott LeGrand got the clearance to spill the beans, the newest 1.07 core allow you to run multiple GPU but with some conditions. The post in question can be found in : http://foldingforum.org/viewtopic.php?f=43&t=3692.

Subject says it all... I've just been given clearance to spill the beans...

But some caveats upfront..

1. Device 0 *must* be a supported CUDA and DirectX display device. It's a client quirk, and there is a workaround in development, but for now, those are the rules.

2. If all your GPUs are CUDA devices and they are all extended onto the desktop, you can just use -gpu n where n is the display device number and it will work just like an ATI client, which means follow the ATI multi-GPU FAQ and you'll be folding away.

3. In the event that 2 above is not the case, there is an override. Skip the -gpu flag and instead set the environment variable NV_FAH_DEVICE=n where n is the CUDA device you wish to use. You may find yourself also playing with CPU affinity here to get the right mix, but that's why we provided the power tools here so to speak. And also it provides a way for anyone with a tesla server/board out there to run F@H today.


3. set VAR=value is the proper syntax. If you put spaces in this, it won't work, i.e. set VAR = value is incorrect.

4. You need to set these variables either in a script that launches the client, or before you launch the client. Doing so after starting the client will have no effect until it is stopped and restarted.

I'm sure we're about to see some quirkiness, but that's what betas are all about... Have fun!

To help setup a multi-gpu config, here is a guide used for ATI, which should be the same for NVIDIA : http://folding.stanford.edu/English/FAQ-ATI (official Stanford site, not FCF).

Also, the 1.07 core allow you to set a precise core affinity you want via a environment variable. More details can be found at : http://foldingforum.org/viewtopic.php?f=43&t=3689

A new Nvidia core (v1.07) has been posted at http://www.stanford.edu/~friedrim/FahCore_11.exe. Scott has added support for a new environment variable, NV_FAH_CPU_AFFINITY, that allows the user to directly set the processor affinity mask or set it to zero to disable all affinity settings. The CPU affinity is set with individual bits (it's a mask, not a CPU selector):
setting NV_FAH_CPU_AFFINITY to
1 ==> CPU 0
2 ==> CPU 1
3 ==> CPU 0 and 1
4 ==> CPU 2
8 ==> CPU 3
15 ==> CPUs 0-3
etc

If this variable is undefined, the default behavior is exhibited. A primer on Windows environment variables is at http://www.chem.gla.ac.uk/~louis/software/faq/q1.html. Please test it by manually deleting an existing core. Thanks.

I tested the newest core and by setting the right affinity, you can get back the performance from the 1.05 core while keeping it more stable. However, this require you to play with affinities till you find the best one and my own best is core 0 and 2 (value of 5). If you have any question, shoot it here or in the FCF threads and we will gladly answer. I'm also asking those who have multiple GPU in a box to test the new feature then help others (My own multi-gpu will be coming next week only).

 

[relic]

Sorry, too many morons there.
If you can stomach the profound stupidity, please post the pertinent data.
Linking to that forum is like linking to go4ts3

 

[Xilikon]

Sorry, too many morons there.
If you can stomach the profound stupidity, please post the pertinent data.
Linking to that forum is like linking to go4ts3

Happy now ?

 

[relic]

Happy now ?

Yes.

Thank you my friend.

 

[BassTek]

So I assume if I want to be lazy I just use 15 and assign it to all cores? I'm assuming that would be the same as me going into the Windows process viewer with 1.06 and manually setting affinity to all cores right?

 

[ND40oz]

16 is core 4, is that correct?

32 should be cores 0-7?

I assume we have to have the GPUs running independently, do they still need to have a monitor hooked up to each?

 

[Kendrak]

The first device has to be cuda capable, does this mean my 6800gs can do something?!

 

[Gray_area]

so, uhh, is there any hints on how to do this for people that have no idea how the gpu client works? Should I start off by making a copy of the folding@home-gpu directory and give it a different name? Should I make 2 batch files to change that set var thats in the forum post?
Will the set NV_FAT_DEVICE=n suffice or do I have to change the machine id in the setup also?
Is all that ATI advice on the forums about having to reboot with a 2nd monitor hooked up still pertinent?

I'm going to start messing around, but you can see, I feel like I'm working in the dark, and doubt my ability to luck onto a working configuration.

Time is short Kendrak is closing fast....

Seriously though, thanks in advance for any advice that doesn't cause my computer to combust

 

[lelliott731]

That was the same question I had Kendrak!!! If that's the case, I have a ton of cards that can now start to fold!!!

Thank goodness this is finally officially supported!!!

 

[ND40oz]

I think they're referring to the device ID for the cuda capable statement.

 

[HighYield]

This new environment variable setting seems to work great for the GPU. I set it to spread the load over all the cores and it actually did it. Can we go back to using affinity changer with two instances of SMP or will the changer mess up the GPU client? (32 bit WinXP) Without affinity changer in WinXP the SMP clients are not doing as well (I'm down about 700 ppd on a Q6600@3.2)

 

[Kendrak]

Time is short Kendrak is closing fast....

So you don't want to hear about the 8800gt I have coming in the mail?

 

[Xilikon]

So I assume if I want to be lazy I just use 15 and assign it to all cores? I'm assuming that would be the same as me going into the Windows process viewer with 1.06 and manually setting affinity to all cores right?

16 is core 4, is that correct?

32 should be cores 0-7?

I assume we have to have the GPUs running independently, do they still need to have a monitor hooked up to each?

This new environment variable setting seems to work great for the GPU. I set it to spread the load over all the cores and it actually did it. Can we go back to using affinity changer with two instances of SMP or will the changer mess up the GPU client? (32 bit WinXP) Without affinity changer in WinXP the SMP clients are not doing as well (I'm down about 700 ppd on a Q6600@3.2)

If you set it to 0 or 15, it do the same thing as unticking the box so the core is assigned to all cores. For now, it's 0 to 15 for core 0,1,2,3 and it's possible that extra cores will get bigger values like 16 for core 4, 32 for core 5 and so on (it's always a multiple of 2 and I'll ask friedrim about that). The main purpose of setting the environment var is to be able to assign to specific cores without using a third-party tool and often, this yield best results because I think Windows assign core 0 to foreground tasks and core 3 to background tasks and middle cores doing nothing.

so, uhh, is there any hints on how to do this for people that have no idea how the gpu client works? Should I start off by making a copy of the folding@home-gpu directory and give it a different name? Should I make 2 batch files to change that set var thats in the forum post?
Will the set NV_FAT_DEVICE=n suffice or do I have to change the machine id in the setup also?
Is all that ATI advice on the forums about having to reboot with a 2nd monitor hooked up still pertinent?

I'm going to start messing around, but you can see, I feel like I'm working in the dark, and doubt my ability to luck onto a working configuration.

Time is short Kendrak is closing fast....

Seriously though, thanks in advance for any advice that doesn't cause my computer to combust

I added a link to the ATI FAQ, which contain a guide to setup multi GPU on the same box. Read it then ask question if you are still unsure. I didn't try multi GPU yet (no box with that setup at home yet) so I'm not very sure how exactly it works.

The first device has to be cuda capable, does this mean my 6800gs can do something?!

That was the same question I had Kendrak!!! If that's the case, I have a ton of cards that can now start to fold!!!

Thank goodness this is finally officially supported!!!

I'm a bit unsure about that statement but I guess that if you want to mix NVIDIA and ATI, the first device must be a CUDA device. It's also possible this mean that if you want a video card dedicated for your daily tasks like browsing, watching videos or gaming, you can put it in the same box but the first device must be CUDA and not the second. I'll ask Scott to clarify that point unless he already explained to someone else.

 

[Gray_area]

Haha, no. (kendrak's 8800gt in the mail)

I read the faq, it confused me.
I'm running Vista 32 with the 177.35 official drivers.
I copied the directory, appended a 2 to the name of the 2nd one. I copied the f@h shortcut into the 2nd directory, and changed the target to indicate the new directory. I opened the client.cfg changed machine id from 2 to 3. Then, I opened a cmd line window hand typed in set NV_FAH_DEVICE=0. Started the gpu client like normal. Then hand typed set NV_FAH_DEVICE=1. Then used the new shortcut in 2nd directory. I now have 2 icons in the tray. Just remembered I forgot to disable SLI. doh. But, fahmon shows both clients running. I didn't delete the work, so I assume they are both crunching the same set. The first client showing 4450 ppd, 2nd 2935 ppd. Also, the 2nd gpu is 10 degrees C cooler than the first. Also running 1 instance of smp. Haven't fooled with any affinity.

Suggestions? Should I shut the whole mess down and turn off SLI, I kinda want to let them keep going and see if they'll both get new wu's when they finish.

This can't be the easiest way to go about this.

edit cuz a post showed up between mine and kendrak's

 

[Xilikon]

I suggest you shut off SLI then restart to make sure everything works as is. You did it correctly but you can also make 2 batch files with those lines :

set NV_FAH_DEVICE=1
"C:\Program Files (x86)\Folding@home\Folding@home-gpu\Folding@home.exe" -gpu 1

Then make 2 shortcuts and it's as easy as clicking on them

 

[Gray_area]

Hmm, now I'm worried. I made a copy of the appdata/roaming directory, not the f@h in my program files directory.
Also, from what I read on the forum link you posted:
"3. In the event that 2 above is not the case, there is an override. Skip the -gpu flag and instead set the environment variable NV_FAH_DEVICE=n where n is the CUDA device you wish to use. You may find yourself also playing with CPU affinity here to get the right mix, but that's why we provided the power tools here so to speak. And also it provides a way for anyone with a tesla server/board out there to run F@H today."

I have no monitor hooked up to my 2nd card, don't have desktop stretched, so I didn't use the gpu flag.
gpu 1 is holding fast at 4450, gpu2 is down to 2378. I'll try to disable sli and start up again. But I'm concerned if I made a copy of the right directories.

 

[Xilikon]

copy the program files folder as well since you want 2 distinct executables.

 

[Gray_area]

ok. just turning off sli didn't change anything. When I start original, gpu climbs 10degrees. When I start 2nd, gpu stays the same. Fahmon shows that work is being done, I'm just not sure what/how.

As to the directories/batch file thing. What seemed simple is now confusing me. If I use your batch file, how does the gpu core know to look in the roaming/appdata folder for the work. The vista shortcut has that folder in the startin line. How does 1 know where 1's data is, and 2 know where 2's data is?

 

[Xilikon]

ok. just turning off sli didn't change anything. When I start original, gpu climbs 10degrees. When I start 2nd, gpu stays the same. Fahmon shows that work is being done, I'm just not sure what/how.

As to the directories/batch file thing. What seemed simple is now confusing me. If I use your batch file, how does the gpu core know to look in the roaming/appdata folder for the work. The vista shortcut has that folder in the startin line. How does 1 know where 1's data is, and 2 know where 2's data is?

Actually, you have a good point but here's a good workaround : Put the same first line in the batch fine but instead of the path I pasted, point to the shortcut (it will act like if it click on it).

 

[Gray_area]

ok, well, I just noticed it had been saying beta6 not beta8, so I uninstalled and reinstalled again.

No matter which client I start, gpu2 is the only one heating up. I'm using 0 and 1 in the fah_device variable. any ideas? was it maybe multitasking 2 cores on 1 gpu earlier??

I tried putting -gpu 1 after the .exe" in the shortcut, just gives me unstable machine error and crashes.
I never did install a 2nd monitor because it looked like it was working at first. Do you think that needs to be done even with the var? I was thinkin not because they mentioned it would work on tesla's (which I'm assuming is that machine with a ton of 8800's that aren't video boards)

 

[Xilikon]

Wait tomorrow since I will attempt to write a good setup guide for the [H]orde. I'll try to gather more data about that along with any tips and tricks

 

[Gray_area]

ok, thanks for all the help, and sorry about turning this thread into a "help gray_area" from a general info.

I'm gonna give up for tonight anyways. Just noticed even with new install they both say 1.06, not 1.07. Even after the reinstall,and I went back and deleted fahcore 11.exe no help.

Here is a question though, right now, I'm running 2 gpu clients. From my temperature monitor they are both running on 1 gpu. The first client didn't really lose any points, and the 2nd client is generating 2k ppd. Whats the deal? Can anyone run 2 clients? More?? Should I not be doing this for some reason?

 

[smileyscout]

Well I think I ghetto rigged it but I have both of my 8800 GTs running.

1. Hooked up a second monitor because I didn't want to mess with environment variables.

2. The GPU client was complaining about some noob things I did so I deleted all traces of the GPU, both the folder in appdata/roaming/folding@home-gpu and /program files (x86)/folding@home/folding@home-gpu

3. Manually download core version 1.07 from http://www.stanford.edu/~friedrim/FahCore_11.exe

4. Copied the /program files (x86)/folding@home/folding@home-gpu to /program files (x86)/folding@home/folding@home-gpu 2

5. I tried to start a single client with a shortcut (flags -gpu 0 -local) and the client failed with a cannot find cudart.dll error.

6. Copied the three dll files from appdata/roaming/folding@home-gpu to /program files (x86)/folding@home/folding@home-gpu and /program files (x86)/folding@home/folding@home-gpu 2

7. The single client started fine.

8. Set the flags on the second client to -gpu 1 -local and it started fine (at first I tried -gpu 2 per the instructions on the ATI multi GPU FAQ (id equals display number -1) but that caused the client to return mdrun_gpu returned -1)

Right now without messing with any affinity setting one fahcore_11.exe is 2 and the other is 3. The two GPU clients are getting 9796 PPD according to FahSpy (4869 and 4927)

I also use RivaTuner to control my cards from speed. With both cards at 70% (a mostly tolerable squeal) both are at a constant 75 C according to Nvidia Monitor.

I started up one SMP client (Project 2665 2,405,11) but it hasn't finished a frame in at least 20 minutes. I need to play with affinity settings I think.

EDIT: Average time for two frames was 23:38 and 1170 PPD

Q6600 Stock
4 GB Memory
2 x MSI NX8800GT
177.35 with hacked INF
Windows Vista x64

 

[Xilikon]

The first device has to be cuda capable, does this mean my 6800gs can do something?!

After asking and receiving a confusing answer, I looked around and scratched my head but I now understand it. No, all cards must be CUDA-capable to fold so no 7 series or below unfortunately.

What the statement really mean is that in those cases, it won't work :

-using the onboard video as primary device and a CUDA card as secondary device. You must set so the CUDA card must be the first device and that sucks since you don't really want to use it for desktop as well.

-Using a Tesla card as first device. This Tesla card is a CUDA card but not DirectX so it must not be the first device, only second or more with a DirectX enabled card as first device.

-(unsure but possible) Mixing ATI and NVIDIA cards on the box. The ATI card cannot be the first device since it's not a CUDA card so it must be second or more.

So in the end, unless you run those kind of setup, just ignore that statement for now. A future client update will fix this.


About the multi-GPU, there is some confusion with the guide and NVIDIA's infos. It seems that if we use the -gpu 0 -locale and -gpu1 -locale flags, we don't need to use environment variables so it will be simpler that way now that beta 8 include a advanced parameters field to put those flags. I'll do more research about that since lots of people is still confused. I hope the whole day today will bring some success stories and a better idea how to setup.

 

[Syribo]

Ahhh, I want to do this SO badly.. It will be so nice to have both of my cards folding, but even after reading the 'instructions' I'm still confused!

I was going to follow smileyscout's steps but... I don't have another monitor to hook up- Well, I DO have my old Dell CRT that's currently being used on my old Dell (Where the text-only console takes days to finish one WU of course )... Is it really difficult to set both GPU2 clients up with just one monitor on the system?

 

[Tigerbiten]

I was going to follow smileyscout's steps but... I don't have another monitor to hook up- Well, I DO have my old Dell CRT that's currently being used on my old Dell (Where the text-only console takes days to finish one WU of course )... Is it really difficult to set both GPU2 clients up with just one monitor on the system?

I've read up on this.
For this to work, Windoze must see the second card.
For this to happen, the second card must have an output for Windoze to expand onto.
So a second monitor is needed.
It can be just the second input on the same monitor.
So both cards need to output to something.
A single output wont work.

This is a Windoze thing.
nVidia could change the drivers so only a single output was needed but then they be non-WHQL drivers, so that wont work.

Luck .........

 

[Tigerbiten]

Just found This.
Its how to make a dummy monitor plug, which will fool Windoze into thinking theres a monitor attached to the vid card socket.

Make one of them.
Plug it into your second vid card.
Fold with two clients, on two vid cards, useing one monitor.

Luck ............

 

[Syribo]

Just found This.
Its how to make a dummy monitor plug, which will fool Windoze into thinking theres a monitor attached to the vid card socket.

Make one of them.
Plug it into your second vid card.
Fold with two clients, on two vid cards, useing one monitor.

Luck ............

Ahh too difficult for me heh. So you ALWAYS have to have the second monitor, or just the second output, connected to the computer in order for it to fold on both cards?

 

[Gray_area]

Well, I guess I misunderstood the purpose of the set variable, I thought it meant you no longer need a 2nd monitor. I'll try and scrounge up a cable and hook my 2nd card into a secondary input on my monitor and try the whole mess again. Hopefully Xilikon can translate all that mess into something I can understand.

I'm still curious about 2 things though, how do you switch to 1.07 core, mine is staying on 1.06 even with the beta 8. And, I'm apparently running 2 clients on 1 gpu and getting more points than when I ran just 1, whats the deal with that?

 

[Tigerbiten]

Download 1.07 from http://www.stanford.edu/~friedrim/FahCore_11.exe.

With Windoze XP you only need the monitor for the inital setup of the client.
Once its running you can unplug the second connection and Windoze will remember your setting through a reboot.
I did that with dual GPU1 clients.

I have not tried this with Vista so I dont know if Vista will remember the setting trough a reboot.

Luck ...........

 

[Xilikon]

With Vista, you need a monitor hooked to each GPU but for those with a LCD, you should have a VGA and DVI port. Hook both of them to each GPU (DVI with the primary display obviously). That's what Scott said to fix this until a future update fix that issue.

 

[Gray_area]

Thanks for all the help. I got the 1.07, and stretched a cable from another monitor to my vid card. Started up both clients again, this time both vid cards heated up to 62. FAHMON says they are both generating over 4k ppd, but I'll wait and see what really happens tomorrow.

I still wonder about running multiple clients on 1 video card. Maybe FAHMON wasn't calculating the ppd right, but it claimed it was making 6k ppd running 2 clients on 1 card.