• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

quad g34

mashtub651

Limp Gawd
Joined
Nov 4, 2010
Messages
200
I was considering getting a quad g34 mobo and running one opteron 6128 and then loading up when the 6200 series came out and running linux does anyone have any experience with these systems and how does smp scale to 32 or 64 cores
 
at the present I dont think the F@H client scales well past 24 cores/threads no matter which OS you run due to deminishing returns. I'm a unix/linux professional by day and eagerly looking forward to the return of linux -bigadv but at the moment it's -smp only right now.

I've had a few 24 core linux boxen running -smp and the TPF's were in the low 1.xx minute area on most of the -smp cores. I'm not sure if any of the 32+ core/thread folks have ever tried linux -smp to see how it works.

once linux bigadv is back I'll take all my boxen back to linux and fold away into the sunset, right now my most powerful rigs are running windows for bigadv, the rest run linux for smp
 
Reminds me of the octo-socket build I tried... Good times lol
Posted via [H] Mobile Device
 
48 core rig thread here - it was the fastest rig in [H] while it was folding for us.

http://hardforum.com/showthread.php?t=1562298

10e gracefully took it elsewhere so I could reclaim the title with SR2#3 :p

Always about you isn't it? Hope the rain hasn't seeped into your head (kidding!). She'll be back now that my other team has reached their billion points. Sounds paltry doesn't it ;)

To answer the OP, folding@home scales almost linearly with more cores on the Magny-Cours. My times-per-frame are almost exactly 50% of what they were with 2 chips/24-cores.

My 24 core rig had a TPF on 2684s of around 27 - 28 minutes and now, with 24 more cores it went down to the 13:50 - 14:00 minute range.

The other 268x units show similar gains. Only 6900s I'm not sure about because the old 24-core rig with chips at stock speeds never ran one.
 
Always about you isn't it? Hope the rain hasn't seeped into your head (kidding!). She'll be back now that my other team has reached their billion points. Sounds paltry doesn't it ;)

To answer the OP, folding@home scales almost linearly with more cores on the Magny-Cours. My times-per-frame are almost exactly 50% of what they were with 2 chips/24-cores.

My 24 core rig had a TPF on 2684s of around 27 - 28 minutes and now, with 24 more cores it went down to the 13:50 - 14:00 minute range.

The other 268x units show similar gains. Only 6900s I'm not sure about because the old 24-core rig with chips at stock speeds never ran one.
Are you going to make it into a 64-core monster when Interlagos is out? o_0
 
Are you going to make it into a 64-core monster when Interlagos is out? o_0

There's always a possibility, but it will depend what the chips cost. I know that I would be able to easily sell my "extra spicy" black edition 6168s for what I paid for them, but I'm not sure if it will be worth it.

The other option is getting two of the Interlagos chips and put them in my Asus KGPE-D16 if the BIOS gets updated to support them, and make a 32 core SkankBox.
 
Always about you isn't it? Hope the rain hasn't seeped into your head (kidding!). She'll be back now that my other team has reached their billion points. Sounds paltry doesn't it ;)

To answer the OP, folding@home scales almost linearly with more cores on the Magny-Cours. My times-per-frame are almost exactly 50% of what they were with 2 chips/24-cores.

My 24 core rig had a TPF on 2684s of around 27 - 28 minutes and now, with 24 more cores it went down to the 13:50 - 14:00 minute range.

The other 268x units show similar gains. Only 6900s I'm not sure about because the old 24-core rig with chips at stock speeds never ran one.

You took my dream rig to another team? :(

Oh well, at least you plan to bring it back!
 
once linux bigadv is back I'll take all my boxen back to linux and fold away into the sunset, right now my most powerful rigs are running windows for bigadv, the rest run linux for smp

Why aren't you using wine to run big advanced units?

It seems to run fine.
 
She's coming back. Turn that frown upside down, amdgamer.

I'll turn it upside down when it comes back. However i'm actually excited about the possibility that your rig may very well be the first one to test and see if F@H scales very well upto 64 physical cores. Your current rig has already proven that it scales extremely well upto 48 cores when they are all real physical cores.
 
Always about you isn't it? Hope the rain hasn't seeped into your head (kidding!). She'll be back now that my other team has reached their billion points. Sounds paltry doesn't it ;)

Yes, me me me me me and me. Thanks for the thought, but I am living in one of the few places in eastern Australia not underwater.

Did you realise you could have switched all your clients over to [H] and added an insane amount of fun to the race I was having with Musky the last few weeks (before he went and spoiled things by adding 2 SR2s in a week pfft) And if we could rustle up a Frenchman capable of about 440K ppd we could have had a grand slam of dueling with you being Wimbledon... me Aus Open etc etc
 
its settled than, the next build after i finish my SB build will be the quad system, im looking at the SM H8QGi-F

so 10e what mobo are you running
 
damn I thought my tyan quad was pricey! (but with the add on board it was over a grand so I guess it makes sense)
 
Yes, me me me me me and me. Thanks for the thought, but I am living in one of the few places in eastern Australia not underwater.

Did you realise you could have switched all your clients over to [H] and added an insane amount of fun to the race I was having with Musky the last few weeks (before he went and spoiled things by adding 2 SR2s in a week pfft) And if we could rustle up a Frenchman capable of about 440K ppd we could have had a grand slam of dueling with you being Wimbledon... me Aus Open etc etc

Yes I saw that. I was lurking a bit here, but didn't get a chance to comment. I was out folding you both last week (add three "e's" to my username for the other folding man-mistress, err... account) until musky upgraded another SR2 courtesy of Tobeat who can go "finger" himself in a mirror :p

Glad to hear that you and your SR-2, and human family are not waterlogged :D It's crazy with the flooding there. Happy summer.

We can always have more races in the future. I think I can add a couple of clients and bitchslap that musky back into humility LOL. Just kidding, couldn't resist. Challenge of the three folding chimps or something. Dunno about the french dude being interested in that though.

My wife loves the computer-part wallpaper in the basement.

I'll turn it upside down when it comes back. However i'm actually excited about the possibility that your rig may very well be the first one to test and see if F@H scales very well upto 64 physical cores. Your current rig has already proven that it scales extremely well upto 48 cores when they are all real physical cores.

Definitely looking forward to finding that out if I can. The old A2 SMP folding binaries didn't scale properly, but A3 is looking good, and I believe Pande Group is looking at that.

She's gonna be back, but for some reason I have to actually hook up a monitor/keyboard to her because ever since I've replaced my old dying D-Stink DIR-655 router with an equally crappy StinkSys E3000, she only seems to respond to my Win7 clients on IPv6, not at all over RDP, and the built-in KVM is awol too. But she's still folding so it's something to do with the routers. Consumer junk routers.

damn I thought my tyan quad was pricey! (but with the add on board it was over a grand so I guess it makes sense)

Price of entry is not cheap, so your memory is bang on. Things don't change
 
Yes I saw that. I was lurking a bit here, but didn't get a chance to comment. I was out folding you both last week (add three "e's" to my username for the other folding man-mistress, err... account) until musky upgraded another SR2 courtesy of Tobeat who can go "finger" himself in a mirror :p

Glad to hear that you and your SR-2, and human family are not waterlogged :D It's crazy with the flooding there. Happy summer.

We can always have more races in the future. I think I can add a couple of clients and bitchslap that musky back into humility LOL. Just kidding, couldn't resist. Challenge of the three folding chimps or something. Dunno about the french dude being interested in that though.

My wife loves the computer-part wallpaper in the basement.



Definitely looking forward to finding that out if I can. The old A2 SMP folding binaries didn't scale properly, but A3 is looking good, and I believe Pande Group is looking at that.

She's gonna be back, but for some reason I have to actually hook up a monitor/keyboard to her because ever since I've replaced my old dying D-Stink DIR-655 router with an equally crappy StinkSys E3000, she only seems to respond to my Win7 clients on IPv6, not at all over RDP, and the built-in KVM is awol too. But she's still folding so it's something to do with the routers. Consumer junk routers.



Price of entry is not cheap, so your memory is bang on. Things don't change

Out of curiosity, were you ever able to find a case for that massive motherboard? Or you are still folding it naked?
 
how is your scaling on the quad g34... I have a dual g34 running 24cores at 1.7ghz...
my ppd is 14k with one client smp -24 or 30k with dual smp -12 clients...

a3 smp on rhel5u5
 
Out of curiosity, were you ever able to find a case for that massive motherboard? Or you are still folding it naked?

Not yet. The difficulty with this board is that the I/O shield is not the "edge" of the board, because there is RAM next to it that extends the board in that direction. The more I research it, it seems like a modified HAF 932 will do the trick, as my Lian Li PC-P80 doesn't have this ability to allow the motherboard to extend past the I/O shield.

Might get on that very soon though.

how is your scaling on the quad g34... I have a dual g34 running 24cores at 1.7ghz...
my ppd is 14k with one client smp -24 or 30k with dual smp -12 clients...

a3 smp on rhel5u5

Scaling is almost identically 2 to 1 on all units in terms of times-per-frame, even those dastardly 670x regular SMP A3 units.

My times-per-frame on all BigAdv units went to almost exactly half of what they were with the dual 6168 setup. It used to take just under 2 days (28:00 per frame) to finish a 2684 and now it takes under 1 day, and these are the slowest units with a time-per-frame of 13:50 - 14:00 on average.

The rest of the BigAdvs get a time-per-frame of 10:30 - 10:45. These would get between 22:00 and 24:00 depending on 2685/2686//2692/6900.
 
I figured I'd piggyback on this thread since its relevant. What about a dual G34 with 16 or 24 cores? For the money you give up quite a bit of clockspeed but can gain cores. Does anybody have any thoughts on a setup like this vs an overclocked i7 quad? I was thinking about a dedicated 2600k bigadv folder, but with p67's problems, that has gone by the boards for now, and a big multicore opteron rig sounds like an even more fun project
 
As far as a cost of operation, SB is hard to beat on a ppd/watt basis.

Bigadv clients scale quite well with more more cores. So if you can find amd # for bigadv on similar chips you can do an easy calculation.......

# of cores x speed = total ghz
divide that by ppd

That will give you a ppd per ghz

Then you can look at any similar amd system and go up and down in speed or cores and have a ruff idea on what you will get.

This only works well with bigadv btw.
 
Based on 10e's numbers for a 24 core it looks like 35k-40k ppd doing bigadv. Anybody else have feedback on this idea? The other big-money alternative is doing a dual quad SR-2 rig for ~300$ more, while the realist in me thinks I should just wait on the 2600k's to come back and do up one of those rigs for 700$ and being smart with the cost difference :p
 
Based on 10e's numbers for a 24 core it looks like 35k-40k ppd doing bigadv. Anybody else have feedback on this idea?
Yeah. For that much PPD, it's not worth the investment or the power consumption. There are definitely better options. It's been discussed to death, but if you can get your hands on Sandy Bridge hardware, that can't be beat for PPD/$ or PPD/W. The defect is really a non-issue at the moment, since it won't affect anything until a few years down the line, can be bypassed if you only use the two SATA 6Gbps ports, and if you get a board now, you can just get it replaced by a fixed version when they're available. Failing that, you could put together an i7 rig for a few hundred, or, as you mentioned, you can go with an SR-2 setup (FLECOM's even selling one right now).

In short, I don't think a dual G34 rig is a good idea, from a practical or a financial point of view.
 
We have a guy in our team with a 48 core rig running 6176se's and win 2008r2 enterprise server

Project: 6900
Average time/frame: 00:09:03 . 223k ppd
CPU: opteron 6176se @ 2.3 GHz
# of CPU sockets: 4
# of Physical cores: 48

# of FAH CPU processes:1
# of FAH GPU Clients:0

RAM GB installed: 32gb Quad channel
RAM Type: DDR3
RAM Speed:1333 9.9.9.24
OS / Linux kernel:Win 2K8r2 enterprise
Running in VM: No
 
We have a guy in our team with a 48 core rig running 6176se's and win 2008r2 enterprise server

Project: 6900
Average time/frame: 00:09:03 . 223k ppd
CPU: opteron 6176se @ 2.3 GHz
# of CPU sockets: 4
# of Physical cores: 48

# of FAH CPU processes:1
# of FAH GPU Clients:0

RAM GB installed: 32gb Quad channel
RAM Type: DDR3
RAM Speed:1333 9.9.9.24
OS / Linux kernel:Win 2K8r2 enterprise
Running in VM: No

Dam in 4 days he would put out more points then i have.....brb think i have to go change my pants
 
But how much would a rig like that cost?
 
$4400 in just cpus according to ebay. Not to mention all the other stuff you would need.
 
We have a guy in our team with a 48 core rig running 6176se's and win 2008r2 enterprise server

Project: 6900
Average time/frame: 00:09:03 . 223k ppd
CPU: opteron 6176se @ 2.3 GHz
# of CPU sockets: 4
# of Physical cores: 48

# of FAH CPU processes:1
# of FAH GPU Clients:0

RAM GB installed: 32gb Quad channel
RAM Type: DDR3
RAM Speed:1333 9.9.9.24
OS / Linux kernel:Win 2K8r2 enterprise
Running in VM: No

Absolute insanity

Yeah. For that much PPD, it's not worth the investment or the power consumption. There are definitely better options. It's been discussed to death, but if you can get your hands on Sandy Bridge hardware, that can't be beat for PPD/$ or PPD/W. The defect is really a non-issue at the moment, since it won't affect anything until a few years down the line, can be bypassed if you only use the two SATA 6Gbps ports, and if you get a board now, you can just get it replaced by a fixed version when they're available. Failing that, you could put together an i7 rig for a few hundred, or, as you mentioned, you can go with an SR-2 setup (FLECOM's even selling one right now).

In short, I don't think a dual G34 rig is a good idea, from a practical or a financial point of view.

Yea, true, and sorry to rehash the debate again. The problem is that no reputable dealers are selling p67 boards right now since intel pulled them and if i was thinking about returning it when the fixed parts come out I want to be the original buyer in order to guarantee a return goes smoothly. It would just be cool to have a big multicore setup just for the sheer awesome factor. Plus at first glance the pricing doesn't look insane. Its just a shame that AMD is so non-competitive :(
 
I was inspired by 10e to build a AMD system, with 4x 6176SE chips.

I was patient in acquiring the chips and negotiated the price to around $700/chip. I missed a deal for 6172 chips @ $450/each.

This system uses budget EK AMD waterblocks with custom brackets since G34 isn't yet supported by EK ($1.50/each + my own labor). Air-cooling options are rather limited, and cost difference vs. water-cooling wasn't that significant.

I had a leftover ATCS840 case, and modified that accordingly -- HAF932 may be very similiar in approach.

The memory is budget 16 x 2 gskill UDIMMs rated at DDR3-1333 7-7-7-21-1T @ 1.5v. There is more to do here, like getting the memory to run at the XMP settings, 7-7-7-21-1T, rather than the jedec 9-9-9-24-1T. The supermicro motherboard doesn't allow manual adjustment of memory timings.

I haven't yet measured the power draw -- will update with more detail later.

Re: scaling up to more processors. The limiting factor currently is around 58-60 processors/threads. This is because the folding client is a 32bit binary, and the commit limit of 4GB memory (when running on x64) is hit -- annoyingly, it will be hit when the work-unit is 100% complete when the .xtc file is being allocated/generated. The folding client will then delete the work unit unconditionally, without submitting, on this memory error. Beyond that, the group and affinity limits are hit at 64 processors/threads with a 32bit binary. Stanford will need to produce a 64bit binary, likely later this year...

Gallery with annotated photos and more details:
http://sfield.smugmug.com/Computers/Quad-AMD-G34-6176SE/15725095_JnZK3

1178669439_JuKsJ-L.jpg


1178669483_65nLh-L.jpg


More to do here... Hint: not just the small fan. There's something else there...

1178669517_vXKpV-L.jpg


And more to do here...

1178669563_TwDi7-L.png


1178671470_3U5pY-L.png


1178671628_YVmmk-L.jpg


1178673243_Qum3q-M.jpg


1178673289_yRdRB-M.jpg


Not yet. The difficulty with this board is that the I/O shield is not the "edge" of the board, because there is RAM next to it that extends the board in that direction. The more I research it, it seems like a modified HAF 932 will do the trick, as my Lian Li PC-P80 doesn't have this ability to allow the motherboard to extend past the I/O shield.

Scaling is almost identically 2 to 1 on all units in terms of times-per-frame, even those dastardly 670x regular SMP A3 units.
QUOTE]
 
Last edited:
Congratulations, you've put together the first system I've ever seen that manages to make the ATCS 840 look small.
 
Look at the power delivery sections and how small the PCIe look. Just look at it.
fALIO.jpg
 
Back
Top