Quad Opteron 6166 HE Folding Information

P

pjkenned

[H]ard|Gawd
Joined
Jan 8, 2010
Messages
1,971
As some may have read, I purchased a total of ten dodeca cores recently and have a few motherboards to test for ServeTheHome. The process has been yielding interesting results.

Performance
I built the box and downloaded the first WU at 11:58PM yesterday, and had the first bigadv uploaded at 2:27PM. After that first WU finished, I switched over to the Kraken and saw something at least remotely interesting:

AMD_6166_8minTPF.PNG


While the above may be exciting, it still gets destroyed by the 2684. Coincidentally the dual 6164 HE also had one:

AMD_6166_2684.PNG


Overall, ouch. -115K ppd from 2684 :-(

Power Consumption
For power consumption, the Kill-a-Watt is showing 430-436w using a Corsair AX850 PSU. Next time you hear about someone "needing" a 1kw PSU to power a monster CPU... this is a 48 core, 16 DIMM (of 32 possible) machine with an onboard LSI SAS2008 enabled running a SSD.

Cooling
I am using Dynatron A6's on both the quad and the first dual dodeca rig. With motherboard PWM fan control, the noise is actually not bad at all.

Although I have four Dynatron A6's cooling this boxen, and Noctuas on the way, I also ordered four Dynatron A1's. They are solid copper 1U heatsinks and very heavy (~620g with no fan), or about as much as the Coolermaster Hyper 212 with fan. Of course, with a passive 1U heatsink, one needs to either have a 1U chassis (preferably with ducted airflow) or a 2U chassis with low ducting. After receiving the heatsinks I stuck a 80mm fan atop and was surprised to see that the center of the screw holes lined up between the outermost fins and the next most outermost fins.
Dyantron_A1_80mm.jpg


I am mocking this up right now with a 3 pin fan (will order 80mm PWM fans soon). Any 80mm PWM fan suggestions are welcome.

BTW for pics/ a bit more info on the build and some of my other Opteron 6100 builds: http://forums.servethehome.com/showthread.php?194-AMD-Opteron-6100-Series-Builds

I have been doing a lot with the Opteron 6100's so if people are interested I can keep posting info as I learn.
 
Last edited:
wow 8 min flat tpf, thats pretty awesome
 
That's is a bunch of CPU power at such a low wattage.

Nice!
 
I am liking the looks of this rig, pj! 620 ppd/W is absolutely amazing!
 
Loving it.

Thinking of getting one... oh that's right, I spent my power budget for the year already... doh.
 
koven said:
what's the approximate cost of that rig?
Click to expand...

Well north of $2K. Figure you spend 4x on everything but the SSD and power supply, and the cheapest quad G34 is about $720 shipped without tax (arguably x4 of a nice LGA 1155 board.)

You need 4:
  • CPUs
  • CPU Coolers (figure $140+)
  • 4 Sicks of RAM per CPU

The big issue with the quad is that the initial outlay is so high. If one cannot absorb those costs, one probably is much better off with a SR-2 or even i7-2600K boxen and building up from there. A 4.4GHz Core i7-2600K is 56K ppd anyway so for pretty much the same wattage (assuming four of them) you get 224K ppd.
 
That is some insane efficiency, if you front the initial cost of such a system. I can just imagine what might happen with the Interlagos 16 cores are finally released.
 
man, you are one crazy motherfolder

really nice rigs.

/looks at SR2 with disdain
 
Kendrak said:
Wow, it is just odd to see that, but the days of the SR-2 as king are gone.
Click to expand...
48 core AMD will soon be king but I believe Patrick will be showing numbers soon that show the SR-2's will still beat 2p 6 and 12 core AMD configs. We were talking about it on IRC this morning.
 
jebo_4jc said:
man, you are one crazy motherfolder

really nice rigs.

/looks at SR2 with disdain
Click to expand...

So I made a spreadsheet that has costs for different 8 and 12 core in 1P, 2P and 4P G34 configs. I included the second hand G34 CPU prices. I then used a $1250 setup (SR-2 L5640/SR-2/6GB) used price based on musky's post and with not super high clocks for 140,000PPD on 6901's. For both setups I added a gold CPU, coolers, and etc as necessary to complete an open-air build.

PPD - 6901 / Cost
Model 1P 2P 4P
Opteron 6164 27.18 45.34 67.05
Opteron 6166 28.38 46.99 69.33
Opteron 6172 34.33 56.45 83.13
Opteron 6128 27.15 49.16 74.60
Opteron 6134 27.46 47.35 70.72
Opteron 6124 21.81 38.86 58.65
SR-2 - 92.11 -

Given the linear scaling we have seen this should be close. Basically, the SR-2 used setups still win on PPD/$ side once you add in all of the extras.

I can post more after lunch if anyone is interested.
 
Kendrak said:
Wow, it is just odd to see that, but the days of the SR-2 as king are gone.
Click to expand...

The funny thing about this, is that we are talking about a cpu architecture that is now arguably 2 full generations behind. I cannot help but imagine how drastically the cpu folding landscape would be different if Bulldozer had not been delayed like this.
 
Tobit said:
48 core AMD will soon be king but I believe Patrick will be showing numbers soon that show the SR-2's will still beat 2p 6 and 12 core AMD configs. We were talking about it on IRC this morning.
Click to expand...

Who's Patrick? SR-2 is still a great folding box, and considering the cost of cheap E5645 chips even at retail.

48-Core IS king, baby. Hail to the king :p


pjkenned said:
Well north of $2K. Figure you spend 4x on everything but the SSD and power supply, and the cheapest quad G34 is about $720 shipped without tax (arguably x4 of a nice LGA 1155 board.)

You need 4:
  • CPUs
  • CPU Coolers (figure $140+)
  • 4 Sicks of RAM per CPU

The big issue with the quad is that the initial outlay is so high. If one cannot absorb those costs, one probably is much better off with a SR-2 or even i7-2600K boxen and building up from there. A 4.4GHz Core i7-2600K is 56K ppd anyway so for pretty much the same wattage (assuming four of them) you get 224K ppd.
Click to expand...

For me it was a bit easier because I bought two chips first, and then the two Black Editions, and the Super Micro H8QGi-F board. But I agree on that front.

It almost equals the purchasing cost of two SR-2s with Xeon e5645s (x4) and associated required components, but with slightly lower PPD, but MUCH lower power usage, and I think that's when things start to even out a bit at some point.

Good job though, and ENJOY.

I await BullDozer (still) :(
 
10e said:
Who's Patrick? SR-2 is still a great folding box, and considering the cost of cheap E5645 chips even at retail.

48-Core IS king, baby. Hail to the king :p

For me it was a bit easier because I bought two chips first, and then the two Black Editions, and the Super Micro H8QGi-F board. But I agree on that front.

It almost equals the purchasing cost of two SR-2s with Xeon e5645s (x4) and associated required components, but with slightly lower PPD, but MUCH lower power usage, and I think that's when things start to even out a bit at some point.

Good job though, and ENJOY.

I await BullDozer (still) :(
Click to expand...

I am a Patrick :)

The above is totally correct. DP systems though are still behind the SR-2 except in power consumption. I have one folding and once the new PSUs get here a second and third DP G34 box will be set up with the dodeca HE's on hand. The jump from 24 to 48 core makes the 48 core reasonable and really fast thanks to extra bonus points. From a price perspective, two SR-2's are probably the way to go for a lot of people.

Interlagos should be awesome, but the question is whether it will be worth the price to upgrade.
 
Tobit said:
pjkenned :p
Click to expand...

Oh that Patrick lol. I thought you were talking about your toooobeta connections or something for some inside scoop. Blonde moment, I had.

pjkenned said:
I am a Patrick :)

The above is totally correct. DP systems though are still behind the SR-2 except in power consumption. I have one folding and once the new PSUs get here a second and third DP G34 box will be set up with the dodeca HE's on hand. The jump from 24 to 48 core makes the 48 core reasonable and really fast thanks to extra bonus points. From a price perspective, two SR-2's are probably the way to go for a lot of people.

Interlagos should be awesome, but the question is whether it will be worth the price to upgrade.
Click to expand...

Oh this Patrick ^^ :D

Strangely enough, up here the Xeon e5645 seems cheaper than the U.S., which is a momentous occasion. I can get them for $542.00 each Canadian, which is about $300.00 less than I paid for my x5650s last year. That's what I get for being impetuous.

From a power consumption point-of-view, the 6166 chips should be quite low. My 6168s in the Asus KGPE-D16 at full load were only 303-308 watts according to Mr. Kill-A-Watt so the 6166s should be lower.

How the heck did you get ten of them and where though? Do these same people have 6168s for cheap too? This is my only dilemma: Wait for BD/Interlagos or get two more 6168 retails, put them in the Supermicro and then use the Black Editions in the Asus and crank 'em up like Patriot did.

Decisions decisions.
 
Last edited:
I will probably be buying the 61XX chips off you all as you upgrade :p
 
of note... 2.5-2.8 seems the limit for black editions... I happened upon some golden chips to be able to breach 3...
4p uses less power than my 2p...570w here at 3.1 for 175k...
kinda loses the economical fight.... now if only we could combine and have 4p @ 2.8 or 2.9 lol.....switch who it folded for every month....
 
Wow crazy. Remind me what clockspeed those chips are?
Posted via Mobile Device
 
1.8ghz x 48 cores - This is by far the fastest 6901 this rig has seen though so not typical.
 
From those pics on STH, you're running unregistered & non-ecc desktop dimms with no compatibility problems right? Overall cheaper cause you need 4 dimms per socket, but what speed, ddr3 1333? Also with interlagos coming out in a few months, would it be possible to run ddr3 1600 dimms at 1333 in current boards with 6100 series processors, and later just upgrade to the 6200 series processors w/ bios upgrade? IIRC interlagos can support ddr3 1600. Correct me if im wrong on anything, thanks for the info.
 
Opty1218 said:
From those pics on STH, you're running unregistered & non-ecc desktop dimms with no compatibility problems right? Overall cheaper cause you need 4 dimms per socket, but what speed, ddr3 1333? Also with interlagos coming out in a few months, would it be possible to run ddr3 1600 dimms at 1333 in current boards with 6100 series processors, and later just upgrade to the 6200 series processors w/ bios upgrade? IIRC interlagos can support ddr3 1600. Correct me if im wrong on anything, thanks for the info.
Click to expand...

I just nabbed 16gb of ddr3 1600 cas 9 ripjaws x
moved from 17 to 54% since the swap...
they are running cas 8 ddr3 1333

you have no settings in the bios to tweak ram... so if you want cas 7 at 1333 you better get cas 8 ddr3 1600 ... I am gonna have to reprogram my spd on my ram to get cas 7.... hopefully thats stable...
 
Opty1218 said:
From those pics on STH, you're running unregistered & non-ecc desktop dimms with no compatibility problems right? Overall cheaper cause you need 4 dimms per socket, but what speed, ddr3 1333? Also with interlagos coming out in a few months, would it be possible to run ddr3 1600 dimms at 1333 in current boards with 6100 series processors, and later just upgrade to the 6200 series processors w/ bios upgrade? IIRC interlagos can support ddr3 1600. Correct me if im wrong on anything, thanks for the info.
Click to expand...


interlagos will support ddr3 1600 natively but will also support ddr3 1333 and 1066.
 
I think musky will have his build up soon if it isn't already and that should yield more data.

In the meantime, I put together a dual 6166 HE build yesterday and somehow got really lucky with both the dual boxes and the quad getting 6901's. The 6164's are only 1.7GHz chips (versus 1.8), but this should show the efficiency gains of going 4P fairly well in terms of PPD. Also of note, the scaling is really good in terms of TPF.

AMD_6166_quad_v_two_dual_6901.PNG


Hopefully within a week or two I will get to make a third dual opteron box, but this should present some decent data for folks.
 
What kind of performance hit would you take if you switched from 4x2GB per socket, to 2x4GB per socket? I notice on cpu-z screenshots, g34 boards still only show dual-channel mode on the memory tab. Now is that a detection error where it really is quad-channel, or is it AMD marketing saying 2 dual channels = quad channel?
 
G34 = 2x c32 dies per chip

so you take 2 hexacores that are dual channel under one hood you get dual dual channel per cpu or quad channel...

so yes ;)
and the performance hit is significant...because each die gets its bandwidth cut in half...severe bottleneck... at 1.7 I think the difference is 3-5minutes
 
Patrick,

Have you tried any software OC tools for the AMD systems?

Bet you could get those things to 3GHZ :)
 
nitrobass24 said:
Patrick,

Have you tried any software OC tools for the AMD systems?

Bet you could get those things to 3GHZ :)
Click to expand...

What did you have in mind? These are not ES (unlocked) chips, so the only thing available would be whatever the AMD equivalent of bclk is. Server board tend to not like that.
 
You must log in or register to reply here.
Back
Top