How good is a 8P Xeon E7-8870 rig compare to 4P Xeon E5-4650?

A

-alias-

Limp Gawd
Joined
Jul 15, 2012
Messages
374
I am thinking of building a 8P folding rig, so the question is, how good is a 8P Xeon E7-8870 rig compare to 4P Xeon E5-4650?

I can have the 8 x Xeon E7-8870 chips for $4800, and the rest for $7500 ~ all together $12000, but is it worth it compare to 2 x 4P Xeon E5-4650 rigs that is cheaper? Maybe I will get a better price, that I will know tomorrow.
 
Wow - an 8P would be pretty damn impressive especially with those 10-core chips. You'd have a 160-thread beast. Be interesting to see how f@h scales with that many threads. Would it be double, triple or more ppd than your 4P E5 rig? Hard to say really as I doubt f@h has ever really been tested on a machine like that.
 
I don't think FAH scales well beyond 48 threads. Testing has been done in the past and doubt anything has changed recently.
 
Tobit said:
I don't think FAH scales well beyond 48 threads. Testing has been done in the past and doubt anything has changed recently.
Click to expand...

It would be interesting to see what people with 62xx 16c CPUs say about that. Remember the 810x WUs are magical.... I mean special :D
 
I recall a while ago we had a member with a 160 thread machine, and his PPD was very bad (below expectations), and IIRC it was thought to be due to the scaling above a certain number ( as mentioned aobve 48), so much so that running 2x clients side by side provided better PPD (not typically the case)
 
Hi, -alias-. Your price for 8x 8870 CPUs is pretty good! While the price for the rest is just normal.
Some people find in some 64T Xeon systems that the TPF of "-smp 32" and "-smp 64" is close, thus make conclusions that FAH has a scaling problem for beyond 48 threads. However, this is NOT the truth. In fact, with "-smp 32" we have already taken advantage of all physical CPU cores of a 64T Xeon system. It's pretty normal that the TPF of "-smp 64" is only slightly (~10%) faster than "-smp 32" in this situation, and the result has nothing to do with "beyond 48T". Even in a 24T Xeon system we would get the same result, that is, find that the TPF of "-smp 12" and "-smp 24" is close at this time.
As for PPD numbers, I think 1.1M for 8101 and 1.6M for 8102 might be not difficult for a fine-tuned 8P E7-8870 rig, though it is not as cost-efficient as (ES) Xeon E5 rigs.
 
Last edited:
I remember he was running the test in Windows?
Windows does have some scaling issues for large number of threads, however, I think this should not happen in Linux.

W.Feather said:
I recall a while ago we had a member with a 160 thread machine, and his PPD was very bad (below expectations), and IIRC it was thought to be due to the scaling above a certain number ( as mentioned aobve 48), so much so that running 2x clients side by side provided better PPD (not typically the case)
Click to expand...
 
Last edited:
Is it just me? Or does $12,000 for a folding rig appear to be EXTREMELY [H]ARD EPEEN? :eek:
Running 3 x 4P rigs with about $9,000 total invested still can get me to around 2M PPD on 8102s (when they show their pretty little heads :) ).
I suppose my PPD/W wouldn't be as good however.
Just sayin' !
 
I thought we had gotten him to test in *nix as well, its been a long while and I cannot remember tbh, however the scaling would be on the GROMACS core side I imagine (He was using Server 2k8, at the least, on the Windows side)


quickz said:
I remember he was running the test in Windows?
Windows does have some scaling issues for large number of threads, however, I think this should not happen in Linux.
Click to expand...
 
W.Feather said:
I thought we had gotten him to test in *nix as well, its been a long while and I cannot remember tbh, however the scaling would be on the GROMACS core side I imagine (He was using Server 2k8, at the least, on the Windows side)
Click to expand...

I would image the task manager would taken about 2GB of memory just to show all the CPU graphs :D
 
402blownstroker said:
It would be interesting to see what people with 62xx 16c CPUs say about that. Remember the 810x WUs are magical.... I mean special :D
Click to expand...

I have made some performance tests on Opteron 6200s. No scaling issues were found for 64T. In the test I was folding on a 6975 WU. The avg TPF for "-smp 32" is 1m28s, and the avg TPF for "-smp 64" is 0m46s. (DLB on for both cases)
The relative efficiency of 64T vs. 32T is (60+28)/(46*2)=95.6%. This result is not bad, I think.

FAHlog.txt for -smp 32:
[01:28:28] Completed 360000 out of 500000 steps (72%)
[01:29:56] Completed 365000 out of 500000 steps (73%)
[01:31:24] Completed 370000 out of 500000 steps (74%)
[01:32:53] Completed 375000 out of 500000 steps (75%)
[01:34:20] Completed 380000 out of 500000 steps (76%)
[01:35:49] Completed 385000 out of 500000 steps (77%)
Click to expand...

FAHlog.txt for -smp 64:
[01:36:31] Completed 359812 out of 500000 steps (71%)
[01:36:33] Completed 360000 out of 500000 steps (72%)
[01:37:19] Completed 365000 out of 500000 steps (73%)
[01:38:05] Completed 370000 out of 500000 steps (74%)
[01:38:51] Completed 375000 out of 500000 steps (75%)
Click to expand...
 
quickz said:
Hi, -alias-. Your price for 8x 8870 CPUs is pretty good! While the price for the rest is just normal.
Some people find in some 64T Xeon systems that the TPF of "-smp 32" and "-smp 64" is close, thus make conclusions that FAH has a scaling problem for beyond 48 threads. However, this is NOT the truth. In fact, with "-smp 32" we have already taken advantage of all physical CPU cores of a 64T Xeon system. It's pretty normal that the TPF of "-smp 64" is only slightly (~10%) faster than "-smp 32" in this situation, and the result has nothing to do with "beyond 48T". Even in a 24T Xeon system we would get the same result, that is, find that the TPF of "-smp 12" and "-smp 24" is close at this time.
As for PPD numbers, I think 1.1M for 8101 and 1.6M for 8102 might be not difficult for a fine-tuned 8P E7-8870 rig, though it is not as cost-efficient as (ES) Xeon E5 rigs.
Click to expand...

I would agree with this assessment regarding the inefficiency of LGA 1567 HT. I have a 4P E7-8860 system folding, and the performance difference between running 40 threads without HT vs. 80 threads with HT does not reflect the kind of scaling we have come to expect based on our experiences with desktop Nehalem platforms. This was confirmed with synthetic benchmarks and has little to do with an "efficiency ceiling" over a certain number of threads as the trend was noted even in a UP configuration. Theoretical PPD for my machine is around 660k on P8102 and 480k on P8101, all while pulling a continuous 750-780w AC. PPD/W is not its strong suit.

I was fortunate in that I found a way to piece together my E7 box for roughly the same price as a 4P G34 rig and I had fun doing it. If I was looking at a potential $12k pricetag, though, I would certainly be looking into alternatives.
 
Dead Things said:
I would agree with this assessment regarding the inefficiency of LGA 1567 HT. I have a 4P E7-8860 system folding, and the performance difference between running 40 threads without HT vs. 80 threads with HT does not reflect the kind of scaling we have come to expect based on our experiences with desktop Nehalem platforms. This was confirmed with synthetic benchmarks and has little to do with an "efficiency ceiling" over a certain number of threads as the trend was noted even in a UP configuration.
Theoretical PPD for my machine is around 660k on P8102 and 480k on P8101, all while pulling a continuous 750-780w AC.
Click to expand...
PPD/W is not its strong suit.

I was fortunate in that I found a way to piece together my E7 box for roughly the same price as a 4P G34 rig and I had fun doing it. If I was looking at a potential $12k pricetag, though, I would certainly be looking into alternatives.
Click to expand...

I am folding on an equivalent 6274 ES @ 3.0 GHz, see attached HFM.net. I have only had 8101, power draw is 770 from wall. Best regards, Charlie
[URL=http://imageshack.us/photo/my-images/339/hfm8101.jpg/] Uploaded with ImageShack.us[/URL]
 
Thanks so much guys!
I always appreciate the response I get from you all. As I understand it, such a rig as this 8P is not a smart choice for me. 2 x 4P with E5-4650 will provide quite a lot better and be overall less expensive, so I will not go any further with this 8P rig for now. I think that the power draw also will be huge, maybe as much as 1400W from the wall, so no, I will not do this. Thanks again!
 
Hi, -alias-. I think you are right. At the moment we have better choices, if let me choose I would wait for 4P E5 v2 and 8P E7 v2, which will probably be released next year and of cource be more powerful.
However I'm sorry that I still have a question, in your signature I see that you have spent $30000 for "3 x 4P G34 + 2P SR-2 + 4P E5-4650 ~ 204 cores / 248 threads ~ a $30.000 investment".
How is it so expensive? Looks like even much more expensive than 8P E7-8870! Or I have missed something important?
 
Last edited:
But still, there is nothing.... nothing like the bling-bling factor of a 8P box. Even an 8P socket F system is pimpin' :D
 
quickz said:
Hi, -alias-. I think you are right. At the moment we have better choices, if let me choose I would wait for 4P E5 v2 and 8P E7 v2, which will probably be released next year and of cource be more powerful.
However I'm sorry that I still have a question, in your signature I see that you have spent $30000 for "3 x 4P G34 + 2P SR-2 + 4P E5-4650 ~ 204 cores / 248 threads ~ a $30.000 investment".
How is it so expensive? Looks like even much more expensive than 8P E7-8870! Or I have missed something important?
Click to expand...

It is 5 rigs that cost::
SR-2 (From Norway) ~ $4350
4P 6272 (From Norway) ~ $6000
4P 6172 (CPUs & mobo from ebay) ~ $5000
4P 6176se (CPUs & mobo from ebay) ~ $5500
4P E5-4650 (CPUs from ebay) ~ $7000
Some other related cost & RMA cost ~ $1250

Total cost 5 rigs ~ $29000
 
So, in my understanding if 8P E7-8870's PPD is higher than 1.71x of 4P E5-4650, it would be more cost-efficient than all of your these rigs? And in my experience this would probably be the truth. However, I still think $12000 is a little expensive. Waiting for the next generation of 4P (ES) Xeons might be a better choice.

-alias- said:
It is 5 rigs that cost::
SR-2 (From Norway) ~ $4350
4P 6272 (From Norway) ~ $6000
4P 6172 (CPUs & mobo from ebay) ~ $5000
4P 6176se (CPUs & mobo from ebay) ~ $5500
4P E5-4650 (CPUs from ebay) ~ $7000
Some other related cost & RMA cost ~ $1250

Total cost 5 rigs ~ $29000
Click to expand...
 
Last edited:
-alias- said:
It is 5 rigs that cost::
SR-2 (From Norway) ~ $4350
4P 6272 (From Norway) ~ $6000
4P 6172 (CPUs & mobo from ebay) ~ $5000
4P 6176se (CPUs & mobo from ebay) ~ $5500
4P E5-4650 (CPUs from ebay) ~ $7000
Some other related cost & RMA cost ~ $1250

Total cost 5 rigs ~ $29000
Click to expand...

I sure feel for all of you that have to pay such high prices for your equipment, and I really do not understand why it has to be that way. Why does something that cost $2500 to build in the US have to cost double that in other parts of the world. :confused:
 
Grandpa_01 said:
I sure feel for all of you that have to pay such high prices for your equipment, and I really do not understand why it has to be that way. Why does something that cost $2500 to build in the US have to cost double that in other parts of the world. :confused:
Click to expand...

You know, when I buy parts from ebay & the US, I have to pay 25% tax to my state, and customs treatment as well, even if there is nothing to pay in customs for data parts. So the price is rising about 30% before the goods arrived to my place.

When I buy brand new parts from webshops etc. in Norway, it is the same as mention over, but of course the importer and the shop must earn some money to, so the price I paid in the US is now double, and that is what it is. We pay a lot of tax in Norway, but we have good earnings, free health care etc. so I think we can live with the price.

And of course, again, I believe I did pay some tax to the US in the price of the goods when i bought it from the webshops in USA. Normally I can get this tax back, but it is a lot of paperwork so I do not care.:D
 
Yes, I could save 25% I think, but it is not quite legal, I think.:) Some times they open gifts also.
 
We could still help out, we buy the goods for you. you then buy the goods from us + shipping. We organise shipping together - as in paperwork- and if we do everything above board its saves hassle at customs. I'm in th UK and i've bought from several members on here from the US without any problems. I've even set up an account with my local customs office so i can pre pay customs charges
 
Thank you Nathan_P
That sounds very good in my ears, we will stay in touch.
 
Just checking in and making sure you were not still considering an 8p 1567 rig...
a second 2011 would be far more practical....not mentioning the fact that a 4p 2011 rig is going to be nearly as fast as an 8p 1567...yes the scaling is that bad

I am only getting about 16% faster than a non-turbo 4p 2011 (and that is with non-realistic 4p-8p scaling) ...
definitely not worth the cost... considering it may not even be faster than a 4p 2011.
And even if the performance was there... you will never be able to sell it.
 
Last edited:
I am still waiting for the price from my supplier, and it have to be damn good if I should go any further with this prosject. I am aware that this 8P rig is not very smart when it comes to folding, but only to be able to put my hands on one like this is almost enough for me to want to make it happen. But the price have to be really good, and $12.000 is not bad, when the real price is around five times more for an beast like this! Anyway, I have not decided yet!
 
Last edited:
How much efficiency would be lost in just running 2 BigAdv clients for now (versus 2 4P rigs)? With the 8P even if you can't run one client right now, you're future proofed for when that many threads become supported, and I can see the PPD being epic.
 
Since the scaling issue in the software, in theory nothing. 2 clients on the one machine should be the same as 2 machines running one client...... if all hardware specs are the same( CPUs, memory type, memory channels, etc..... )
 
Nightbird said:
How much efficiency would be lost in just running 2 BigAdv clients for now (versus 2 4P rigs)? With the 8P even if you can't run one client right now, you're future proofed for when that many threads become supported, and I can see the PPD being epic.
Click to expand...

NOOOOOOO
This is the farthest thing from future proof.

The reason why Untitledone's 8p is awesome is because its nearly the same performance as a 4p g34 and it cost him much much less.

12k for an obsolete system is painfully not futureproof.

402blownstroker said:
Since the scaling issue in the software, in theory nothing. 2 clients on the one machine should be the same as 2 machines running one client...... if all hardware specs are the same( CPUs, memory type, memory channels, etc..... )
Click to expand...

The software scales just fine....
S1567 is where the shitty scaling comes from. They abandoned it for a reason.

-alias- said:
I am still waiting for the price from my supplier, and it have to be damn good if I should go any further with this prosject. I am aware that this 8P rig is not very smart when it comes to folding, but only to be able to put my hands on one like this is almost enough for me to want to make it happen. But the price have to be really good, and $12.000 is not bad, when the real price is around ten times more for an beast like this! Anyway, I have not decided yet!
Click to expand...

but its not a beast... its a power hog and a waste of resources....
If you need help spending your access money I can give you my paypal...
 
Hi, Patriot.
In 13# of this thread, deadthings shows a nice FAH performance for 4P E7-8860, however, seems his result is conflict with your result of 4P E7-8870 in the spreadsheet of google docs.
The theoretic PPD of 8P E7-8870 scaled from deadthings's 4P data are 1.47M for 8101 and 2.02M for 8102, while the numbers scaled from your 4P E7-8870 are only 768K and 1.06M (as listed on sheet 5 of the spreadsheet).
So, I think first we should find out what makes the difference, and then we could fairly judge whether E7 is good or not for FAH.

Patriot said:
NOOOOOOO
This is the farthest thing from future proof.

The reason why Untitledone's 8p is awesome is because its nearly the same performance as a 4p g34 and it cost him much much less.

12k for an obsolete system is painfully not futureproof.

The software scales just fine....
S1567 is where the shitty scaling comes from. They abandoned it for a reason.

but its not a beast... its a power hog and a waste of resources....
If you need help spending your access money I can give you my paypal...
Click to expand...
 
Last edited:
quicks deadthings states Theoretical PPD, Patriot was tested PPD either you or I am reading it wrong but I would tend to accept the tested before the theoretical. That could be what makes the difference ;)

quickz said:
Hi, Patriot.
In 13# of this thread, deadthings shows a nice FAH performance for 4P E7-8860, however, seems his result is conflict with your result of 4P E7-8870 in the spreadsheet of google docs.
So, I think first we should find out what makes the difference, and then we could fairly judge whether E7 is good or not for FAH.
Click to expand...

Dead Things said:
I would agree with this assessment regarding the inefficiency of LGA 1567 HT. I have a 4P E7-8860 system folding, and the performance difference between running 40 threads without HT vs. 80 threads with HT does not reflect the kind of scaling we have come to expect based on our experiences with desktop Nehalem platforms. This was confirmed with synthetic benchmarks and has little to do with an "efficiency ceiling" over a certain number of threads as the trend was noted even in a UP configuration. Theoretical PPD for my machine is around 660k on P8102 and 480k on P8101, all while pulling a continuous 750-780w AC. PPD/W is not its strong suit.

I was fortunate in that I found a way to piece together my E7 box for roughly the same price as a 4P G34 rig and I had fun doing it. If I was looking at a potential $12k pricetag, though, I would certainly be looking into alternatives.
Click to expand...
 
Grandpa_01 said:
quicks deadthings states Theoretical PPD, Patriot was tested PPD either you or I am reading it wrong but I would tend to accept the tested before the theoretical. That could be what makes the difference ;)
Click to expand...

I want to know his tpf... I noticed theoretical as well.... it may just be because he isn't including upload time dunno...

I can't find all of mine...only found 1 random unit and I got performance a touch better later on...

I do know that 1567 4p is 20% slower than harbringer...but I don't remember if that was ppd or tpf...so im hosed.
Grandpa you could probably find the tpf if you checked summer logs in the secret place lol

What I do know...is 1567 exhibited terrible scaling compared to 2p 1366 ... I would expect scaling to be worse going from 4p to 8p...definitely not half the time.]

but even if the performance is there... I don't see a way to justify the power draw at the performance level on a dead-end socket. (and I have arguments with the performance level)

I also know that harbringer beats an 8p of octos on 1567.... soo.... I really don't expect tons better for the 10c
 
Since Dead Things actually has a 4P E7 rig, he should have no difficulty to get real PPDs of 8101 and 8102.
I guess he states it a theoritical PPD is just because the number was calculated from the avg TPF instead of the real points from stanford. Would Dead Things make a clarification for it?

Grandpa_01 said:
quicks deadthings states Theoretical PPD, Patriot was tested PPD either you or I am reading it wrong but I would tend to accept the tested before the theoretical. That could be what makes the difference ;)
Click to expand...
 
Sorry lads, was a bit optimistic on the P8101 PPD...

P6901 Avg. Time / Frame : 00:05:17 - 486,862 PPD
P6903 Avg. Time / Frame : 00:12:19 - 613,356 PPD
P6904 Avg. Time / Frame : 00:17:20 - 601,497 PPD
P8101 Avg. Time / Frame : 00:11:27 - 448,336 PPD
P8102 Avg. Time / Frame : 00:08:53 - 656,067 PPD

Yeah, when I said "theoretical" I just meant going by the HFM reported PPD and not the real-world results taking into account upload time, server downtime, unforseen urges to run wPrime, stuff like that...
 
You must log in or register to reply here.
Back
Top