it might be time to consider SB-E

Discussion in 'Distributed Computing' started by jebo_4jc, Oct 24, 2012.

  1. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    I noticed today that the Xeon E5 2660 is selling for $350-$400 apiece if you buy a spicy variant or scoop up a good deal on an ebay auction. This cpu is an 8 core, 16 thread, 2.2ghz nominal, 3ghz max turbo, 20M cache, 95W CPU. I also noticed that Supermicro dual 2011 boards can be found as low as $350. So, for a little over $1000, you could have the base of a nice folding system.

    The question in my mind is, at what clock speed will these chips fold? I'm sure the 3ghz advertised max turbo is for single threaded performance, but I wonder if the 30x multi could be forced 100% of the time, or at least a 29x multi. At 2.9 to 3.0ghz, 16 cores/32 threads would be pretty competitive.

    Could somebody educate me on what I could expect in terms of a 24/7 100% load clock speed with these chips? Has anybody tried throttlestop or a similar method to force a higher multiplier?
     
  2. LigTasm

    LigTasm [H]ardness Supreme

    Messages:
    4,993
    Joined:
    Jul 29, 2011
    I have had several and I currently run an unnamed chip that is advertised at 1.6ghz with 2.1ghz max turbo speed. It seemed to have settled at 2.0ghz when folding on all 8 cores even with the short/long duration power limit turned up. I experienced the same with a pair of 2650's, they seem to fold 1 bin below max turbo.

    Go 2P if you do it, as a single chip any but the ones that will run at 2.9ghz+ 24/7 (2680/90, 2687W) will be beaten out by a 2600k/3770k in SMP folding.

    You can see my machines here, the E5 at 2.0 falls squarely between my 3570k at 4.4ghz and the 3770k at 4.4 as well, the results were the same on both 6.34 under Ubuntu 10.10 and now using the V7 beta client under the same OS. The plus side is it uses much less power, about 50w less than the 3570k machine.
     
  3. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    Great info. Thanks.

    So the 2660 would probably fold at 2.9ghz 24/7
     
  4. tkam

    tkam [H]ard|DCer of the Month - Dec. 2012

    Messages:
    436
    Joined:
    Dec 18, 2007
    I believe the spicy 8-core 2.2ghz chips run at 2.5ghz all-core turbo.
     
  5. pjkenned

    pjkenned [H]ard|Gawd

    Messages:
    1,971
    Joined:
    Jan 8, 2010
    That sounds about right.

    Also you have to be sure you can get a board that supports ES chips in 2P configurations.

    Oh... and for $350 you get a basic 2P motherboard. For ~$550 you can get an onboard LSI SAS2308 plus either 4x GbE (not including the IPMI NIC) or dual 10GbE. If you ever wanted slightly slower folding performance trading off for a storage server for example.
     
  6. LigTasm

    LigTasm [H]ardness Supreme

    Messages:
    4,993
    Joined:
    Jul 29, 2011
    Also a note: almost none of the supermicro boards will work with spicy chips in a 2P configuration, your best bet is an Asus board unless you get OEM/retail chips.
     
  7. tkam

    tkam [H]ard|DCer of the Month - Dec. 2012

    Messages:
    436
    Joined:
    Dec 18, 2007
    The supermicro x9dai works fine with every spicy chip I've ever tried with it.
     
  8. LigTasm

    LigTasm [H]ardness Supreme

    Messages:
    4,993
    Joined:
    Jul 29, 2011
    I think thats the one most of the auctions list for compatibility, since there are a ton of SM boards thats why I said almost none, I couldn't remember the one that did off the top of my head.
     
  9. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    Any idea about the 2ghz E2650?
     
  10. LigTasm

    LigTasm [H]ardness Supreme

    Messages:
    4,993
    Joined:
    Jul 29, 2011
    I ran a 2P setup with those and they typically ran at 2.4ghz turbo after I increased the short/long power limits, the chips run really cool at a max load temp of around 50*C so there is plenty of thermal headroom. I was running bigadv and only getting 8101's, I'll see if I can find my TPF data for those.
     
  11. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    That would be appreciated
     
  12. LigTasm

    LigTasm [H]ardness Supreme

    Messages:
    4,993
    Joined:
    Jul 29, 2011
    Running Ubuntu 10.10 and the 6.34 client (kraken wrapper) with an 8101 WU it was getting an average TPF of 23:02 with 280w draw at the wall, very little deviation in the frame time. Minimum was 22:59 and maximum was 23:07. I wish I had a chart for you, but I wasn't using HFM at the time so it wasn't logged anywhere besides where I wrote it down.
     
  13. tjmagneto

    tjmagneto [H]ard DCOTM x2

    Messages:
    3,014
    Joined:
    Aug 6, 2008
    Based on the Linuxforge Bonus Point Calculator estimated total points would be 251345. With an estimated completion time of 1.6 days (average tpf of 23:02) to complete the WU that would net you an estimated157136 PPD.
     
  14. Pocatello

    Pocatello DC Moderator and [H]ard DCOTM x4 Staff Member

    Messages:
    6,194
    Joined:
    Jun 15, 2005
    How does this compare with the 4P AMD stuff $ to $ ?

    Seems like, if I were to guess, the 4P AMD still has this beat? For example, $1,100 spent on 4P would net you more PPD (and more PPD/watt) then with Intel?
     
  15. theGryphon

    theGryphon [H]ard|Gawd

    Messages:
    1,295
    Joined:
    Nov 21, 2011
    For $1,100, you could build a 4P Opteron 6128 system, which would net around 170K-230K PPD with 8101 WU, depending on overclocking.
     
  16. M0rph3us

    M0rph3us 2[H]4U

    Messages:
    3,704
    Joined:
    Sep 5, 2002
    So it seems to me that given the current information we have that the SB-E dual setup is now better price/point, but only slightly. Given the running temps, I wonder how well these (temp wise) would run in a 2U case. I'm intrigued to know this as 280w for 250K PPD seems rather impressive. Is it also possible to get a quad board and only run 2 like with the optys? Seems if you could get a quad you would have a pretty awesome setup running at ~500W. Just curiosity here speaking hypothetically since I need to buy an engagement ring soon and all PC spending for me has to be halted, but maybe someone else could use the info.
     
  17. Nathan_P

    Nathan_P [H]ard DCOTM x2

    Messages:
    3,377
    Joined:
    Mar 2, 2010
    Just in the process of closing out a 2660 deal myself, the 2680's give much better PPD for not much more power but the setup costs are higher, going for an asus mobo for now as i plan on upgrading again later. My daily driver and slower rig are both being retired from folding, depending on power draw my slower dedicated rig will either be switched back on early next year, rent a boxen'd if there is any interest or sold.

    260w from the wall will give 600PPD/w on a 8101 - not bad and much better than the 300PPD/w i currently get
     
  18. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    Which Asus mobo Nathan? Will it support espaniol chips?
     
  19. Wyluliraven

    Wyluliraven Gawd

    Messages:
    975
    Joined:
    Apr 27, 2010
    Z9PE-D8 WS works well with ES chips.

    I would like to try and get a Z9PA-D8 board to play with (YAY ATX FORM FACTOR! YAY!), but that is down the line for the moment.
     
  20. quickz

    quickz Limp Gawd

    Messages:
    256
    Joined:
    Jul 30, 2012
    4p G34 es is the winner of PPD per dollar, but 2p/4p E5 might be better at PPD per watt.

     
  21. quickz

    quickz Limp Gawd

    Messages:
    256
    Joined:
    Jul 30, 2012
    The compatibility issue of E5 ES chips is a bit complicated, it's highly dependent on the CPU stepping and even the QDF code. In general, C0/C1 stepping is perfect and free of compatibility issues (no matter 2p or 4p), while B1 stepping is bogus.
    One thing should be noted is that Z9PE-D8 WS only works well with E5 26xx ES chips, it doesn't support any E5 46xx ES chips.

     
  22. Untitledone

    Untitledone [H]ard|DCer of the Month - April 2012

    Messages:
    1,486
    Joined:
    Feb 28, 2011
    So about 157K PPD
    Looks like 4P 6128 wins on the PPD/$ still, but it will pull more power
    It is 157K PPD for 280W, which is still pretty good, 560PPD/W. Thermodynamics is thermodynamics, I have a 2P 1366 rig that pulls 315W. With good coolers and adequate air flow you should be fine. It does not matter what the core temps are as far as heat production goes. 280W at 70*C is the same amount of total heat as 280W at 50*C, the delta T will just be higher on the 70*C one. As long as you are within equipment specs you are fine, but lower is better and a better cooler and better air flow will drop the temps.

    Only the 4xxx series can run in 4P mode, 2xxx are 2P. I do not know if the 2xxx series will boot in a 4P 2011 board. Last I remember, 4650 ES were in the $700+ per CPU range if you got lucky and are at $1500 per CPU buy it now at this time on eBay. The 4P 2011 boards are around $1000.

    A high overclock on a 4P G34 with 6180SE / ES (around 3Ghz) will net you 477K PPD on 8101, and a 4P 4650 (2.7Ghz) machine will get you around 631K PPD on 8101.

    The 4P 4650 machine is no joke with more PPD and less watts than a 4P AMD, but it is 2-3x the cost of a 4P AMD machine, and the CPU's are also scarce.
     
  23. quickz

    quickz Limp Gawd

    Messages:
    256
    Joined:
    Jul 30, 2012
    I think 157K PPD is not a good score for 2P E5 rigs, 2P Xeon E5 3.0G (QA92) with slightly overclocking could get 280K PPD on 8101, and QA92 is also cheap.
    And as far as I see, 46xx chips could work on many 2p boards without difficulties, but no 26xx chips could work on 4p boards at the moment.

     
  24. Untitledone

    Untitledone [H]ard|DCer of the Month - April 2012

    Messages:
    1,486
    Joined:
    Feb 28, 2011
    It looks as if there is a steep price jump from the 2650/2660 (around $350-$400 per CPU) to the 2670/2680/2687W range ($650-$720 per CPU).

    I did not see any QA92's, but the QA91's are going for $725-$785

    2($350) + $350 for motherboard would get you 157K PPD ($1050)
    An SR2 with dual hexen a health OC would be in this range of PPD and price but more power.

    2($650) + $350 for motherboard would get you 280K PPD ($1650)
    Around the price of a 4P 6172 AMD machine which could get anywhere from 300K to 350K PPD on 8101 depending on overclock.

    If power is a concern, then the 2P 2011 should do fairly well. Looks like 2P 2011 is pretty competitive now. We will have to wait and see if 4P Intel prices drop any for more of those to get adopted. We will also have to see how Pile Driver shakes things up on the G34 socket coming soon.
     
  25. 3one5

    3one5 Gawd

    Messages:
    579
    Joined:
    Jan 6, 2005
    What motherboard would you be using on that 6128 system? The only board I see on Newegg runs close to $800
     
  26. tear

    tear [H]ard|DCer of the Year 2011

    Messages:
    1,568
    Joined:
    Jul 25, 2011
    Supermicro H8QGL (preferred) or H8QGi. Check wiredzone.
     
  27. 3one5

    3one5 Gawd

    Messages:
    579
    Joined:
    Jan 6, 2005
    That board is 1K. How are you going to build the system for 1100?
     
  28. 402blownstroker

    402blownstroker [H]ard|DCer of the Month - Nov. 2012

    Messages:
    3,179
    Joined:
    Jan 5, 2006
    Dont look on crapBay. As pointed out wirezone.com is the source.
     
  29. Nathan_P

    Nathan_P [H]ard DCOTM x2

    Messages:
    3,377
    Joined:
    Mar 2, 2010
    Eyeing up both boards, one now and one early next year, for all flavours of chips
     
  30. -alias-

    -alias- Limp Gawd

    Messages:
    374
    Joined:
    Jul 15, 2012
    I am the very lucky owner of a 4P 4650 machine, and it fold the 8101 with a TPF of 8:52 that give a PPD of 658K at the most. The average is 655K PPD for the 8101, but the 6901 is much better with average of 690K. My 4650 run at 3.1GHz in turbo mode on all cores. The power draw is 725W from the wall, meaning a PPD/Watt at 907.5 for the 8101. This machine cost me around $8000 ++ (if my memory is not damage from Alzheimer in this hour), that is very expensive because I have the barebone box from SM, and I live in Norway. At the moment all my 5 rigs is working with 6901, that is very unusual this days with only 8101 around.

    My mother died of Alzheimer the 21 of January this year, as 2 of her siblings has done the last 5 year. As you can understand, I am a little worried for that I can have inherited this horrible disease my self, but I do not think really so! Anyway, this is the reason why I do all this folding for the research in regi of Stanford. Have a nice day & tonight all of you folding people.

     
  31. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    Alias, its great to hear your folding story.

    What does "regi" mean? It doesnt translate :)

    That's an amazing rig you have. Enjoy that monster, and I hope we can find the cure for you and your family!
     
  32. -alias-

    -alias- Limp Gawd

    Messages:
    374
    Joined:
    Jul 15, 2012
    Regi in my language, means: "on behalf of"...Sorry about my bad english writing!
     
  33. jebo_4jc

    jebo_4jc [H]ard|DCer of the Month - April 2011

    Messages:
    14,521
    Joined:
    Apr 8, 2005
    No apology necessary :)

    I love when folders from other countries join our discussions.
     
  34. extide

    extide 2[H]4U

    Messages:
    3,441
    Joined:
    Dec 19, 2008

    Wow, dude that is way impressive. I don't kno what else to say but that is some serious dedication, and that 4P box is NUTZ fast! Good work sir!
     
  35. Untitledone

    Untitledone [H]ard|DCer of the Month - April 2012

    Messages:
    1,486
    Joined:
    Feb 28, 2011
    No worries sir! I am just trying to get the most output from the lest money possible. The more output we can get online the more science that can get done! The 4P 2011 is prohibitively expensive for most folders so it does not get recommended. Since for $8000 you could build 3-4 AMD 4P's for that price which would put out anywhere from 1 million to 2 million PPD depending on work units folded, and overclock obtained.

    I have great sympathy for you. We are all here to fight these diseases the best we can. We try to help everyone get the most out of their systems, and budgets so that we can bring the most production against these diseases.

    Don't take my responses as anything negative against you. Your information is valuable to us as well!