Help me build my next WCG Rig

MGMCCALLEY

MGMCCALLEY

[H]ard DCOTM SEP 16 / NOV 17
Joined
Jul 3, 2012
Messages
361
Hey all. I've been looking to build a dual 2011-3 system for a while. I have a couple of systems crunching already, but this one is going to be special. I'm retiring my 1366 main system this year after almost 8 years of faithful service. This new rig will not only be crunching for WCG, but hosting at least two 1080 Ti cards for F@H.

The goal is to start small, with maybe two 2620 V3s, and expand into faster v3 or v4 chips when those start flooding EBay. I may consider some ES versions if they're the right price, but I'd need a confirmed source and guaranteed motherboard compatibility.

These two articles sparked my interest late last year:

http://www.techspot.com/review/1155-affordable-dual-xeon-pc/
http://www.techspot.com/review/1218-affordable-40-thread-xeon-monster-pc/

Both excellent articles.

Now I know the case I want it to be in (because I already own it), and it supports up to HPTX:

http://www.enermax.com/home.php?fn=eng/product_a1_1_1&lv0=2&lv1=55&no=179

but it makes no mention of SSI EEB. You'd think that it being smaller, it would fit, but can anyone find confirmation that it would? I'm no good at modding and don't have time anyway.

A motherboard size comparison:

https://lanoc.org/review/motherboards/5791-motherboard-sizing

shows that I should have plenty of room for an SSI EEB if the case supports HTPX, but will it work on that case?

Also, I've been eying this board:

https://www.newegg.com/Product/Product.aspx?Item=N82E16813132268
https://www.asus.com/us/Motherboards/Z10PED8_WS/

The 7 full PCI-E slots (yes, some of them are X8 electrical) give me tons of options for expansion and cooling. The M.2 slot will go pretty far in providing fast boot speeds and a great primary drive. I don't really need the on-board SATA, as I'll have an 8 drive RAID6 installed, but they're nice for other things. The all solid caps and the heatsink layout is nice for a top-venting system.

The goal is 3 or 4 1080 Ti cards when they're released, will likely have to be the blower style fans without backplates if I have 4, but likely I'll only have 3 because I need a slot for my RAID card.

Power will be the other issue. The cards can pull 1,000 watts on their own, and add another 300 for two high-end 2011 chips, plus memory, drives, etc. I'm thinking the math comes out to just under 1,500 watts. I'm a big fan of Corsair PSUs, and like their new USB link that I use on another server to monitor fan speeds and temps, so I'm looking at this PSU:

https://www.newegg.com/Product/Product.aspx?Item=N82E16817139079

Fully modular is almost implied these days. The efficiency is nice, and I can still put it on a 15 amp outlet and protect it with a battery backup (assuming 1,500 watt output, not 1500 VA). I have a 20 amp outlet available, so I'm likely going to be using that just in case.

Long post, I know. but there are still many details to has out. I'd love to hear from you guys on the idea, the parts choice, suggested HSF if anyone has this board, the fit for the case, and the PSU requirements. It's by far my most complex build for my own computer to date, and I'd like to get it right the first time. I'm also hoping for another 5+ year run on this rig, so I'd like to get the base components as solid as possible without resorting to Supermicro or other custom motherboards, cases, etc.
 
All I read in this post is 3 or 4 1080 Ti GPUs. lol

I'm gonna have to up my game when this happens, too. :(
 
No worries Skillz. I'm actually downgrading my electrical and space requirements with this build. Afterwards, all other cards will be sold with the possible exception of a couple of 980 Tis and my 1080. I'm going to try to stick to the 3-4M PPD range. I don't see myself attempting to reach the top 20 folders. It's a never-ending upgrade game to get there. I'm just happy to be keeping up with the top 100, and I plan on staying there. For now, I have the advantage of free electric and cooling during the summer, so I hope to catch up again if I fall off the list before I build this next rig. There is a possibility, too, that I lose this office before too long, so thus another reason for the multiple downgrades in space and electrical consumption. This may become my one and only folding and crunching rig before too long, so it's gotta be a good one. And I doubt I'll catch up to your WCG performance in the near future. Even with another dual rig, you'll still have a 40% lead in PPD there, and you'll pass me up before too long. I didn't get that much of a head start on you there.
 
Well damn it! That's a problem I was hoping to have. :(

I would definitely look into ES CPUs though. I have a 10 core, 20 thread ES Intel chip in one of my 2011 boards. Was extremely cheap on eBay and works great.

I get between 60k - 70k WCG points on it daily and that's with only 16 WCG instances running at once. I have the other 4 threads (2 cores) dedicated to the two GPUs I am folding with. It's two 980 Ti's and it averages around 1.2M PPD with it.

It's a Intel Xeon Processor E5-4620 v3 QS CPU 2.0GHz 10-Core QH99 ES Similar to E5-2650 processor. I paid $300 shipped for it a couple months ago.
 
Nice. I never even considered a 46xx chip in a single-cpu board. Glad to know it's working out. One of the articles mentions compatibility for most boards with ES chips, but I'd need to know that I have a return option if it doesn't work out. Not as many sellers looking to take them back once they part with them. If I start out with two 10 core chips, 40 threads, I might get 30 of them on WCG, and save the other 10 for F@H and multitasking. I'll be transferring my 48 TB RAID array to the new rig as well and retiring my 24 TB array. It's always too full, so it needs to go, too. In theory, I could scale back down to a single laptop and a single PC and still get 3M PPD on F@H and 100K on WCG in a single box.
 
I looked up to make sure the ES chip would work in the board IIRC. It would have sucked if I got a $300 CPU that I couldn't use though. So I definitely understand that. I can't remember what my sellers return policy was. I'll check here shortly.

edit:
30 day return policy is what he would have offered. I would have just had to pay shipping.
http://www.ebay.com/itm/351845189931?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT
end edit:

I need to fix my damn NAS as well. Keep getting bad 8TB drives. This is my second one that's already experiencing bad sectors and keeps getting dropped from Windows constantly. It's an RMA unit from WD. Leaving on a vacation in 2 hours so nothing I can do until I get back after the 13th.

I do have another 4P rig that I need to put together but I don't have any electricity to feed it unless I use the bathroom or kitchen. Haha.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
I've had very good luck with the 6 TB WD Reds. Not a single failure in about 20 drives. I try not to purchase anything from Seagate anymore, their QC is going downhill fast. I'm going to experiment with an 8 drive SSD array here soon, so that should be fun. Trying to get M.2 level performance from a RAID6 Samsung 840 array.
 
There is a good site here on extra spicy chips. It's a long posts, so you will need to filter for the right info. Our experience [H] member, Nathan_P may know more about S-spec/stepping. Also might want to read more about it here on chip issues and Intel's policy.

BTW, I may also be interested in a 2p rig but in no hurry.

When Naples comes out sometime this year, hopefully the xeon v3/v4, particularly the production version, will come down in price.
 
I haven't looked at my xeon list in any meaningful way since the end of F@H's bigadv so anything haswell related is incomplete and there is no broadwell xeon info on there at all. If people want i'll see what I can do
 
Nathan_P,

You don't see the need to update it for people running WCG? It might be very useful for those seeking to get the best bang for the buck on ES chips.
 
MGMCCALLEY said:
Nathan_P,

You don't see the need to update it for people running WCG? It might be very useful for those seeking to get the best bang for the buck on ES chips.
Click to expand...
To be honest I've never thought about the WCG people as I didn't think many would be looking at running 2p/4p machines.
 
Nathan_P said:
To be honest I've never thought about the WCG people as I didn't think many would be looking at running 2p/4p machines.
Click to expand...

I believe all of the top WCG producers are using 2P or 4P rigs now. I have two 2P and one 4P, I know Skills uses them, I'm assuming Brilong did, and I'm betting that ChelseaOilman does as well. We might be the minority of users, but knowing what ES chips to use might be some very handy information. If it's a lot of work for you, though, I would say it might not be a good investment of your time. I just know that I'm going to be looking for V3/V4 ES chips likely for that Asus motherboard, and I'd love to know what my best bang for the buck would be. My 2Ps currently have 2670s, but the board doesn't support V1/V2 chips, so I'm delving into an unknown area for me.
 
ChelsaOilman does for now. Not sure what he is upgrading to, but he might benefit from the information as well.
 
Same position as you guys -- almost all (95%+) of my production is coming from 2P and 4P systems.
 
Losing my office. Might have to move the 4P and the 2Ps down the road. Only enough power at the home office to run one project or the other.
 
Since we are all volunteers, donate what you can... No pressure buddy.
 
Oh no. That's not good.

I'm gonna have 4, 20a breakers installed here soon with all the plugs inside the garage so I can move all my boxen in one location. Having them spread out in 4 rooms is a PITA when they go offline.

I'm guessing you're gonna stick with F@H if you have to choose?
 
But having them all in one location can cause heat issues. It's why I have mine spread throughout the house.
 
Well I want to move them to the garage so my HVAC system doesn't have to over compensate during the warmer months with keeping the house at a livable temperature. While we do see our fair share of 90F+ degree weather here in North Carolina I don't think it would be enough to render them completely off.

That plus my roommate tends to not know exactly how power distribution works and she's already tripped the breakers about half a dozen times since she moved in late last Jan. Moving them to dedicated circuits, in the garage, will ensure she doesn't interfere with them anymore.
 
Yeah, I'm likely going to continue with F@H, and maybe halve my contributions to WCG. My garage is a machine shed, and prone to being warmer inside than out, not to mention having dozens of leaks in the roof. I would agree with housing them all in the same location, but for additional reasons. It's much easier to cool one space than to try to cool an entire house to compensate for additional heat generation in a single or just a few rooms.. A simple exhaust fan can keep the temps down most of the time, and a basic window AC unit, or a rolling AC unit with a window kit can keep even the hottest tight spaces manageable in the summer months. Most of them are relatively inexpensive to run if you keep the temp set to 80 degrees Fahrenheit or so and alternate them with an exhaust fan and fresh air intake at night when it's cooler. I have a much older home with only 200 AMP service for the entire house. I can only dedicate 40 amps or so to my office, and I'm frequently working on 5 or more computers simultaneously. So I have to watch my energy usage for that reason, not so much for the heat generation.
 
I do have a small window unit AC, but my garage doesn't have any windows except the garage door and their is a side door that leads outside that has windows in it also. Not sure how useful it would be to run the unit inside the garage as it would probably dump just as much heat back into the garage as it displaces. I do leave the garage door partially open though.

My house also only has 200A incoming. When I go to add those additional circuits I intend to ask about that.
 
Well yeah, but I was hoping to get feedback, or possibly links to cheaper versions, or maybe the newer version 2699 V4. This is supposed to be the OEM equivalent of the 2699 V3. I noticed that a lot of the 2699 V4 ES chips don't have as high of a max turbo as is listed on ARK. The version I posted "technically?" isn't an ES chip, so it would have the same specs in theory as the OEM versions sold with original units. Anyone want to chime in?
 
According to this website, it is original OEM processor with similar spec as E5-2699 v3 as shown in Table 2 of Intel package specification. Someone in Amazon explains this:

upload_2017-4-3_22-13-37.png


I think the price may come down when Naples shows up (if Ryzen is any indication of the future AMD server performance, and would certainly be cheaper than Intel but performance needs to be demonstrated). Overall this is still a very good chip for crunching. My 2695 v2, I overclocked the BCLK and all 12C/24T running at 3.02GHz and is stable for WCG.
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
What board are you using to adjust BCLK? I hadn't considered OC'ing these much, given that the power requirements are already pretty beefy and I'm putting at least two 1080 Tis in it, but I guess it's an option with the right board? Looks like this one is one of the few dual 2011 v3 that has overclocking available, but it seems miniscule, and not really worth worrying about. How are you getting every core to 3.02? I show the 2695 to be 2.3 Base, and 3.3 Max Turbo. Is a manual BCLK all it takes to set it statically on your board?

http://www.tweaktown.com/reviews/68...12-workstation-motherboard-review/index8.html

And an interesting OC article using this board:

http://techarx.com/indian-overclockers-break-world-record/
 
See my email sig below. ASUS P9X79 Pro. Awesome board but hard to get this at a bargain nowadays. As far as I know on Xeon (if MB and other components allowed it other than 1p xeons) you can only do BCLK overclocking. The second link by Indian overclockers lack details so I'm suspicious somewhat on how they achieved it. Could be a scam report?

I've Ivy bridge and Sandy bridge xeons. Best overclocking is with Ivy bridge xeons (BCLK only). No idea if haswell or broadwell will do better than Ivy bridge, except the 1p xeons.

Took some time to find stable BCLK for WCG, PG and other BOINC projects.... esp on PG LLR.

In BIOS, you can enable "Turbo Mode" which will run all my cores/threads at 2.8GHz x 108 = 3.024GHz assuming you have good stable power delivery and still within CPU thermal limit. When I ran 10 cores on PG LLR project, the 2695v2 reaches the CPU thermal limit (115W) and throttles down when you run two more cores to 12 cores.

I'm guessing it is easier to overclock with 1 CPU system but I'm not sure if haswell or broardwell will give the same performance as ivy bridge which is respectable 8% xeon 2p chip but in the long term, means 8% more points. :D

upload_2017-4-4_12-36-35.png
 
Got a E3-1230v5 on order for a new 1p rig I'm going to build. Tired of my i7 Server going offline while I'm working out of town so going for Xeon stability. I run WCG and a couple other BOINC progs on the Server and GPU fold too on a Maxwell. Should do me OK. Only 8 threads @ 3.4-3.8 GHz stock so I won't see the numbers you guys are getting. Gigabyte workstation mobo I ordered ('cause it was inexpensive) claims to have OC abilities but we shall see. I'm power limited by this circuit and UPS so opted to go newer for efficiency gain. My electricity is currently included with my rent but heat is an issue in my small office which also houses my main rig. If I'm in town for any length of time I'll try to get some comparison numbers.
 
Last edited:
1p xeons are good at overclocking. I think you can go above 4 GHz easily. For us with 2p xeons, all we can do is a tiny bump in the BCLK...
 
Great discussion. I just filed my taxes. Now I start the investigation phase. I'm seriously considering the E5-2696 V3 at about $800 per chip. The board is rather inexpensive, my case allows for two power supplies, and I've got two that will fit the bill already. I'll dedicate one to the motherboard, CPUs, and peripherals, and one strictly for the video cards. I'm thinking 2 or 3 1080 Ti cards when availability levels out and I get some solid review on stability of non FE boards. Starting with 4 GB per stick of RAM, going to 32 GB total for the board. That particular board, it seems, only takes ECC REG, so I'll wait for those chips to flood the market in a couple of years and buy it up to 128 GB or more of RAM then. Hopefully gaming goes the same direction as apps and they start taking advantage of more than 2-3 cores. If so, this rig will scale wonderfully, and allow for heavy multitasking and WCG/F@H contributions. If anyone has a lead on cheaper E5 V3 chips, or cheaper ECC REG DDR4 please chime in.
 
Came across 32C/64T system for sale by our [H] member.

He also listed this item in Ebay which is still active. He may give discount to [H] member.

The SM X10DAl has two PCIE x16 slots. Not sure why he/she add 3 fans per CPU. This is ATX motherboard.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Still active. He relisted it. See here. Same picture and location.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Yeah I took a look at it. It's a pretty ugly setup. Not only will I need more PCI-E slots, but this will be a demo unit for my business. I'll have to show it off, and Supermicro boards aren't pretty to begin with. I need to build something that a customer will absolutely drool over. I'm involved with people that are invested in cryptography, security, 3D modeling, and the like. If I can show them a beast of a computer for half the cost of an equivalent OEM workstation, I can start shifting my business out of corporate I.T. consulting and get into much more fun hardware builds.
 
You must log in or register to reply here.
Back
Top