My Farm Network Plan - Please comment.

Jon855

[H]ard DCOTM January 2008
Joined
Aug 25, 2005
Messages
11,788
Funds are about 20% done...

Please comment on my networking and configurations... Simply there are two set of farm in a different location in the apartment. Want to know about the network configuration, I think I've got it nailed down. Feedback of course would not hurt, I repeat this farm's fund is at about 20%. I will start building this farm in the summer and grow it as I go. This is the configuration I'll be following.

NetworkLayout.png
 
Network setup looks fine.

I do worry about have 10+ rigs on one Circuit ....


They will be divided up into four sections. Five per circuit. First, I'll need to have the maintenance label my breakers so that I can know exactly where is where and its maximum output, etc, etc.

EDIT - From what I am able to tell, I have three 20A and three 15A circuits, just needed it to be labeled so that I know where is where.

EDIT 2 - I calculated the total power if I'm not mistaken - ( ( ( 20A * 115v + 15A *115v ) * 3 ) * 0.8 EffRate ) == ~ 9660 Watts

EDIT 3 - Assuming each C2Q draws 140W * 20 = 2800 Watts
 
Jon, Is this the project you had in mind recently? If so, great that you have begun the preliminary stages to see it to fruition. BTW, what will be the hardware specs of each machine in your proposal?

 
Jon, Is this the project you had in mind recently? If so, great that you have begun the preliminary stages to see it to fruition. BTW, what will be the hardware specs of each machine in your proposal?


Sure, the parts and so on are listed below... Each RIG will have a goal OC of 3Ghz if am able to get any higher, then that's great but 3Ghz is the min.

RIG -

20x Case: Antec NSK4480B w/ Earthwatt 380
20x Motherboard: GIGABYTE GA-G31M-S2L
20x CPU: Intel C2Q 6600
20x HSF: XIGMATEK HDT-S1283 120mm Rifle CPU Cooler
20x RAM: Kingston 2GB (2 x 1GB)
20x Storage: Kingston DataTraveler I 1GB Flash Drive

For every five units I plan on hooking it up to this UPS:
http://www.newegg.com/Product/Product.aspx?Item=N82E16842101066 for a peace of mind and insurance.
 
John,

Quick question for you. When you're building this many boxes, how do you configure the OC?

Do you attach a monitor, configure your OC, then stress test it? If stress testing, how do you check temps?

 
Your router is going to choke! There is not one 4 port router on the market that is going to be able to handle what you propose. I just upgraded mine to a Linksys RV016 and wow... the stability has really kicked in! Also make sure you do static IPs.

Next the failure of the internet is not the issue. I've unplugged my router from my switch and as long as the switch stayed on the boxes were okay. Although cable modem is alot less reliable from my experiences. You may want to add a DSL line as well and if you use the RV016 it allows up to 7 WANS which can be any type of internet portal!

Finally ditch that expensive UPS and go take a look at Fry's they have some refurb'd ones that are an excellent deal in the 83buck range for 900W/1500VA. 5 units is about right on what I spec'd not the UPS you have you have a 750 watt UPS you will be overloading it with a 750 watt UPS.

Just my .02 from someone that has 7 computers folding in somewhat similar fashion. Also you have one computer monitoring? Why is it not folding? I run FahMon on one of my Quads and use it to monitor. It drops it maybe 25pts... and your not wasting money with that box not folding.

Your assumption is a bit low on the Q6600. It is more like 160 minimum assuming NO overclocking. @ stock mine varied by build between 144Watts to 168Watts YMMV but plan for more. Also are you overclocking? If so that will really ramp up production.

How are you cooling 20 machines? even 10 in each location is going to be very warm!

Good Luck it will take a while to get it all stable, be patient! :)



 
For the router, you can also setup one of the folding boxes as a monitor/router box (use a VM to run either Endian Firewall or Untangle).

 
Your router is going to choke! There is not one 4 port router on the market that is going to be able to handle what you propose.

I'm curios, what purpose will the router have besides as needed send and download the work unit? And that little bit doesn't sound like much... since they don't need a constant connection to the internet except at those times.

Just asking since I'm in the process of upgrading my network, not for folding reasons but for other, even tho I think I put more load on my little linksys router then he will with those folders.

I'm running 2 gaming rigs, and a Dedicated Server for COD4 and it has no problems at all...
 
John,

Quick question for you. When you're building this many boxes, how do you configure the OC?

Do you attach a monitor, configure your OC, then stress test it? If stress testing, how do you check temps?

I've just set up 5 new Quads.
The way I worked it was to first Install a copy of Win XP on a hard drive with CpuZ to monitor Vcore with, Core Temp for temp monitoring, SMP Seasaw Pro so I could set affinities easily and two copies of Orthos to stress the cpu with.
First booted with UBCD and ran Memtest86+ to check the memory.
Then booted of the hard drive and checked that it ran ok at stock speeds.
Overclocked it to what I hoped to get.
Re-ran memtest if not running memory at stock speed.
Then ran dual Orthos for 24 hours.
Once I found a stable setting I then set up the next box to the same specs and tested to see if it ran as well.
If it didn't then all the setting got lowered a bit.
Kept going like that to find a nice common overclock for all boards.
Theres less testing of the latter built boxen doing it that way.

My final common setting are FSB at 345 x 9 for 3.1 Ghz with the Vcore at +0.1v and the memory running on a 2.5 multi.
My Quads are pulling ~165 watts from the wall.
Now 2x p2605 inside the VM's gives me just over 4k per box, which is what I was aiming for.

Luck ............... :D
 
John,

Quick question for you. When you're building this many boxes, how do you configure the OC?

Do you attach a monitor, configure your OC, then stress test it? If stress testing, how do you check temps?

I plan to configure all of my rigs to 9x333 with 0.10v bump for stability and run a custom XP installed on one HD and use it to test all of them for the time being. When I can verify that the temps and etc are all good to go, I'll just let it go.

Your router is going to choke! There is not one 4 port router on the market that is going to be able to handle what you propose. I just upgraded mine to a Linksys RV016 and wow... the stability has really kicked in! Also make sure you do static IPs.

Next the failure of the internet is not the issue. I've unplugged my router from my switch and as long as the switch stayed on the boxes were okay. Although cable modem is alot less reliable from my experiences. You may want to add a DSL line as well and if you use the RV016 it allows up to 7 WANS which can be any type of internet portal!

Finally ditch that expensive UPS and go take a look at Fry's they have some refurb'd ones that are an excellent deal in the 83buck range for 900W/1500VA. 5 units is about right on what I spec'd not the UPS you have you have a 750 watt UPS you will be overloading it with a 750 watt UPS.

Just my .02 from someone that has 7 computers folding in somewhat similar fashion. Also you have one computer monitoring? Why is it not folding? I run FahMon on one of my Quads and use it to monitor. It drops it maybe 25pts... and your not wasting money with that box not folding.

Your assumption is a bit low on the Q6600. It is more like 160 minimum assuming NO overclocking. @ stock mine varied by build between 144Watts to 168Watts YMMV but plan for more. Also are you overclocking? If so that will really ramp up production.

How are you cooling 20 machines? even 10 in each location is going to be very warm!

Good Luck it will take a while to get it all stable, be patient! :)

I don't think it'll be choked, it only will choke and slow down the whole internet usage when they are all uploading at the same time, elsewise I'm not too worried about my system. And yes, the "monitoring rig" is infact my main rig which I use on a daily basis, yes it'll be folding. Just that the rig counts are the dedicated units for folding.

I guess I was wrong with the power it'll use, well which is fine, I can change the numbers - no big deal there. Yeah, I'm planning on having it overclocked to 3Ghz min. Like I said earlier, well I'm gonna split the 20 units into four sets of five each. They will be spaced out quite eventfully. I'll find out how to deal with the heat as the farm grows, until then, this is the only thing I can do.
For the router, you can also setup one of the folding boxes as a monitor/router box (use a VM to run either Endian Firewall or Untangle).

I have thought about doing that, probably will be setting up a webserver on that as well so that I can have the entire montioring data up on the web etc... We'll see about that.
I've just set up 5 new Quads.
The way I worked it was to first Install a copy of Win XP on a hard drive with CpuZ to monitor Vcore with, Core Temp for temp monitoring, SMP Seasaw Pro so I could set affinities easily and two copies of Orthos to stress the cpu with.
First booted with UBCD and ran Memtest86+ to check the memory.
Then booted of the hard drive and checked that it ran ok at stock speeds.
Overclocked it to what I hoped to get.
Re-ran memtest if not running memory at stock speed.
Then ran dual Orthos for 24 hours.
Once I found a stable setting I then set up the next box to the same specs and tested to see if it ran as well.
If it didn't then all the setting got lowered a bit.
Kept going like that to find a nice common overclock for all boards.
Theres less testing of the latter built boxen doing it that way.

My final common setting are FSB at 345 x 9 for 3.1 Ghz with the Vcore at +0.1v and the memory running on a 2.5 multi.
My Quads are pulling ~165 watts from the wall.
Now 2x p2605 inside the VM's gives me just over 4k per box, which is what I was aiming for.

Luck ............... :D

Great, I hope to learn more and how to configure them properly. I too want to aim for 4k per rig.
 
I'm curios, what purpose will the router have besides as needed send and download the work unit? And that little bit doesn't sound like much... since they don't need a constant connection to the internet except at those times.

Just asking since I'm in the process of upgrading my network, not for folding reasons but for other, even tho I think I put more load on my little linksys router then he will with those folders.

I'm running 2 gaming rigs, and a Dedicated Server for COD4 and it has no problems at all...

Routers balance loads, function as DHCP servers, and do network translation (among other things). If your router craps out and re-assigns IPs this WILL interrupt your folding. 90% of my hung boxes were due to the router re-assigning IP addresses because with that amount of network traffic it was losing track of what box had what IP... I would check the IP in the morning and after I came home to a hung box walla new ip... read reviews of the linksys router on newegg.com I'm not the only one that had that issue. The version I have now is not only more secure, but more robust and built to handle a higher volume of IP addresses. Its all about stability

 
Your router is going to choke! There is not one 4 port router on the market that is going to be able to handle what you propose. I just upgraded mine to a Linksys RV016 and wow... the stability has really kicked in! Also make sure you do static IPs.
While I agree with the general advice here, a $400 router is a bit overkill, even for Jon's proposed setup. A midway solution with the overall budget in mind like a Linksys BEFSR81 8-port router for under $100 should more than suffice. This is also a staged deployment. The full complement of clients won't be implemented until much later in the project.

Next the failure of the internet is not the issue. I've unplugged my router from my switch and as long as the switch stayed on the boxes were okay.
Yes, I have also noticed that. The systems connected to my router are much more prone to connection gremlins than my systems connected to my switch. For this reason among others, I have decided to move all but one system off my router to my switch.

Although cable modem is alot less reliable from my experiences. You may want to add a DSL line as well and if you use the RV016 it allows up to 7 WANS which can be any type of internet portal!
You mean it permits ISP teaming/bonding/load balancing? Is that why you purchased the RV016?

How are you cooling 20 machines? even 10 in each location is going to be very warm!
That's what I was wondering. Hot is more like it, especially in an apartment. It must be a pretty big apartment to house 20 systems comfortably in the summer, unless there's major AC going on. Seriously, how is the environmental temperature going to be regulated? It's a major consideration that shouldn't be overlooked because the ambient temperature will drastically affect the performance of your clients, comfort aside.
 
I don't think it'll be choked, it only will choke and slow down the whole internet usage when they are all uploading at the same time, elsewise I'm not too worried about my system.
Agreed. I currently have 13 SMP clients and one GPU client and I almost never have two uploading at the same time. Even with double the number of clients, I don't see a major strain on your network except on the occasion many clients will be connecting to the servers at the same time, but that will be quite uncommon. Besides, Stanford is issuing slower and slower processing WUs. The days of fast WU turnovers are numbered.

I have thought about doing that, probably will be setting up a webserver on that as well so that I can have the entire montioring data up on the web etc... We'll see about that.
That's a very interesting idea. I would like to see how that turns out. Best of luck with everything. :)

 
Good luck, Jon! Keep us informed of your plans and projects.
 
Any final comments on the hardware per se?

I want to be sure that the parts are good stuff and so on, concerns about cases, etc?
 
About the hardware, I'm having second thoughts about the motherboard. I dunno if it is good enough for overclocking...

 
About the hardware, I'm having second thoughts about the motherboard. I dunno if it is good enough for overclocking...


I have one and have not had much luck overclocking either CPU that I tried past ~315 FSB. It could be the CPUs (e2160 / Q6000s), but it doesn't seem that way.


Just my .02
 
Any final comments on the hardware per se?

I want to be sure that the parts are good stuff and so on, concerns about cases, etc?
The cases look OK from a cooling perspective given the budget. Personally, I'd simply remove one side panel off each case to alleviate any immediate concerns for heat build up. Ease of maintenance is also much improved. The fact that the cases are perfectly flat on every side is very good in the event you decide to stack them. I strongly recommend stacking due to my mental image of limited space. Correct me if I have the wrong impression. Also, get them all in black if possible. That color greatly diminishes the overall hardware 'presence' in any designated space, living or otherwise.
 
a 4 port router of any sort will do. Your router can easily handle the traffic that is bottlenecked at your modem. The computers are only going to push out the data as fast as your router can. (as fast as your weakest link) If you've got several computers each uploading they'll each just get a portion of the total bw available up. They'll just have to wait on your broadband connection. Buy a $50 router, then spend the extra $350 on upgrading your broadband connection to a faster speed for $10-30/month. (depending on whats available in your area)

Set your IP's on the machines statically, (not via DHCP) so you don't have IP conflict problems.
machine 1: ip 192.168.1.101
machine 2: ip 192.168.1.102
etc.
 
The hardware selection looks excellent, although I wouldn't be able to comment on the overclocking potential on that Gigabyte motherboard.

As for the network setup, looks good. As long as your router is running DHCP and giving out an IP address to just your Main Rig and not all 20 of the boxen, any router should do fine.
As others have suggested, make sure your set static IP addresses to each boxen.

For cooling the boxen, a simple portable AC unit should work fine.

http://www.homenethelp.com/web/howto/static-ip-address.asp - A quick guide I searched for on how to setup static IPs incase you don't know how to already :

Good luck. Looking forward to future updates :)
 
Alright, what motherboard can be suggested in exchange, I'll be happy to spend a little bit more and also I think the cases are sufficient in this sense. And for my internet, it's 12DL/1UL which should be more than enough anyways since I'm able to torrent up at speed excess of 100kilobits/sec.

The SMP WU weights in at about 15 megabits or bytes?
 
What OS are you putting on the USB flash drives?

I'm planning on running the notfred's folding on it, I may just maybe make a custom distro for it so that I can completely have control over it, ssh and all that. But I'm thinking notfred's folding'll be good enough. Though I've been hearing about some issues regarding the old work units being saved up and not being removed after it's done with. What are the options on this then?

Any suggestions on which portable AC Units to use? I'll have to check with my lease agreement whenever this is allowed or not, just want to be sure my rear-end is in all the passing lane.
 
http://www.newegg.com/Product/Product.aspx?Item=N82E16813127031&Tpk=IP35-E

Abit IP35-E they are by far the most consistent I have had at obtaining over 400mhz. The Gigabyte version has had issues with being rock stable at 400mhz both of them that I have. The 3 Abits are all easily stable at 400.

just my .02


Which brings me to another question, no video cards. How will I be able to configure this? I have another one but it's basically good as a dead horse can be - WORTHLESS. So how would I go about configuring this then. I do have this exactly board, I can run 9x400 so yeah I know what you are talking about here. One thing concerns me a little is, they're all out of stock :(
 
I'm planning on running the notfred's folding on it, I may just maybe make a custom distro for it so that I can completely have control over it, ssh and all that. But I'm thinking notfred's folding'll be good enough. Though I've been hearing about some issues regarding the old work units being saved up and not being removed after it's done with. What are the options on this then?

Any suggestions on which portable AC Units to use? I'll have to check with my lease agreement whenever this is allowed or not, just want to be sure my rear-end is in all the passing lane.

If you have central air and a good quiet box fan in each location you might be ok. I plan on keeping my AC at 70F and based on room versus mini-server room temps I see a 8F difference and that is with my unique fan solution. I've also noticed that a fan toward the ceiling blowing out of the hot room does more for cooling than fans on the floor blowing in. I experimented over the weekend with an additional Vortex fan and putting it on the floor actually dragged more hot air in then it did if I just put it on a shelf near the ceiling blowing hot air out!

Hope that helps!
 
Which brings me to another question, no video cards. How will I be able to configure this? I have another one but it's basically good as a dead horse can be - WORTHLESS. So how would I go about configuring this then. I do have this exactly board, I can run 9x400 so yeah I know what you are talking about here. One thing concerns me a little is, they're all out of stock :(

check ebay for IP35-E's ;)

Also.... if your going to fun diskless like me, get one or 2 $20 vid cards use them to set you OC settings, then remove.

I have 3 boards with no onboard vid and one PCI-E vid card between them :p The only time you will ever use it is when you need to change settings (and if you get it stable this will be almost never)

 
Am I mistaken in thinking that the suggested gigabyte board would need RAM that would run faster than 800mhz for a greater than 300 mhz oc?

I have this board and it doesn't have a 1:1 divider. I know you are looking at a different board already, but just more info for ya...
 
Am I mistaken in thinking that the suggested gigabyte board would need RAM that would run faster than 800mhz for a greater than 300 mhz oc?

I have this board and it doesn't have a 1:1 divider. I know you are looking at a different board already, but just more info for ya...

400mhz FSB = 800 mhz DDR2 at 1:1


Also to another poster SFF mobo's tend to not be the best overclockers. Remember he wants 3+Ghz so that is 333 FSB/1333 QFSB

 
It has a 2.00 multiplier which the bios is funky and I think the 2.0 is actually 1:1 having gotten one to 3.6ghz (which was a chore!) 2.0 made the 800mhz run at 800mhz which on the abit is Synch'd 1:1

I'm not a big fan of gigabyte's bios. +.1 v over what??????

oye!

 
If you went with these shuttles and you're looking at 3.0 or higher OC's, what can you to cool a Q6600 that will still fit? I'm pretty sure a TRUE 120 won't fit inside one of these things. Will a ZeroTherm Nirvana 120 fit?

 
Any final comments on the hardware per se?

I want to be sure that the parts are good stuff and so on, concerns about cases, etc?

I still do not understand this "concern about cases"

Turnelle_stack.jpg


Lay this puppy on it's side and it's a blade server.

 
Back
Top