Datacenter folks! Talk to me about doing a ground up build.

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
I have been given a huge project and a great opportunity at my new job and need some input from the community here.

What I am starting with:
Large open 1000sqft warehouse
200 amps 220v single phase
150 amps 240v 3phase (WYE) (Upgradeable to 800amp)
5 Chatsworth TerraFrame N series 36" deep racks with vertical cable management (27.5" wide)
Cable raceways and grounding
EIther 1, 2, or 3 30amp circuits at each rack.

The power and racks were put in by someone before me and sadly, they are short depth N series for large frame routers. And they are expensive :(. Since the initial reaction from my boss was, well. not good, when I told him we needed to re-do everything that was put in, I will be making what we have work for now. The N series will work for my main processing and storage clusters, as long as I use 2u PDU's, but this limits me to two 18 plug PDU's (due to current power limitations). Two 48" deep Chatsworth are on order for just my IT stuff.

As far as power goes, I am afraid I am going to have to rip out the existing. We will be installing a new 800amp 3phase main and because of this I am thinking of using a switch box off the main and connecting a large 80kva line filter, then in to a sub panel for distribution for the first row of racks. In the future, I can add the same thing at the end of each new rack and just change to the new 800amp service at the main.

Cooling is going to be... interesting. The current idea is to use large fume hoods above the rear of the racks with a central roof mounted exhaust fan. Fresh air will be pushed in down the front of each rack from a side wall, and curtains will divide the hot/cold isles, along with the rest of the warehouse. I am looking at getting two Big Ass Fans to mount in the ceiling, to push the hot air down so it can be extracted. I can have a roof mount AC, but I am limited to 20tons. (Which is just enough for 10 fully loaded racks).

So far the big things I can identify that we will be doing different are as follows:
1) Our gear runs at 100%. Each 1u is a Quad Hex Core xeon, maxed on ram, linked by Infiniband.
2) We want to try "green" cooling methods, such as external air cooling.
3) Due to the constant product cycle/ ect only two of the racks will have doors. The others will be open for easy access by engineering.
4) No battery backups on the eng/rd gear, just a power filter.
5) No raised floors/cracs. The CEO wants to keep a "warehouse" feel- this may bite me this summer, we will see.

My questions thus far are:
1) What racks have you used? Prefer?
2) Has anyone dealt with Tripp-Lite for large battery backups/ power filters? We use them for everything else.
3) Has anyone worked in a open air datacenter before?



I am headed in to work is afternoon to get a clamp meter on the current gear to see what type of power we are actually using. A engineer told me one of our 10u (40 processor) systems with controller (anther quad hex core) is the most they can fit on a 30amp circuit- one more node and it trips. Next up will be getting a LVE in Tuesday, meeting with HVAC Thursday, and getting quotes for the new power setup. I am going to have one heck of a week. Thankfully Friday and Monday I have Datacenter tours from some local friends. :)

Input is welcome, pictures to come!
 
Last edited:
Thanks for the read. I found a bunch of good white papers on Datacenterknowledge.com and this google book.

http://www.morganclaypool.com/doi/pdf/10.2200/S00193ED1V01Y200905CAC006

One of my biggest hurdles is that this is a small environment that operates at a much higher power usage then others I have worked in. The increased power and resulting HVAC needs are crazy and if I do NOT do this right from the start, we will suffer later. I am trying to avoid that.
 
Call APC and get one of their Infrastructure people out there ASAP.
They have pretty much turnkey solutions for a lot of what you are wanting to do.
 
APC In Row Cooling..........

other than that wish I had something more productive to say
 
You should be calling in to the other big players of power and cooling like Libert. Use their knowledge to both help and back you up, always best to cover you bases.
 
I will make some calls. My biggest concern is that I have no idea what type of a budget I am working with. I need to have a good chat with my boss in the AM. Some of these big guys are not cheap, but so far we would have been better off calling APC/ one of the larger companies from the start.

I should have a better idea next week of what I am dealing with/who to call in the area. I am definitely diving in head first. It is pretty darn cool to be here only a week and be trusted to handle such a large build out. I just wish I did not have to handle the desktop support too.
 
Liebert for sure, they are very rock solid- just had a whole lab (1000 square foot or better) dont by them with redundant cooling hot/cold aisles, and it is just the best ever.
 
Cooling is going to be your problem. Everything else is simple.

I would consider the APC in-row cooling solution as previously suggested. Or build a raised floor area, and push the cold air under plenum. Since you said this is a warehouse, what is the ceiling height?

EDIT: Sorry, just re-read your post and saw that your boss outlawed the raised-floor. I would engage APC to at least provide a proposal and go from there.
 
Liebert looks perfect for my powerfiltering/monitoring and for my cooling needs, holy crap! The CRV is perfect for my setup.

I called APC and I am at least a week out until I can get someone out here. Oh well, plenty of time to drool over the Liebert units...
 
Last edited:
Your biggest obstacle is going to be your fiances (or at least what your boss is willing to spend).
 
Last edited:
Liebert looks perfect for my powerfiltering/monitoring and for my cooling needs, holy crap! The CRV is perfect for my setup.

I called APC and I am at least a week out until I can get someone out here. Oh well, plenty of time to drool over the Liebert units...

This may seem obvious but it doesn't hurt to bring it up...

Make sure the Liebert is specced properly for your space and heat load, we have a Liebert CRV and while the unit is good, the company who installed it overspecced the model for our small datacenter and we've had nothing but problems because we can't put enough load on it - something about the coolant not fully cycling through the system because it's not running for long enough. I'm no HVAC pro, but I suggest that you do some research on the company you choose to spec out the environment. To be fair, the CRV was brand new when they implemented the system, but that's no excuse, I'd be sure they have plenty of experience with the equipment they plan to install.
 
Your biggest obstacle is going to be your fiances (or at least what your boss is willing to spend).

Definitely had this reinforced at my meeting today. My boss wants me to stage the upgrades, which is completely doable. Considering all the other work I am doing at the same time, I can not blame him. I have to build a new VM host, two new 130tb NAS, and rebuild our core network.

I am going to start with the networking (which needs to be done anyways), get the power filtration and distribution in place, order all the accessories (pdu's, UPS's, cable management, ect), install the environmental monitoring, and then install two Isis fans from bigassfans. When we start to hit above 80, I will already have the HVAC ready to be installed. It should work out fine :).

I should have the plan drawn out some time this week and I will get some pictures up.
 
Aaron, hopefully this will not be an issue. We have a very large area and some VERY hot gear. Our nodes are composed of 144 core Xeons, 128gb of ram, and 300+ terrabytes of storage. We will have at least 5 of these running at once. They should not be under full load 24/7, but they will be operating at much higher then normal conditions.
 
One thing to note. Make sure you have backup power for your HVAC system. You can build all the redundancy + generator for your power needs, but it won't mean squat if your HVAC system isn't online during an outage.
 
One thing to note. Make sure you have backup power for your HVAC system. You can build all the redundancy + generator for your power needs, but it won't mean squat if your HVAC system isn't online during an outage.

Yep, and also it needs to be calculated what happens to HVAC during an outage. The UPS system can't power that, so you have 10-15 seconds of no HVAC if you have super fast gensets, longer if you don't.

Apparently, it's been an issue a few places that the HVAC takes forever to catch up if the outage is too long. Truthfully, I have no personal experience here, but Ive heard it been talked about enough that it must be a serious consideration. But I think you're building small scale, so probably not an issue.
 
Last edited:
APC in row cooling, with the in row coolers also on backup power. Take a look at Symmetra PX in a 250K frame. Awesome stuff that is scalable (ie buy what you need up front to keep costs down) and parts are hot swappable.
 
From the conversations I have had the past few days, it looks like I am doing a bare bones build out. Which is ok. We are a growing company and right now I have the budget for 30kVA of power conditioning (which matches our available 3phase). As soon as we max it out, I can bring in 800 amps of 3phase and buy a new 100kVA power conditioner.

If the power goes out, so does 95% of the gear which requires the most cooling. If it goes out for longer then the run time of my UPS's, we will shut it down and go home. As long as the data in the sets gets written to the battery backed up storage arrays, we are good :). I am just happy that I get a big enough budget to outfit 7 racks, with filter power, proper power distribution, cat6a network runs everywhere, new switch gear, new servers/sans, and proper cooling. Plus the 800 little things I am forgetting.

This is such a cool job :)
 
From the conversations I have had the past few days, it looks like I am doing a bare bones build out. Which is ok. We are a growing company and right now I have the budget for 30kVA of power conditioning (which matches our available 3phase). As soon as we max it out, I can bring in 800 amps of 3phase and buy a new 100kVA power conditioner.

If the power goes out, so does 95% of the gear which requires the most cooling. If it goes out for longer then the run time of my UPS's, we will shut it down and go home. As long as the data in the sets gets written to the battery backed up storage arrays, we are good :). I am just happy that I get a big enough budget to outfit 7 racks, with filter power, proper power distribution, cat6a network runs everywhere, new switch gear, new servers/sans, and proper cooling. Plus the 800 little things I am forgetting.

This is such a cool job :)

So you're not going to have backup power at all?
 
So you're not going to have backup power at all?

depending on the nature of the work being done by the servers, this isn't always a requirement...

sounds like the op's system is crunching numbers... not something that needs to be particularly redundant, as far as computing goes...

probably wants to hold on to those numbers though :p
 
I didn't read the whole thing. However right off the bat I have a few questions.

Where is this warehouse located? City, State.

Have you considered co-locating your equipment?

Water supply at the warehouse?

Network interconnects? Multiple ISPs available?

For cooling work with a local multiple manufacture representative. They can design a system for your load, redundancy, and combine different cooling technologies to get the best performance out of your location (free cooling, humidity, etc...)

Raised floors is the thing of the past. Hot/cold isle is the most efficient way to cool. APC has amazing products for just that.

Be sure to think of how you are modulating everything for future expansion. Standardization is also key.
 
Leibert and a local electrician are coming out Monday. The HVAC contract is coming out Tuesday and hopefully next week I will find out if we have fiber available between the two locations. I got a tour of the Pittock building this week, but my tour at a local University tomorrow was just canceled. I am getting some good ideas and a better grasp on VA, BTU, kW, kVA, and every other fun term/calculation possible.

Honestly I am a bit disappointed, I was told I was going to be able to build this out like I wanted and money was not a huge concern. I am finding for daily purchases, this is true, but these big projects take quite a bit of planning/review/approval. Which is honestly a good thing, I am just not used to it yet. There are just a unbelievable number of other large projects that I have to stage this around, which means everything is going to take more time.

Some of the other projects include:
Building two identical new NAS (40tb)
Rebuilding two existing NAS (140tb raidz2) with SSD's and the latest version of Napp-it
Rebuilding VMHost2 with more ram, move VMHost1 to rebuilt Host2, rebuild VMHost1 and purchase vCenter.
Replace phone system, link site to site
Replace core switches and all switchgear
Get a grasp on our current software licensing
Setup AD,WDS,WSUS, ect
.....

My list just keeps growing.... Oh well, at least I am not bored :-D
 
Last edited:
Leibert and a local electrician are coming out Monday. The HVAC contract is coming out Tuesday and hopefully next week I will find out if we have fiber available between the two locations. I got a tour of the Pittock building this week, but my tour at a local University tomorrow was just canceled. I am getting some good ideas and a better grasp on VA, BTU, kW, kVA, and every other fun term/calculation possible.

Honestly I am a bit disappointed, I was told I was going to be able to build this out like I wanted and money was not a huge concern. I am finding for daily purchases, this is true, but these big projects take quite a bit of planning/review/approval. Which is honestly a good thing, I am just not used to it yet. There are just a unbelievable number of other large projects that I have to stage this around, which means everything is going to take more time.

Some of the other projects include:
Building two identical new NAS (40tb)
Rebuilding two existing NAS (140tb raidz2) with SSD's and the latest version of Napp-it
Rebuilding VMHost2 with more ram, move VMHost1 to rebuilt Host2, rebuild VMHost1 and purchase vCenter.
Replace phone system, link site to site
Replace core switches and all switchgear
Get a grasp on our current software licensing
Setup AD,WDS,WSUS, ect
.....

My list just keeps growing.... Oh well, at least I am not bored :-D

I wish I got to do all of that...
 
Ok. Haha, it has been a crazy week. First up, all of my projects have been OK'd by my boss as unclassified. Pictures and info are allowed!

Second, I had Liebert out and I am 100% sold on a CRV. Their APM however (battery backup) is about 16k more then I want to spend... It is very nice, but damn expensive.

Third. I hate electricians. It took three quotes to get even a reasonable idea of what we need to do. All three gave a different picture. The third one I was hesitant on using, as my boss is a friend of his, but he is the only one who seemed decently competent.


Question! I need to order three racks, preferably 27 wide by 48 deep. Possible deeper. I am looking at Tripp Lite and APC tomorrow. Liebert and Chatsworth are just too expensive IMO ($6.6k for three racks).
Has anyone used the large 15va APC Smart UPS VT? Or rather can anyone recommend a 15-20va UPS?
 
Last edited:
Has anyone used the large 15va APC Smart UPS VT? Or rather can anyone recommend a 15-20va UPS?
The SmartUPS is a great line. I think they go all the way to 20kVA. I've dealt with many of their 8kVA's and 10kVA's, rock solid.
If you need more than that though you'll need to move to their symmetra's which I loathe.
 
I have installed a few 6-10k ones before and I was hoping to hear just that :) I am meeting a APC rep here next week. I need a few UPS's for multiple sites and it looks like they will be either Tripp Lite or APC.
 
Last edited:
Back
Top