The Spyder
2[H]4U
- Joined
- Jun 18, 2002
- Messages
- 2,628
I have been given a huge project and a great opportunity at my new job and need some input from the community here.
What I am starting with:
Large open 1000sqft warehouse
200 amps 220v single phase
150 amps 240v 3phase (WYE) (Upgradeable to 800amp)
5 Chatsworth TerraFrame N series 36" deep racks with vertical cable management (27.5" wide)
Cable raceways and grounding
EIther 1, 2, or 3 30amp circuits at each rack.
The power and racks were put in by someone before me and sadly, they are short depth N series for large frame routers. And they are expensive . Since the initial reaction from my boss was, well. not good, when I told him we needed to re-do everything that was put in, I will be making what we have work for now. The N series will work for my main processing and storage clusters, as long as I use 2u PDU's, but this limits me to two 18 plug PDU's (due to current power limitations). Two 48" deep Chatsworth are on order for just my IT stuff.
As far as power goes, I am afraid I am going to have to rip out the existing. We will be installing a new 800amp 3phase main and because of this I am thinking of using a switch box off the main and connecting a large 80kva line filter, then in to a sub panel for distribution for the first row of racks. In the future, I can add the same thing at the end of each new rack and just change to the new 800amp service at the main.
Cooling is going to be... interesting. The current idea is to use large fume hoods above the rear of the racks with a central roof mounted exhaust fan. Fresh air will be pushed in down the front of each rack from a side wall, and curtains will divide the hot/cold isles, along with the rest of the warehouse. I am looking at getting two Big Ass Fans to mount in the ceiling, to push the hot air down so it can be extracted. I can have a roof mount AC, but I am limited to 20tons. (Which is just enough for 10 fully loaded racks).
So far the big things I can identify that we will be doing different are as follows:
1) Our gear runs at 100%. Each 1u is a Quad Hex Core xeon, maxed on ram, linked by Infiniband.
2) We want to try "green" cooling methods, such as external air cooling.
3) Due to the constant product cycle/ ect only two of the racks will have doors. The others will be open for easy access by engineering.
4) No battery backups on the eng/rd gear, just a power filter.
5) No raised floors/cracs. The CEO wants to keep a "warehouse" feel- this may bite me this summer, we will see.
My questions thus far are:
1) What racks have you used? Prefer?
2) Has anyone dealt with Tripp-Lite for large battery backups/ power filters? We use them for everything else.
3) Has anyone worked in a open air datacenter before?
I am headed in to work is afternoon to get a clamp meter on the current gear to see what type of power we are actually using. A engineer told me one of our 10u (40 processor) systems with controller (anther quad hex core) is the most they can fit on a 30amp circuit- one more node and it trips. Next up will be getting a LVE in Tuesday, meeting with HVAC Thursday, and getting quotes for the new power setup. I am going to have one heck of a week. Thankfully Friday and Monday I have Datacenter tours from some local friends.
Input is welcome, pictures to come!
What I am starting with:
Large open 1000sqft warehouse
200 amps 220v single phase
150 amps 240v 3phase (WYE) (Upgradeable to 800amp)
5 Chatsworth TerraFrame N series 36" deep racks with vertical cable management (27.5" wide)
Cable raceways and grounding
EIther 1, 2, or 3 30amp circuits at each rack.
The power and racks were put in by someone before me and sadly, they are short depth N series for large frame routers. And they are expensive . Since the initial reaction from my boss was, well. not good, when I told him we needed to re-do everything that was put in, I will be making what we have work for now. The N series will work for my main processing and storage clusters, as long as I use 2u PDU's, but this limits me to two 18 plug PDU's (due to current power limitations). Two 48" deep Chatsworth are on order for just my IT stuff.
As far as power goes, I am afraid I am going to have to rip out the existing. We will be installing a new 800amp 3phase main and because of this I am thinking of using a switch box off the main and connecting a large 80kva line filter, then in to a sub panel for distribution for the first row of racks. In the future, I can add the same thing at the end of each new rack and just change to the new 800amp service at the main.
Cooling is going to be... interesting. The current idea is to use large fume hoods above the rear of the racks with a central roof mounted exhaust fan. Fresh air will be pushed in down the front of each rack from a side wall, and curtains will divide the hot/cold isles, along with the rest of the warehouse. I am looking at getting two Big Ass Fans to mount in the ceiling, to push the hot air down so it can be extracted. I can have a roof mount AC, but I am limited to 20tons. (Which is just enough for 10 fully loaded racks).
So far the big things I can identify that we will be doing different are as follows:
1) Our gear runs at 100%. Each 1u is a Quad Hex Core xeon, maxed on ram, linked by Infiniband.
2) We want to try "green" cooling methods, such as external air cooling.
3) Due to the constant product cycle/ ect only two of the racks will have doors. The others will be open for easy access by engineering.
4) No battery backups on the eng/rd gear, just a power filter.
5) No raised floors/cracs. The CEO wants to keep a "warehouse" feel- this may bite me this summer, we will see.
My questions thus far are:
1) What racks have you used? Prefer?
2) Has anyone dealt with Tripp-Lite for large battery backups/ power filters? We use them for everything else.
3) Has anyone worked in a open air datacenter before?
I am headed in to work is afternoon to get a clamp meter on the current gear to see what type of power we are actually using. A engineer told me one of our 10u (40 processor) systems with controller (anther quad hex core) is the most they can fit on a 30amp circuit- one more node and it trips. Next up will be getting a LVE in Tuesday, meeting with HVAC Thursday, and getting quotes for the new power setup. I am going to have one heck of a week. Thankfully Friday and Monday I have Datacenter tours from some local friends.
Input is welcome, pictures to come!
Last edited: