Small datacenter build-out

[RiDDLeRThC]

Figured I would start a similar thread to The Spider outlining the buildout of our new datacenter.

We are renovating our current space (10th floor of an office building). The current datacenter will remain up until the new one is finished. Over one weekend we will move all the current gear and then demo the old room.

We are using this time to replace our aging 15kVA battery backup, 2-post network racks, and current AC units.

Hopefully you guys enjoy this build thread. For now I can leave you with just the current Visio drawing. Build-out is slated to start somewhere around 11/8-12/5.

I'm sure some will say the room is too small and the rack clearances aren't that great but I had very little space to work with. I think we did a pretty good job maximizing the space. We have an IT work area on our 4th floor that we will stage the gear before bringing it up.

Look forward to any questions.

 

[RiDDLeRThC]

Floor Plan

Demolition has started...

Old computer room still running...

 

[RiDDLeRThC]

reserved for additional updates

 

[p3n]

Is that a window at the top? I presume servers etc will be 'facing' this? If so where is the hot/cold separation?

 

[RiDDLeRThC]

Correct, 1hr fire rated floor to ceiling window.

We have talked about doing a hot aisle containment but haven't really decided on it yet.

The original plan didn't include the Liebert units, those were just added. We had just planned vents behind the racks to just put the hot air into the plenum space. With the return air vents are on the front of the Liebert's we will need to come up with something. We may need to put a box on the right side of the air handlers that goes up to the ceiling for return air.

 

[RiDDLeRThC]

Taking into consideration the return air. We are going to switch out the AC unit and just stock a few spare parts.

 

[Ockie]

You should ditch the DS unit and go with two or three CRV's instead. A CRV produces 35KW (10Tons) in a line matching cabinet footprint. It will take up less floor room, provide better air management, more peak cooling capacity, more ideally designed for your senario... and the most important part is that you would have some sort of redundancy. The DS units are great, but they do break down, and when they do, you're going to regret it.

 

[RiDDLeRThC]

I'm struggling with the change to the DS from the two Challenger units. We will be stocking spare parts but its still scary for me. Such a little room will heat up quick.

Spare parts we would stock are
microprocessor
fan motor
condenser motor
spare belts & filters
humidifier bulbs

 

[Ockie]

I'm struggling with the change to the DS from the two Challenger units. We will be stocking spare parts but its still scary for me. Such a little room will heat up quick.

Spare parts we would stock are
microprocessor
fan motor
condenser motor
spare belts & filters
humidifier bulbs

No, not challenger units, but rather CRV's instead of the DS. With a single unit you can't account for ground out protection when your compressor blows, breaker failures, refrigerant leaks, compressor failures (although you should have two on a 12ton unit), main shaft failure, shaft bearing failures, blower fan becoming unbalanced, high head shutout due to dirty coils, and coil cleaning (fans needs to be off which knocks out your associated unit). Having parts on site isn't as much as a problem as much as having contractors available on that moment of time to do a quick swap (a compressor takes a good two days... a shaft replacement takes quite a bit more time)... and these failures usually happen on very hot days when contractor houses are slammed with calls.

Sounds like you already have some challengers... in that case, I would rather add more for capacity while still maintaining redundancy while keeping your costs down.

 

[schnell]

I would agree on the CRV statement. You have the space to extend the rack row for 2 of them if you remove the DS unit. Otherwise if your building can supply chilled water you can go that route. I use APC in row chilled water units and they are great.

 

[RiDDLeRThC]

We currently have nothing, we were talking originally about going with two 5ton challenger 3000's.

The AC units in our old computer room is are water cooled units but we want to move off the dependency of the building.

Its either a single DS or two 5 ton 3000's. I wasn't a fan of the CRV's but will look at them again. We have no raised floor so piping the units would be interested.

 

[RiDDLeRThC]

Had a good talk with our AC vendor today, I got a little more insight into why he recommended the DS (mostly to save costs).

Hes making some modifications to the plans for the Challenger 3000's and I'm pretty sure that will be the route we go.

I appreciate everyones feedback.

 

[SysOps]

i would go with the two systems, or have a emergency plan.

If you go with the one primary unit, like i had to, then make sure you have a back up plan. i have a contract with a local vendor to have two 5 ton mobile AC units at my site in under 3 hours. I also have two 1.5 tons mobile units for spot cooling and because they are handy..

In the design of my server room, i incorporated automated external air blowers / ductwork gates to permit the use of filtered outside air to save power and for emergency cooling. (at one point it was controlled by a arduino before i got a industrial controller)
During the winter, my primary AC unit usually only turns on for its its automated maintenance cycle.

 

[RiDDLeRThC]

We are doing 2 3000's and things will be fully redundant.

 

[RiDDLeRThC]

finally started the demolition/construction on our new computer room... added a few pictures above

Walls should be going up starting end of next week.

 

[RiDDLeRThC]

Some UPS Install Pictures and Startup also have some AC units

 

[p3n]

hah the 90 degree switch is very dramatic

 

[Red Squirrel]

Cable and power porn. *fap*

 

[NTAuthourity]

Love the time they are taking to make sure it is level. We had Emerson come in to install our Liebert UPS systems and they messed up on 2 Battery units next to the PDU and we cannot open the doors..

 

[Red Squirrel]

At the hospital I used to work at they put a Powerware UPS too. Pretty nice systems. Never got to see the management side of it though, there was so much red tape in that place the whole install took over a year and there was a need for a jack to plug the management in and that was a whole can of worms that never got opened.

Are these UPSes dual conversion? I wonder at what point it's more cost effective to do a telco style setup where you have rectifiers, inverters, and a battery bank. I imagine these arn't cheap. I guess they are more turn key than a telco setup though.

 

[RiDDLeRThC]

this one set us back around 30-35k, not too bad. Started at 20kva and can be changed to 30 and 40kva with a small payment and a tech onsite to install a license key.

 

[VisBits]

Is this thread dead ? I'd love to see updates!

 

[RiDDLeRThC]

The day after the UPS turn-up we had someone hit a sprinkler head in an office outside of the room. No equipment was damaged but the flooring was. We are on hold till that gets taken care of. Should be back on track middle of next week.

 

[Red Squirrel]

The day after the UPS turn-up we had someone hit a sprinkler head in an office outside of the room. No equipment was damaged but the flooring was. We are on hold till that gets taken care of. Should be back on track middle of next week.

Ouch, it's not a dual stage system? Where I work the system is air pressurized, if a sprinkler head goes it releases the air which triggers one stage. A smoke detector also has to go off before it puts water into the system.

 

[MrGuvernment]

watching!!

 

[RiDDLeRThC]

Computer room is pre-action and requires a smoke detector and a head to pop. Room also has a clean agent and only uses water as a last resort.

The rest of the floor is normal wet pipes

 

[VisBits]

Crazy!

 

[jeffmoss26]

Very cool pics, thanks for sharing!!

 

[RiDDLeRThC]

few more pictures, things are starting to come together. will post pictures of the racks and what not tomorrow. The fire rated glass was being installed when I left today.

 

[radgoos]

Is it good for that to be running at 99% capacity?

 

[RiDDLeRThC]

It has no load so I'm guessing that's the battery capacity.

 

[radgoos]

Oh right, I was thinking about it all wrong.

 

[RiDDLeRThC]

VNX 5800 delivery today

 

[cyr0n_k0r]

I've never understood why EMC can't put their crap in a normal rack.
Or let you choose a rack model you want and they will put it all together for you then ship it to you.
What if your entire datacenter uses APC racks, then you have this non standard junker rack at the end of a row.

 

[RiDDLeRThC]

You can order the gear without the rack...

on a VMAX the way the drive trays are mounted, i dont think a normal rack wouldn't work.

 

[Red Squirrel]

How much is that rack in terms of disk space?

 

[RiDDLeRThC]

65tb usable

Some more detail

19 200gb SLC EFD (Fast CACHE)
18 200gb eMLC EFD (Fast VP)
92 600gb 15k SAS
25 1000gb NL-SAS

 

[VisBits]

65tb usable

Some more detail

19 200gb SLC EFD (Fast CACHE)
18 200gb eMLC EFD (Fast VP)
92 600gb 15k SAS
25 1000gb NL-SAS

The $/Iop those units give you is terrible. Your into that what half a mill? Those 200gb drives are nearly $3500/ea from emc if not more.

 

[RiDDLeRThC]

We aren't paying that much for them.

 

[VisBits]

We aren't paying that much for them.

Can't be paying to much less, even refurbished drives are $2500 right now.