Official Red's server room thread

Lames.

Gawd
Joined
Nov 17, 2007
Messages
736
If you do not mind me asking - how much have you spent in wood prepping all of this?
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I have no idea tbh. I bought about 3k worth of lumber (that includes the dricore tiles) for my entire basement project. At one point I also did another purchase for the plywood sheets, 2x6's and the door, which came up to a bit over 500 bucks. I still need to decide if I put some acid drip trays as well. Got a quote from a company to custom make some to fit and it would be around 700 bucks for 5 of them.

Still have to buy an electrical panel too, and redo lot of the electrical, not only for the room, but the house. Lot of circuits that are tapped into other circuits that I'd want to rerun from this new panel. If I had to guess, I probably put maybe a bit over 2k into this between lumber and other supplies but I may need to put another 1k easily.
 

wizdum

[H]ard|Gawd
Joined
Sep 22, 2010
Messages
1,943
I have no idea tbh. I bought about 3k worth of lumber (that includes the dricore tiles) for my entire basement project. At one point I also did another purchase for the plywood sheets, 2x6's and the door, which came up to a bit over 500 bucks. I still need to decide if I put some acid drip trays as well. Got a quote from a company to custom make some to fit and it would be around 700 bucks for 5 of them.

Still have to buy an electrical panel too, and redo lot of the electrical, not only for the room, but the house. Lot of circuits that are tapped into other circuits that I'd want to rerun from this new panel. If I had to guess, I probably put maybe a bit over 2k into this between lumber and other supplies but I may need to put another 1k easily.
Save yourself some money and pick up some thermoform plastic on Amazon and fit it to your shelves with a heatgun.
 

scotty do

Limp Gawd
Joined
Jun 20, 2004
Messages
164
Not sure I still get the point of all this...

You mentioned your reason for going with an automotive battery was to save cost. I don't see how the cost of inverter, acid drip trays, shelving, hydrogen monitoring, etc. can cost less than a new or refurbished UPS. Especially taking into consideration the safety issues of charging indoors could potentially put your family at risk.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Runtime and expandability. A conventional UPS that can last for as long as mine would cost 10's of thousands of dollars, and be very proprietary and closed, so getting replacement batteries would cost a lot too. There is a certain upfront cost to all this but in the end I will save when it comes time to expand capacity.

In the future I'd also like to look at a solar or wind turbine setup, so I would probably tie into these batteries, or add more batteries. I will only be using 1 shelf with the existing UPS setup, leaving room for lot of expansion.

The safety issues are also pretty small with only 4 batteries but all this stuff will ensure safety even as I add more. Been running with 2 batteries for over a year now with zero issues.

Also I googled thermoform, isin't that just a covering for kitchen cabinets? I need to actually contain the acid and have it flow to a bigger container, otherwise there's not much point to doing anything. If I do cheap out I will just use vapor barrier sheets on the shelf, so it will protect from small droplets from when I do maintenance, but wont really protect from a cell breaking. I may be overthinking all this though, even at work with the 96 big ass 4,000 ah cells there are no drip trays or anything. They just have some spill kits around the room. Reactive, instead of preventive, I guess. :p

I wonder if trays could be made with a big enough 3D printer though... is the result of those fairly strong for this kind of application? I still need to get more quotes from different companies. I only need 1 tray now but may as well get 5 identical ones made and piped so it's a done deal.
 

Brak710

[H]ard|Gawd
Joined
Oct 27, 2008
Messages
1,424
You probably could have built your own standby generator for less.

All you would need was a rackmount ATS, and a portable generator that can be remotely turned on. Have the ATS in front of your small UPS units so it gives the effect of two sources of utility power.

Just put your generator outside in a dog house type enclosure, run a power line into the house into the B side of the ATS. From there, have a Raspberry Pi or something monitor when the ATS loses A side, it powers on the generator to bring up B side.
 

PigLover

[H]ard|Gawd
Joined
Jul 11, 2009
Messages
1,180
Yup, he could have...used 3kw apc ups with new(ish) batteries regularly on eBay for ~$400 shipped. 3kw generator with ats and autostart ~$800. Less cost than the lumber alone. Much less. And safer.
 

Brak710

[H]ard|Gawd
Joined
Oct 27, 2008
Messages
1,424
Yup, he could have...used 3kw apc ups with new(ish) batteries regularly on eBay for ~$400 shipped. 3kw generator with ats and autostart ~$800. Less cost than the lumber alone. Much less. And safer.
Couldn't agree more.

I appreciate the dedication on projects like this, I still think it's an awesome build... But when you're building acid spill catch trays, and using wood which burns better than metal... It's just not something you plan on running your kit on 24/7 inside your house...

OP, you clearly had the budget to go all out and hit it out of the park. I would have gone with this genset and called it a day. And with that size, you could done just done your whole house. Generators also add value to your house if you live in areas with unreliable utility/storm prone areas.

Fuels is cheaper than battery soon after the point you want longer than 1hr of runtime on high wattage.
 
Last edited:

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Remember guys I'm in Canada, everything is more expensive here. A standby generator starts AT LEAST a couple grand, though the smallest available would be plenty for the whole house and home depot does sell them. I still need to hire a licensed pipe fitter to install it assuming I go with natural gas which would make the most sense for this. I've thought of it, but that's a whole expense for another day. Might also just install a generator inlet and get a portable, though the 120/240 models are still a grand or two so it almost makes sense to go with a Generac if I'm going to get one.

For rackmount UPSes I'm looking AT LEAST 1 grand. Most places on ebay are asking like 900 bucks minimum just for shipping alone. They list it at like 200 bucks, but you have to look at the big picture, it's the shipping where they get you. For equivalent capacity I'd have to buy UPSes anyway + genny, it would cost WAY more. Let's not forget the price of gas. (though if I go natural gas that's probably not that bad) Most of the cost that are going towards the server room would also apply no matter what. The walls and ventilation system are not strictly for the batteries, I still want that room to be isolated and have it's own ventilation system for cooling. The cooling will be optimized so I will never need a dedicated AC unit as it will use a combination of indoor and outdoor air based on the temperatures.

The drip trays and all that are way overkill, I'm just playing it extra safe. These marine batteries are designed for harsher environments than what I'm subjecting mine to so their chance of a crack is very small.

Lot of my cost has also went towards tools that I did not have, which I would have at some point bought anyway.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Yay my vertical PDUs just came in! I had almost forgot about those. One will be fed by the UPS and the other one by a surge protector. Did not get a chance to look at them, had to leave for night shift.
 

scotty do

Limp Gawd
Joined
Jun 20, 2004
Messages
164
Where in Canada do you live exactly? I am in Southern Ontario, and while a UPS is more expensive than the US, it's no where near $900.

What are you running that you need such a high capacity setup? Have you considered Colo? A UPS isn't meant to run for hours. As mentioned before...a generator is the way.. Solar or wind would cost way more than a generator & natural gas.


To each their own however...
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I'm in Northern Ontario. Colo would cost more because I'd be paying per month vs all this being a one time cost, and I would not have physical access not to mention network access would be limited by what I can afford to pay for there and my internet connection (which is good, but not gigabit good).. Nice to be able to just walk down and have access to everything. In fact if my ISP allowed servers I'd probably move my online stuff like my websites to my basement. I'd be able to upgrade ram, disk space or w/e without having to pay extra per month. Most of this stuff is basic services I use around the house like my home automation/monitoring server, file server, mail/spam filtering server etc... I do plan to eventually build a proper VM server though. My main server does have VMs on it too that are used for various things but I'm limited by it's 8GB of ram.

So once all this setup is done and cleaned up the next step is adding more equipment. Possibly a Cisco lab too, that's what the beige rack will be for mostly, and future rectifier/inverters. I'll stick with the 750w inverter-charger setup for a while though.
 

Berg0

[H]ard|Gawd
Joined
May 16, 2005
Messages
1,038
Wow, I've got a 15KVA liebert in my garage I would have sold you for a lot less than you've spent, haha
 

Berg0

[H]ard|Gawd
Joined
May 16, 2005
Messages
1,038
Pay for it and it's all yours, sitting in my Saskatoon warehouse right now. All of my gear is moving into colo, so I have piles of racks, servers, SANs and UPS's, pretty much everything to start a small datacenter.
 
Last edited:

/usr/home

Supreme [H]ardness
Joined
Mar 18, 2008
Messages
6,161
Pay for it and it's all yours, sitting in my Saskatoon warehouse right now. All of my gear is moving into colo, so I have piles of racks, servers, SANs and UPS's, pretty much everything to start a small datacenter.
I'm listening... lol. I'm up to Stoon almost every 3 weeks or so. Bit broke ATM though :(. Going to do a FS thread or anything?

(Sorry for the thread hijack)
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
lol nice, I'd hate to see the cost to ship that though. :p But damn, 15kva, that would do the whole house. I'd just put it right before the whole panel, nevermind the server room. :D
 

Berg0

[H]ard|Gawd
Joined
May 16, 2005
Messages
1,038
lol nice, I'd hate to see the cost to ship that though. :p But damn, 15kva, that would do the whole house. I'd just put it right before the whole panel, nevermind the server room. :D
yup, I contemplated it. I have a 100A ATS and an extra subpanel for it, along with all the wire, twistlock plugs etc.

/usr/home PM me, don't want to highjack the thread, wasn't planning on doing a FS thread on here, I mostly sell the equipment through my company to other local IT providers.
 

wizdum

[H]ard|Gawd
Joined
Sep 22, 2010
Messages
1,943
lol nice, I'd hate to see the cost to ship that though. :p But damn, 15kva, that would do the whole house. I'd just put it right before the whole panel, nevermind the server room. :D
If I wasn't building a garage right now, i'd dust off my snowmobile and go pick it up.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Sub panel installed:


Let there be light! Soooo much better now. Installed 4 lighting fixtures today, 2 on each side: (all these pics taken without flash, just to show how bright it is now)









In the process of running a temp feeder to the electrical panel I also added a proper outlet and I am now feeding my stuff off it, so I retired my trusty ol' 1960's hanging plug:




Still need to decide how I want to feed that panel though, right now it's just a temp 15 amp feed. Running a 8/3 cable would give me 40 amps and be easier, but running conduit with 2awg feeder cables would be the right way, so I can make it a 100a sub panel giving me more options. Probably only do that next year.
 

bds1904

Gawd
Joined
Aug 10, 2011
Messages
1,007
Still need to decide how I want to feed that panel though, right now it's just a temp 15 amp feed. Running a 8/3 cable would give me 40 amps and be easier, but running conduit with 2awg feeder cables would be the right way, so I can make it a 100a sub panel giving me more options. Probably only do that next year.
What would the total length from one panel to the other? In some places 6ga 60A is code depending on the distance. 60A 240V will be more than enough for anything.

Personally, I would just run 4ga THHN and run 60A. If I remember correctly canada requires anything more than a 30A panel to have a 6ga stranded ground run back to the main ground braid. Ground and neutral must also be separated in the subpanel, that is everywhere though.

P.S. Let me know when you want to get some of those netapp enclosures off your hands;).
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
What would the total length from one panel to the other? In some places 6ga 60A is code depending on the distance. 60A 240V will be more than enough for anything.

Personally, I would just run 4ga THHN and run 60A. If I remember correctly canada requires anything more than a 30A panel to have a 6ga stranded ground run back to the main ground braid. Ground and neutral must also be separated in the subpanel, that is everywhere though.

P.S. Let me know when you want to get some of those netapp enclosures off your hands;).
Without going down to measure I'd guess about 20 feet. The big issue right now is my main panel is recessed as I had to frame around it so there is no room to add conduit. But I may get the panel moved up so I can insulate behind it, and depending on how much that cost's I might just get the electrician to change it out to a brand new one. At that point there will be more room for conduit. So I'll probably end up going that route and go with the 4 or even 2ga cabling. I have a bunch of 2ga telcoflex cable already but don't think I'd have enough considering I'd need 3 runs + ground. Oh and currently the ground is also separate. I made sure to keep it that way. TBH I'm not sure if what I did is actually legal though, but it's temporary, and it's still safe. Basically it's fed by a 15 amp breaker with 14/2 + ground, but the hot is split into two (pig tailed) to feed both sides of the panel. The ground is connected to the panel and not bonded to neutral.


As for the IBM enclosures I've thought of giving/selling them as I barely use them and they're more there for show. The biggest issue is getting them into a box suitable for shipping, and actually carrying them to the shipper. They're over 100lbs each and I can barely lift them, was quite a job to get them where they are. I'd also hate to see the cost of shipping that. :eek: I'll stick with the top two which have 400GB drives but I'd be willing to let go of the two bottom ones at some point maybe as I don't really ever use them. I kinda ruined one though by putting the drives in another system and now they don't register. The IBM enclosures are really picky, you can't just put any random drive in there.
 

bds1904

Gawd
Joined
Aug 10, 2011
Messages
1,007
So the netapp's have a raid module in them or are they just JBOD modules? I would think that if they were JBOD it wouldn't really matter. I've use xiotech and HP FC arrays and as long as they are just JBOD controllers it doesn't care what drives are in use.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
They are jbod but for some reason they're still very specific on the drive.They're sata drives rebranded by IBM (they're not Netapp, they're IBM) and they have some kind of special flash on them. Oddly if you put the drive in a regular system it also breaks this flash. Learned that the hard way...


But yeah if I connect the fibre to the fibre HBA each drive shows up individually in the OS, which is quite nice as I just use mdadm raid. Nothing like sying 56 drives show up on a single system. :D
 

bds1904

Gawd
Joined
Aug 10, 2011
Messages
1,007
what are the model numbers on the broken drives and the enclosure? I've had to re-flash drives before for some brand of enclosure (don't remember which one, it was a while ago). Can't be hard to fix your issue.

I have a 14x 146GB array set up right now, 4x RAIDz1 with 2x hot spairs. 2Gb/sec FC split loop (7 drives per FC connection). I get 360MB/sec all day out of the array. I serve my 2x esxi boxes with them. Combined with a good SSD cache drive I see 2-3ms latency max even when recording 6x hd streams from my HDHomeRun Prime's, playing back 4, someone surfing on a VM, and vcenter server constantly writing to the drives.

FC+ZFS=Awesome & Cheap. I paid like $200 for my entire drive setup and a quad port fc controller.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
This is how the drives look:



I looked into this before and I'm pretty much out of luck though. There is no way to actually interface with the enclosure, no serial port or anything like that.

This is the controller:

 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Installed room outlets, including one for my fibre, so I don't have that ugly extension cord going across the floor anymore:



Two #10 BX cable runs to the rack:
Currently two 15a circuits (hardware store had no 20a breakers) but when I get the chance I'll swap for 20a breakers. Used the #10 because I had it on hand, and I can upgrade to a 30a twistlock in the future if needed. This will be the main feed that UPS(es) plug into.






More or less done the AC electrical now. I also glued a few pieces of trim around the racks to finish off, then tomorrow I'm hoping to prime and maybe even get a 1st coat of paint in.

Getting closer and closer to completion... then I can save up and buy some more servers!

I can't wait to do the cable management too, once those ducts are all painted and ready to go, will be nice.
 

agrikk

Gawd
Joined
Apr 16, 2002
Messages
930
Red,

Beautiful work, man. I'm looking forward to the finished product and love all the in-progress pictures. Thanks for sharing!
 

parityboy

Limp Gawd
Joined
Nov 13, 2010
Messages
390
@Red

With that drive being a Serial ATA drive, is there no way to remove it from its enclosure? If one were to be sent back to IBM (or Maxtor), would they not have some way of extricating that drive from its surroundings?
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
@Red

With that drive being a Serial ATA drive, is there no way to remove it from its enclosure? If one were to be sent back to IBM (or Maxtor), would they not have some way of extricating that drive from its surroundings?
Oh they come out and are normal SATA drives, but the enclosure itself will not accept any random SATA drive, nor will it accept one of it's own drives if it was used in another system (learn from my fail). That makes the enclosures sadly almost useless once too many drives fail. So far I did not get any failures though.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
I don't know if it's worth calling this an "update", but I started to add labels on stuff. It's a sign that I'm near completion. :D



Though, I'm debating on swapping those outlets out for 30a twist lock plugs (the wiring is rated for 30a - had it on hand), and running 2 more 20 amp plugs, perhaps even more. Once equipment starts going into this rack it wont be possible to run any more outlets. Though, I don't plan to put any major equipment in there for a while. In the future the rectifiers and inverters will go in there, but for now it will just be my inverter/charger sitting on a shelf.



Need to do the other side cable management system then I'm ready to paint.



I also installed the door to the room, which I forgot to take a pic of, so I'll post it later. :p
 

gatekeep

n00b
Joined
Mar 20, 2006
Messages
24
Oh they come out and are normal SATA drives, but the enclosure itself will not accept any random SATA drive, nor will it accept one of it's own drives if it was used in another system (learn from my fail). That makes the enclosures sadly almost useless once too many drives fail. So far I did not get any failures though.
I *think*, we discussed this in the other thread quickly... IIRC, you only have the expansion controllers for the enclosures without a head controller, which would only allow you to see the enclosure as a giant JBOD.

He needs the IBM FastT head controller, that has a serial port on it. You can then login to the port and do some tricks to make the controller accept any disk. These controllers are an LSI based design. I outlined a procedure to hack them on another forum (for an Sun StorageTek enclosure, but the IBM is the same design): http://forum.hddguru.com/viewtopic.php?t=19247&start=

FWIW, the drives aren't so much as "flashed" as they contain data that is easily erasable when the drive is used else where.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Hmm so could I technically use dd to make any drive work? I thought the sector size was different too? Though I should be at least able to maybe fix the drives that I broke by putting in another system. And yeah this was talked about as well, I think it was determined I would need the head unit to be able to do anything.
 

gatekeep

n00b
Joined
Mar 20, 2006
Messages
24
Hmm so could I technically use dd to make any drive work? I thought the sector size was different too? Though I should be at least able to maybe fix the drives that I broke by putting in another system. And yeah this was talked about as well, I think it was determined I would need the head unit to be able to do anything.
Well here is the problem with using dd, the actual sled the drive is in has electronics on it that communicate with the enclosure backplane. What is done is that there is a section of the disk (the last 40MB or so) that contain data called "DACStor" that contains information such as the enclosure serial number, some signature data and particularly the sled serial number (the sleds have unique serial numbers).

The reason you can't just dd the DACStor region from one disk to another is because the cloned DACStor wouldn't contain the correct sled serial number. Basically the disks are "married" to the sleds at the factory.

The procedure I outline in the link I sent (if you had a full controller), outlines a method to force the controller to accept disks that weren't factory signed, as well as a procedure to force the controller to write a new DACStor to the disk inserted. This process will basically marry the disk to the sled for you. However, again, because there is no way for the controller to sign the DACStor region (again this is done at the factory), you have to do some tricks to force disable disk signing in the controller.

This procedure isn't possible with the JBOD expansion controllers you have however, because those controllers don't contain a serial connection and are really just "dumb" expansion devices with small amount of logic to run things.

And as far as the sector sizes, it is possible for the controller to do odd things like 520 bytes per sector instead of 512 bytes per sector. But I've personally never seen it with the LSI based controllers (which is what the IBM Fast-T and Sun/LSI StorageTek enclosures use).
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Wow this was a task and a half... believe it or not this took 3 hours to paint!..... and this is just the primer!





Getting closer to the end though. Tomorrow I will apply the black paint. I really hope I can get away with one coat, but that's probably wishful thinking, considering it's black. It's an enamel paint so it should be interesting to see the results, it will have a bit of a shine to it.


Oh, and there's a door now!




Once the paint is done, I have to do the DC electrical and PDU install, debate on if I want to run some more outlets, then I'll be more or less done. Hoping to be done mostly everything by end of this week. This project has dragged enough, just want to finish it, so I can start looking at buying new servers now. :p

As far as the entire basement project goes I just have a small section of dricore to do then I'll be done so at least there's that. I definitely put more work into the server room than I had originally planed for with this whole project. Still have to do the hvac and ceiling insulation/air seal but that will wait till next year probably. hvac will consist of a custom HRV system with bypass mode based on outdoor/indoor temp.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
And the paint is done!

Some spots could use a touch up though, but I'll leave that for the end, if I feel like it. :p Need to get a small paintbrush for some of those.






Side:



Back:










Tomorrow I should be good to install the two PDUs and start on the DC wiring. Running a full backup of my server environment right now to prep for the shut down when I move the inverter-charger. The main server is kinda sketchy when it comes to being shut down, it usually wont come back up on it's own without replacing a couple drives. Once this basement project is settled I will want to start looking at migrating over to my new storage box. I should have enough UPS capacity for it and the extra batteries will make up for the extra load in terms of run time.

The big thing too is now I'll be able to organize the cables better, that will be another project for later, but I'll do the power right away as to avoid having to shut down the main server again.

The board above the file cabinet will have the power meter for the UPS, as well as any misc button/displays I choose to add in the future. I'm also going to get two 100a battery switches so I can isolate the batteries for maintenance. I'll have two banks basically, so I can take one offline and pull the batteries out to maintain. Because of how tight they will fit in the shelves I don't think I'll be able to maintain them in place.

Still need to decide what to do about the acid mitigation, those trays are kinda pricy. Might go with battery boxes, I'll have to buy 4 of them and make sure they fit properly. Not really that worried about a leak, these batteries are designed to be bounced around in boats and RVs, but I figure better safe than sorry.
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Decided to put two batteries per shelf instead of 4. This enabled me to get away with buying cheaper plastic containers as I have more room to play with on each shelf. The shelves were designed to fit exactly 4 batteries, with the plan to get custom trays made, but figured if ever I need that kind of density I can always do that later. So two batteries per shelf (4 batteries total) and they'll be in two banks so they can be isolated easily if I need to remove them for any reason such as replacement. Best part is, I faced the caps towards the door so I should be able to maintain them without removing them from the shelf. Have to criss cross the wires of course, or I'd be making a 24v short. :p



I will place a box of baking soda in each container so if by chance there is a leak it will absorb into the box and get neutralized by the baking soda. I imagine the box would eventually crumble and it would release it's content. I don't want to pour it into the container as there will be fans in there and I don't want that stuff blowing around and possibly getting inside the batteries.

Should get the job done I imagine. Those containers can't hold the content of the entire battery, but the odds of all 6 cells going at once is quite slim I'd say. If something like a hurricane or earth quake happens I have much worse things to worry about than a couple litres of paltry battery acid. :p

Should be able to move the power over to it's proper setup by end of tonight. Since the battery switches arn't in I can just connect the wires together for now.

I also have some fuses that I'll be using:




I'm onto the fun stuff now, and getting near the end! Then I can save up to upgrade my aging server.
 
Top