A small datacenter build and other projects

Hope you like your phone, i have the same one and love it.

DSCN0034.JPG
 
Nice stuff as always! I think I have the same db meter, I got it when I was running sound. VERY helpful!!
 
I downloaded and installed FreePBX for right now. This is just for eval purposes. I have a few local friends who build/sell/support Asterisk based systems that recommended it. One is running 50 Yealinks/FreePBX and with the recommendations I saw here and from other sites, I wanted to give them a try. Nice phones so far. This is my first real dive in to phone systems outside of old PBX's, so there is a bit of a learning curve.
 
Any use of 3cx? I'm a windows guy so always prefer

Me! We've got 142~ extensions running on it. A mix of Polycom 650's primarily. With some Yealinks peppered into the mix. As well as some Grandstream ATA's for analog stuff like paging and cordless phones.

We've been running it for about a year now. Our only current hiccup is some SIP issues with nexVortex and an issue with 3CX v11 that does not show when you have a missed call from an outside line on our Polycoms. However we have a support ticket open with 3CX on that one and will be moving our SIP services away from nexVortex. Possibly to Time Warner in the next 9-12 months. Maybe sooner depending on what we have going on.
 
More awesome stuff Spyder!

Also a side story about DB Meters, since your post reminded me of it and thought it would be funny to share. So I work in a telco central office and we had a coop student. It's a tradition to eventually get the student to go get a bucket of dial tone. It's the telco version of sky hooks. Well we did something different. Gave the student a db meter to go check the distribution frame for dial tone leaks. :D
 
Well, it finally happened- after 9 months I have burnt out. Last week was just hellish. I got incredibly frustrated over a simple fix, and I hardly ever get truly ticked. We have had some very very stressful projects going through the production side of things and the stress is getting to everyone. My solution: re-group, get through 200 resumes, and get someone hired. It's going to end up being a slow week, but otherwise I would go insane :)

More pictures of what little I got done last week.

I asked our vendor if they had one of the new i7 ITX supermicro boards, this showed up.
Bankhl.jpg

pjx4Ll.jpg


And this lovely little D525 atom came with it. (2x1.8ghz, 4gb, 40gb SSD- takes a DC power brick input.)
POGknl.jpg

TBTZal.jpg


Some very, very heavy boxes showed up.
QnkjHl.jpg


And so did nearly 700tb of disks + controllers.
BmzKWl.jpg

eBk0bl.jpg


Got all the racks blocked off.
rPeFql.jpg

Tynuzl.jpg

Ul0Sel.jpg

ranrzl.jpg


I keep telling my boss it would be cheaper just to fill them full of servers, but for some reason he thinks I'm full of it. haha.

Found this today after some office shuffling- hope the monitors OK- they are temperamental suckers.
S2gpll.jpg
 
Last edited:
BmzKWl.jpg


The 2 shelves at the bottom of the case with all of those drives models are those?
 
Well, it finally happened- after 9 months I have burnt out. Last week was just hellish.


Spyder, be careful. I have personal experience with a health crisis that was a result of running myself @ redline for months... Not pretty.

Get that IT assistant hired, so we can continue to enjoy and be inspired by the good work.
 
Last edited:
I caught myself well before I had any physical issues. It happens to everyone here at some point, we all work ourselves down, take a bit to recover, and go back at it. I have sadly watched it happen a few times to coworkers.

As far as the Supermicro boxes, we deal mainly with IXsystems who handles the assembly, burn in, and RMA's. For internal use, I build my own using CDW/Provantage/Newegg. No complaints so far. IMO, they are great boxes. We have over 1000 deployed with only a handful of bad powersupplies being the
biggest issue. (High heat/Desert environment)
 
Nice to see those Supermicros are treating you well, I'm strongly considering going that route from now on for my server stuff. Been thinking about upgrading my whole environment. Wont be a while till I do that though, have to save up first. Those 2.5" enclosures look pretty nice... I never really considered 2.5" drives for mass storage but considering even at the consumer level you can get like 1TB 2.5" drives now it's definitely a neat option. Think they are less susceptible to vibration issues too.
 
Sadly I am working on quite a bit of production tasks (Writing guides/researching tape backups for 5.xPB of data...) and I have very little to post.

Here is a small teaser of other projects I have been working on- Mike will recognize this.

WUU7ml.jpg
 
Almost looks like industrial size electrical tape. What someone would use if they nicked a 500,000 volt cable by accident. "Bah, just wrap it in electrical tape and call it a day" :D

Says PVC Strip though, I'm guessing it's something else, some kind of covering for something.
 
My guess is its for a hot isle air curtain.

However it does look like a massive roll of electrical tape.
 
yes it does took like tape,
but as stated before it must be for a curtain to separate the hot/cold air.
But how wide is it , because it looks to narrow to me to be the actual curtain
but please keep on posting the pics spyder, its really cool to see that stuff you do.
 
Last edited:
You'd be correct in assuming it was for hot/cold isle separation. As those vents on the floor are the cold air and the vents on the ceiling behind the cabs looks like the return air. No sense in mixing the air up, I did the same thing with our two cabs. We'll be doing the same thing when we build our new building as well. :)

I would not be surprised if they used blocking plates in the cabs with empty space as well.

Still on a smaller scale though. :(

Either that or it's late and I'm mixing up comments. Either way I'll find out tomorrow. :D
 
Mike is correct-
It's 8" wide Anti-Static curtain material for hot/cold isle separation.
I have a roll of 48" on order too :).


336TB and two controllers magically appeared in my rack.
uZ3PQl.jpg

Next to it is 144TB- same project. Almost have a Petabyte racked now. Can't wait to get my other 336TB SAN :). Wayyyy to much storage :)

Oh and racecar runs again :D
AKR8Yl.jpg

(It got a fresh motor/turbos/larger injectors/better clutch/tons of freaking time and money.... They don't call them (F)inancial (D)isasters for nothing :D)
 
I need that type of storage. Great looking racks.

rotary motors....how do they work? lol. no im not serious but if you see them in action, they make no sense.
 
I need that type of storage. Great looking racks.

rotary motors....how do they work? lol. no im not serious but if you see them in action, they make no sense.

lol! i agree completely. some manner of sorcery i guess. or aliens. :p
 
Magical spinning triangles on a stick?

Boost in-> Apex seals out?

3 moving parts-> 99 possible problems??

:D I can make fun of myself right? Honestly, there is not another car that drives like a 3rd Gen RX7 for under $50k.
 
Good to see another Mazda guy. I have two. 2003 6S and 2009 CX-9 Grand Touring. Yeah, the rotarys are weird. Saw a four rotor on youtube yesterday.
 
Magical spinning triangles on a stick?

Boost in-> Apex seals out?

3 moving parts-> 99 possible problems??

:D I can make fun of myself right? Honestly, there is not another car that drives like a 3rd Gen RX7 for under $50k.

I agree that the car itself was built incredibly well in terms of handling. Damn near 50/50 weight distribution, but they just need to do something with the actual driving force of the car.
 
Good to see another Mazda guy. I have two. 2003 6S and 2009 CX-9 Grand Touring. Yeah, the rotarys are weird. Saw a four rotor on youtube yesterday.

they make some cool power, however they make more heat than anything else, 90% of the issues are heat related..
 
Let's not turn this in to a Rotary thread :). I do agree the RX8 has a terrible engine/coolant seal issue and don't get me started on idiots at Manna- the reman company. The FD however mainly suffers from a lack of maintenance. I have flogged the every living heck out of mine for years and never had a engine issue, outside of the control system no longer turning on the primary turbo due to a bad check valve. :)
 
So a new theoretical was placed before me today: Build a 60 node cluster.

Our current hardware would give us 720 cores/11.5TB ram/15.5TB of localized storage for 30U of rack space and need 2 36 port IB switches, 2x 48 port for IPMI, and 30 power cords.

Switching to 6 7U Twinblade chassis would provide 1920 cores/61TB ram/28.8TB of localized storage/Built in 40GB QDR IB with no cabling, 24 power cords, and 12 network cables. Sadly it costs about $400k more then what the current system would....

I have no idea how a blade system would hold up in the environment we are working with.

224- Cores per 7u 4 processor 8 core per processor
320- Cores per 7u TwinBlade 4 processor 8 core per processor
 
So a new theoretical was placed before me today: Build a 60 node cluster.

Our current hardware would give us 720 cores/11.5TB ram/15.5TB of localized storage for 30U of rack space and need 2 36 port IB switches, 2x 48 port for IPMI, and 30 power cords.

Switching to 6 7U Twinblade chassis would provide 1920 cores/61TB ram/28.8TB of localized storage/Built in 40GB QDR IB with no cabling, 24 power cords, and 12 network cables. Sadly it costs about $400k more then what the current system would....

I have no idea how a blade system would hold up in the environment we are working with.

224- Cores per 7u 4 processor 8 core per processor
320- Cores per 7u TwinBlade 4 processor 8 core per processor



If your environment involves high heat and sand like i think you are talking about, it will not be good.
 
I ended up spec'ing the following to replace our 1366 based systems.

1U Supermicro TWIN 2011 E5-2600 with 40GB Infiniband
4x E5-2670 2.66ghz/20MB cache/8 core
16x 16GB DDR3 1600 ECC REG
2x Intel 520 240GB SSD

$10.5k in just parts (fair market value)

Should be getting one ordered here in the next few weeks for testing. This gives us 256GB of ram/32 Cores per 1U vs the existing 196GB/24 Core we currently run. Less power too!
 
Repeat "I love my job." Even at a cool place like this, politics can suck, burn out happens, and time gets wasted. My only solace is for every 1 BS thing I have to deal with, I get to do 10 cool things :). Hiring has been temporarily frozen, so has capital expenditures. Combine those with long hours on 30 projects at a time = burned out. It's been 2-3 weeks now and I haven't even began to recover. Hopefully this is just a small bump in the road. I have a failing SAN that holds our main company files, no backups for the VM hosts, and a ton of infrastructure spending coming up next year. The biggest being AC for the datacenter. Most importantly though is I need a helper, like yesterday. Rumor has it this is only until the 1st week in December after a visit from the main program managers is over.

/Rant.

I have some really neat photos to show you guys once I find my card reader. :)
 
Repeat "I love my job." Even at a cool place like this, politics can suck, burn out happens, and time gets wasted. My only solace is for every 1 BS thing I have to deal with, I get to do 10 cool things :). Hiring has been temporarily frozen, so has capital expenditures. Combine those with long hours on 30 projects at a time = burned out. It's been 2-3 weeks now and I haven't even began to recover. Hopefully this is just a small bump in the road. I have a failing SAN that holds our main company files, no backups for the VM hosts, and a ton of infrastructure spending coming up next year. The biggest being AC for the datacenter. Most importantly though is I need a helper, like yesterday. Rumor has it this is only until the 1st week in December after a visit from the main program managers is over.

/Rant.

I have some really neat photos to show you guys once I find my card reader. :)

Awesome, can't wait to see them :)
 
Repeat "I love my job." Even at a cool place like this, politics can suck, burn out happens, and time gets wasted. My only solace is for every 1 BS thing I have to deal with, I get to do 10 cool things :). Hiring has been temporarily frozen, so has capital expenditures. Combine those with long hours on 30 projects at a time = burned out. It's been 2-3 weeks now and I haven't even began to recover. Hopefully this is just a small bump in the road. I have a failing SAN that holds our main company files, no backups for the VM hosts, and a ton of infrastructure spending coming up next year. The biggest being AC for the datacenter. Most importantly though is I need a helper, like yesterday. Rumor has it this is only until the 1st week in December after a visit from the main program managers is over.

/Rant.

I have some really neat photos to show you guys once I find my card reader. :)

I thought you had been able to hire an assistant already? Has that still not happened??
 
Take it easy, when the projects will slow down, take some time to travel and dont think of work.
 
heh i would have loved to be a part time student intern/coop at your place... work for free and get job experience and a letter of recommendation... might look into that if that's possible, good for the physical time consuming stuff without letting them be dangerous...

Having the flexibility and resources to set things up as you want is a great gift... I've had to finagle and haggle to set things up the way I want... didn't setup backups because I focused on the steps to recover from failure... and we had no budget for it...

They can only push you as far as you let them... Remember to stop and breathe... take a 3 day weekend at least once a month... call out sick and let them consider what happens if you really get sick...
 
So as it turns out, everything worked out. :) A Teir 1 tech was brought on for another project that got canned, so I adopted him. He doesn't start for two weeks, but it's better then nothing! This week several program managers from the government department that oversees our programs are visiting for a inspection. In prep we reorganized our warehouse, cleaned, and I got a ton done.

Anyways, pictures!

Did I mention I refuse to step foot in the R+D lab? Jebus. Tie down straps on $30k of SAN on 3" of foam. Can''t blame them though, lost 9 disks to the sound system from the bar below. I just lost 6 in my in house storage for the same reason. My solution is to move the SAN to our warehouse racks though.
5W0ftl.jpg


Found this in there too. L6-30P to two L6-30R's.
RJhR3l.jpg

Added it to the collection of other WTF were you thinking electrical nightmares.
nqrnJl.jpg

My curtain showed up.
HH8HKl.jpg

In the mean time, I built a small workstation for system testing/validation/troubleshooting. Missing is the antistatic mats.
S1msxl.jpg

Aluminum showed up for the wall.
je6k5l.jpg

Finally the hangers showed.
p3UXEl.jpg


Starting on the wall... the building is 1/4 out of square over 20ft... sweet!
3HbFil.jpg

XqVlHl.jpg

uw263l.jpg

Y034Jl.jpg

Built a TV mount for the small conference room, looks 10x better then a folding table.
xPUI0l.jpg


Uh-oh. Someone opened the mains... And I worked 12h on Saturday.
KB3spl.jpg

For my new 200amp 240v 3ph panel! I can finally use all 400 amps of service instead of the 350 the last panels limited me to.
5QL19l.jpg

Added some more outlets for the racks.
xYW6dl.jpg

uWBazl.jpg

And another 200amp 3ph panel :) in prep for a 400amp 480v service and AC. :)
C4l4ql.jpg

i8Tbcl.jpg


My boss came in over the weekend and pressure washed the driveway for our visit from the government project managers...
gvoQnl.jpg


Racks have not changed much.
jMhqIl.jpg

NoyAkl.jpg


Added another full depth rack to move the above mentioned SAN from our headquarters into.
I7AuOl.jpg


More wall progress.
qmFOGl.jpg

pbjlDl.jpg

I6GUal.jpg

bHOcll.jpg

pqV8rl.jpg

nVCg7l.jpg


And it is done!
L4tAal.jpg

3o5opl.jpg

mjiYEl.jpg

MRIsHl.jpg

P3lyMl.jpg

rnnCkl.jpg

2tVztl.jpg

YMNFcl.jpg


So far we have a 20db drop from 20ft away, which is huge. It means I will need AC ASAP, but I should make it in to the spring no problem. The wall structure is 2x2x1/4 6063 aluminum and should easily support a matching roof system to finish off the cold isle. AC will be a ducted in from rooftop units. Did I mention both my boss and the CEO now think I am some crazy fabricator/car guy/IT guy.... Wait a minute! They're right! :D

The next update will probably be a bit. I am taking a long needed vacation and working on rather boring projects. Hopefully I will start ordering hardware to rebuild my VM cluster and triple our in house cluster. Until then! Servus!
 
Back
Top