A small datacenter build and other projects

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Haha, thanks guys.

And yes, I am a bit rotard'd. I'm building a new hobby shop to support the addition- check out FD3S Engineering on Facebook if you want to see the progress and cars I build.
 

Das_Goose

n00b
Joined
Oct 10, 2012
Messages
27
Spyder,

Do you happen to have a updated list of stuff for sale, or a link to your ebay store?
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Some tiny updates:

Check out my "ghetto" hot/cold isle. We used left over curtain from the warehouse and bought some extension ducts for the MovinCool26. It already dropped the temps 10f and once the new storage boxes are racked up, there should be no holes. I can't put a ceiling on it though.





This week we built some of our new in house storage. (1) Is 22x240gb Intel 520 series and the other two are 24x3TB.
 

DICOMdude

n00b
Joined
Sep 22, 2012
Messages
8
Yes, Solaris 11 and Napp-It.
Any experience using FreeNAS/TrueNAS in your deployment? I am thinking of FreeBSD vs illumos/napp-it variants for a Radiology image storage server (not a lot of IOps, but availability and scalability is key).
 
Last edited:

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
For us, the lack of infiniband support killed freenas and truenas licensing prevented us from using any existing hardware- so outside of demo's- no. We have just over 4PB total. Our basic station ships with 336TB raw and we add 168TB JBODS any time we need to expand. Adding them is as simple as connecting to the HBA, adding them to the pool and you're good to go. Setting up duplication between systems is pretty simple.
 

DICOMdude

n00b
Joined
Sep 22, 2012
Messages
8
For us, the lack of infiniband support killed freenas and truenas licensing prevented us from using any existing hardware- so outside of demo's- no. We have just over 4PB total. Our basic station ships with 336TB raw and we add 168TB JBODS any time we need to expand. Adding them is as simple as connecting to the HBA, adding them to the pool and you're good to go. Setting up duplication between systems is pretty simple.

Thanks for the insight! :)
 

parityboy

Limp Gawd
Joined
Nov 13, 2010
Messages
390
@The Spyder

Did you ever consider NexentaStor for your ZFS storage management? If so, what were your reasons for rejecting it? Cost? Performance?
 
D

Deleted member 238539

Guest
Spyder. I just sat here and read this entire thread. You sir are doing great things. Keep it up! I hope one day when I get to run the IT show here where I work I can do just as good of a job. Thanks for the inspiration bud.
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
@The Spyder

Did you ever consider NexentaStor for your ZFS storage management? If so, what were your reasons for rejecting it? Cost? Performance?
Our original datastation was based off it, back when there was no 18tb limit. We just recently had to look at replacements and due to things outside our controls, we were only allowed three choices of OS. As far as my in house storage goes, Nexanta, while being very inexpensive compared to the other big players, just is extra overhead. Solaris/Omni/OI with Napp-It has been stable and reliable. In the future I would love to have EMC/Netapp for my ESXi hosts, but for now, it works.
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Spyder. I just sat here and read this entire thread. You sir are doing great things. Keep it up! I hope one day when I get to run the IT show here where I work I can do just as good of a job. Thanks for the inspiration bud.
Glad you enjoyed the read! It's been a interesting journey so far. I should really update the thread with the latest progress.
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
It's been busy again around here. I am happy to say, I finally have AC for our warehouse DC. My storage project is moving along and hopefully later this fall I can order some new VM hosts to replace our 5yr old supermicro white boxes.

Let's start with a backplane change out- this was a PITA- the new chassis came with a single expander and I switched to a non-expander





I could hear this as soon as I walked into the warehouse.


Awesome dev system. Anyone guess what it is?


Inside units placed.


Crane service for the AC units- keeping it classy.




Our new production guy doesn't know his own strength.


More AC progress.




Condensate pump and power wrapped up.


Return vents completed.


Thermostat


Ouside air intake duct on the left.


View of the ducting from the other side.


External air economizer temp control.


By keeping the thermostats set at a ~75 temp, the units cycle nearly half as much. This was later set to 75. Ignore the 80 :).


Supermicro 2bay rear 2.5" drive tray.


Drives Installed


All buttoned up.
 
Last edited:

p3n

Gawd
Joined
Sep 29, 2002
Messages
631
What are those other 3 expansion cards in the storage box? Some sort of APU? No idea what that dev board is but seems to have every connector under the sun!
 

calvinj

[H]ard|Gawd
Joined
Mar 2, 2009
Messages
1,738
I want your job.... Enough said.


Looks like you have enough AC power there to keep that 68 year around :D
 

Red Squirrel

[H]F Junkie
Joined
Nov 29, 2009
Messages
9,211
Nice stuff!

Funny, I've been planing out my cooling solution for my own server room at home and it will also be intaking air from outside depending on weather. I just hit a set back though, my system was going to be based on a series of electric dampers controlled by relays that open the proper ones based on the inside/outside temp but it seems those are not very easy to find and the few places I found that sell em want like 600 bucks a pop.
 

MurphysLaww

Limp Gawd
Joined
Feb 12, 2007
Messages
182
Nice stuff!

Funny, I've been planing out my cooling solution for my own server room at home and it will also be intaking air from outside depending on weather. I just hit a set back though, my system was going to be based on a series of electric dampers controlled by relays that open the proper ones based on the inside/outside temp but it seems those are not very easy to find and the few places I found that sell em want like 600 bucks a pop.
Try this place. I see quite a few used relays running through there from time to time.

http://stores.ebay.com/liquishightechequipment
 

QHalo

2[H]4U
Joined
Sep 30, 2002
Messages
3,432
Our original datastation was based off it, back when there was no 18tb limit. We just recently had to look at replacements and due to things outside our controls, we were only allowed three choices of OS. As far as my in house storage goes, Nexanta, while being very inexpensive compared to the other big players, just is extra overhead. Solaris/Omni/OI with Napp-It has been stable and reliable. In the future I would love to have EMC/Netapp for my ESXi hosts, but for now, it works.
With 4PB, I'm sure EMC/NetApp would love to have your business as well.
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Today we finished up installing 144TB of spindle storage and 5.7TB of SSD storage for each location. The AC has been amazing, keeping the warehouse nice a mild during our 90 degree weather. (I realize this is rather mild compared to elsewhere, haha). Hopefully I can order some replacement hosts for my VM infrastructure soon, but we shall see. I've been pondering the question of to white box or to not. Dell Poweredge R720's are dollar for dollar identical to the Supermicro configs I have looked at. On the flip side, I would LOVE a set of Cisco UCS blades. I guess the budget will tell :).
 

QHalo

2[H]4U
Joined
Sep 30, 2002
Messages
3,432
If you must go rack mounts and you can't get Cisco blades, look at their C Series stuff. I was able to get it cheaper than my R720s.
 

schnell

Gawd
Joined
Jul 22, 2005
Messages
763
If you must go rack mounts and you can't get Cisco blades, look at their C Series stuff. I was able to get it cheaper than my R720s.
Huh? UCS chassis are rack mountable.

From an infrastructure management standpoint UCS is the the way to go. Best investment our company has ever made.
 

SysOps

Weaksauce
Joined
Feb 22, 2010
Messages
72
Awesome dev system. Anyone guess what it is?


looks like a advantech SOD-DB5800.
What are you using it for?
they are a interesting peace of tech...
i used a similar Computer on Module system in a "extremely" remote site.

keep up the good work
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Internal R+D project I can't speak about. We may or may not use them due to some issues we have come across. Neat little boards.
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Seem's there is a rather lack of updates on my behalf, sorry about that. With our second restructure, things have really quieted down and frankly I am not sure what's going to happen. While things have settled, I'm having a hard time keeping my tech busy. Not trying to rant, but it's the downside to all the awesome hardware/systems here.

We did replace our Unifi Pro AP's with AC ones- it fixed our problem with Apple clients dropping.
 

101001101

Weaksauce
Joined
Aug 8, 2013
Messages
69
I just spent the best part of a couple of hours reading this thread, awesome tech, how do you find the SuperMicros in production? solid? many failures? Also on the subject of cars, I have a weekend warrior Subaru STI with 400bhp, its particularly awesome in the snow; that is if you remember all that awesome grip comes when you apply power, as soon as you let off its basically a sledge unless you have studs or chains :D
 

Yariko

n00b
Joined
Oct 24, 2013
Messages
11
Someone has the most epic job! Building a small datacenter and keeping it alive (+ all the other stuff you're doing)... And all those portable units... :drool:

Very nice! Must be so much nicer to have the freedom compared to a bigger datacenter environment.
 

Jay_2

2[H]4U
Joined
Mar 20, 2006
Messages
3,583
I had a site in the middle of Kenya and another at the top of a mountain in Chilli, that's remote! :D
 

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
Thanks for the continued support. I won't rant about the company I work for, but let me put it this way- some less then popular decisions have been made recently by the upper management and no one is happy- including myself. Things are rocky in the contract world and we sadly laid several people off due to refocusing our efforts, including my tech.

Some good news:
A handful of projects has been approved and I can resume posting cool stuff like this:


It's going to be a interesting year- I can promise that.
 
D

Deleted member 238539

Guest
I am so happy this thread keeps popping up. I even take time when I am slow and find it again to re read the entire thing.
 

Hadrien

Limp Gawd
Joined
Mar 17, 2005
Messages
327
Some good news:
A handful of projects has been approved and I can resume posting cool stuff like this:


It's going to be a interesting year- I can promise that.
Hmm homebrew GPU compute node or maybe some sort of GPU offload for VDI? Been foaming at the mouth for GPU offload for VDI for awhile. Would love to slice and dice our engineering boxes into VDI clients that can take resources as they need it.
 

Fenris_Ulf

[H]ard|Gawd
Joined
Nov 29, 2004
Messages
1,727
Awesome stuff Spyder. I was an end-user of your company's service last year in the desert. I always wondered what hardware was used for the processing and storage and it's more impressive than I thought. Nice to see the evolution from a few dozen TB to the PB level.
 
Top