The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Zarathustra[H];1041179933 said:
It would be nice to move to the same model they use in the photography, and show off your rig threads.


Have an edition for every year.

That way things stay fresh, and we don't wind up with these ridiculous huge threads that are difficult to muddle through.
I like that idea. Look for a new thread for 2015.
 
not as much as you would think, according to the 2 UPS' the whole rack is drawing <2500W

Yikes! 2.5kw :eek:

Well, I guess it's not much for an entire rack.

My entire (48TB, 32TB available) basement setup (dual socket Xeon, 12 cores/24 threads total) with 96GB RAM, 13 spinners, 5 SSD's, 24port switch, Wireless AP and FiOS ONT box and two APC Smart-UPS units combine to average just over 300W, while running ESXi and 5 guests.

That being said, you probably live in a cheaper electricity market than I do. :p
 
Zarathustra[H];1041208052 said:
Yikes! 2.5kw :eek:

Well, I guess it's not much for an entire rack.

My entire (48TB, 32TB available) basement setup (dual socket Xeon, 12 cores/24 threads total) with 96GB RAM, 13 spinners, 5 SSD's, 24port switch, Wireless AP and FiOS ONT box and two APC Smart-UPS units combine to average just over 300W, while running ESXi and 5 guests.

That being said, you probably live in a cheaper electricity market than I do. :p

you sure about that? average spinning disk uses ~12W so just your disks are half your consumption... 96gb of ram in 16gb sticks is 6 sticks, a 16gb stick of ram is about 7W so thats another 42W right there... most efficient 6 core xeons were like 60W each?

about 1KW of my rack is just drives
 
you sure about that? average spinning disk uses ~12W so just your disks are half your consumption... 96gb of ram in 16gb sticks is 6 sticks, a 16gb stick of ram is about 7W so thats another 42W right there... most efficient 6 core xeons were like 60W each?

about 1KW of my rack is just drives

Well, that's what I register on my kill-a-watt and it corresponds with the percentage load on my APC Smart-UPS.

Drives are all 5400rpm reds, which are essentially green drives, from a power perspective. Low power L5640's and 12 8 GB sticks of low voltage RDIMM's.

All that said a lot of it is probably due to the fact that it idles a lot of the time, despite 5 guests running.
 
Wow 2.5kw continuous, that's a nice heating offset. LOL. My rack only pulls about 400w last I checked. I need to get a true RMS meter so I can get more accurate readings though, I don't think the one I have is true RMS so it could be way off. My PDU does say 3 amps though (no decimal point so it could be close to 4)

Funny thing is I find the last 2 servers I built use way less power than my core i7 workstation. That box alone uses like 300w. Guess the server stuff is more efficient, though it also helps it's all motherboards with built on video . Video cards now use tons of power.
 
Wow 2.5kw continuous, that's a nice heating offset. LOL. My rack only pulls about 400w last I checked. I need to get a true RMS meter so I can get more accurate readings though, I don't think the one I have is true RMS so it could be way off. My PDU does say 3 amps though (no decimal point so it could be close to 4)

Funny thing is I find the last 2 servers I built use way less power than my core i7 workstation. That box alone uses like 300w. Guess the server stuff is more efficient, though it also helps it's all motherboards with built on video . Video cards now use tons of power.

we have racks here that exceed well above 10kW, 2.5 isn't a lot
 
Indeed it isn't much. Racks at work (50U) have 70kw of power available, but we try not to go over 30kw usually. We have the cooling capacity to back it too. My personal colo is using about 1.6kw and the rack isn't even 1/4th full. Home rack is down to about 350w ever since I moved the majority of the servers to the datacenter.
 
Indeed it isn't much. Racks at work (50U) have 70kw of power available, but we try not to go over 30kw usually. We have the cooling capacity to back it too. My personal colo is using about 1.6kw and the rack isn't even 1/4th full. Home rack is down to about 350w ever since I moved the majority of the servers to the datacenter.

wow 70kW is a lot, but with blade servers and such, it goes quick!
 
Maxed out my rack runs at 1450w or so. Keep in mind that's 10 processors 37 spinners (in 2 sas expander enclosures) and 2 ssd's as well as a quanta lb4m switch. What's even more impressive is that it's 11 power supplies.

Not too bad in my eyes.

Zarathustra[H] how much do you have invested in the system without drives? I would be interested in calculating power savings for myself.
 
Last edited:
wow 70kW is a lot, but with blade servers and such, it goes quick!
Yep, it's pretty much all blades. Superdomes were the biggest power hogs followed by the C7000 chassis we have. Unfortunately posting pictures would jeopardize my employment. :(
 
L
Zarathustra[H] how much do you have invested in the system without drives? I would be interested in calculating power savings for myself.

Well, mine is actually somewhat of a ghetto system built on the (relative) cheap, apart from the case and drives. A lot of these parts I already had from other various projects and upgrades, and I kind of piece by piece upgraded my way to this point, so I never spent a ton of money all at once.

In order to put together something like mine in one go though, it would look like this:

Necessary parts (excluding drives):

CPU: Pair of L5640 Xeon CPU's: $120 for the matched pair used on eBay
Motherboard: Supermicro X8DTE: $150 New on ebay. No I/O plate
I/O Plate: $8 on eBay
RAM: 12 8GB low voltage DDR3 Registered ECC sticks. $40 per stick from seller in [H] FS/FT
SAS Controllers: 2x IBM M1015 for ~$100 each on eBay
Case: Norco RPC-4216 $320 on Amazon
SAS Cables: 4x for $39 on Monoprice
PSU: Antec Earthwatts 550W 80 plus Platinum: $95 on Newegg
Adapter cable for second EPS power: (PCIe to EPS): $6
CPU Coolers: 2x SNK-P0040AP4: $48 for both on Amazon

Sub total $1,466

Optional Extras:
120mm fan divider: $10 on eBay
Fans: 3x Yate Loon 120mm D12SH-12: $15 Amazon
Fan Controller: NZXT Sentry 2 $23 Amazon
Hot swap for 5.25" bay: $14

Sub total $62

Drives:
Boot drive/Datastore: 128GB Samsung 840 Pro $120
L2ARC Cache Drives: 2x 128GB Samsung 850 Pro $120 ea
SLOG ZIL Drives: 2x Intel S3700 100GB $189 ea
Storage Drives: 12x WD RED 4TB $169 ea

Sub Total $2,766


Total: $4,294

(Actually a lot more than I thought, seeing it all added up at once since I bought one piece here, and another there over a span of 4 years.)

Results in this guy:

15268538436_a8ac02e4ed.jpg


I didn't include the spare WD Green drive I have in the 5.25" hot swap bay, the keyboard, UPS, old tiny LCD monitor I picked up on ebay for $30, ETC.

Worth it? I don't know. I like it, but looking back on it, that's more money than I thought I was spending :p
 
Last edited:
we have racks here that exceed well above 10kW, 2.5 isn't a lot

Well it is a lot when you consider other things in the house, that's what I meant. But 10kw. Wow. That would be like half my house's power capacity.

As for 70kw... comparing work stuff is kinda cheating. :p I don't know how much power each rack uses but all equipment total at my work is pulling around 90kw if I'm calculating that right. 1600 amps at 54vdc. More than half of that is probably the DMS100. At the main power switch we're pulling about 400 amps per phase last I checked. They're putting in 2 30ton AC units so that will probably change.

Come to think of it, how would 70kw even be physically installed in a single rack? That's a LOT of big cables. Just a bunch of individual 30a twistlock feeds straight from the inverters? Everything running at 240v I'm guessing?
 
Need to think bigger than that. PDU input is 415V 3 phase 48A (two per rack). The input cables are fairly thick and the quick disconnects are about the size of a bowling pin. There are no inverters as we don't float batteries between the equipment and the grid. We have kinetic energy storage that lasts about 60 seconds instead (generators kick in after 2 seconds). As for how, the aforementioned Superdome has 12 x 2400W power supplies per chassis (18U) for example. Fully loaded C7000 chassis (10U) use close to 10KW each and our racks accommodate 5 of them. With GPU blades, it's even easier. 8 sockets and 12 nVidia GRID cards per 4U.
 
2.5kw for a home setup is a lot of power utility wise. I am a cheap bastard when it comes to energy spending :) I would factor power usage into the equation to determine if I shouldn't just halve my disks and get disks double the capacity instead.
 
One of our products includes a processing rack with a Petabyte of storage... those obviously don't count though. :p
 
well this is not at my house but it is my personal equipment... it's at a colo facility... that rack is basically my media server :)
 
Oh right you had mentioned that in network pics thread, got it for cheap/free if I recall? Personally I like having stuff at home though. I wish I could even host my online stuff at home but ISP does not allow servers. Nice to have physical access.
 
SAN

-Node-A (Top): 2x Raid-6 - 11 disks each + 2 hotspairs (24x2TB disks)
Raw/Usable: 36TB/32.74TB

-Node-B (Bottom): 2x Raid-6 - 11 disks each + 2 hotspairs (24x2TB disks)
Raw/Usable: 36TB/32.74TB

SAN is setup in an active/active HA Cluster with 1:1 replication and automatic iSCSI failover. Total SAN Cluster Usable space: 32.74TB

Synology DS1511+
Raid-5 (5x3TB disks)
Raw/Usable: 12TB/10.91TB

Dell R620 (esxi 5.5)
Raid-5 7-disk +1 hotspair (8x 146GB 10K)
Raw: 1022GB/816.75GB

Total Usable Storage: ~44TB

Additional Hardware:
2x Supermicro (esxi 5.5) -1 one is unfortunately down with motherboard failure atm.
1x MacMini - (1 of 2 Domain Controllers, the other is virtual)
1x Cisco ASA
4x Netgear Switches
2x SmartUPS (2k Watt)

This is my lab and it is inside my house. It runs various services/hardware some to include:

- 2x Firewalls
- LoadBalancer/Reverse proxy
- Exchange Server
- VMware Horizon View (Virtual Desktops and applications)
- Remote Desktop Services Hosts (Application virtualization)
- File Server
- Plex Server
- System Center
- Private Cloud Storage / Mobile Device Management
- Active Directory
- Various other things depending on what I am working on....

PJbTFgb.jpg

jlBygKN.jpg
 
SAN

-Node-A (Top): 2x Raid-6 - 11 disks each + 2 hotspairs (24x2TB disks)
Raw/Usable: 36TB/32.74TB

-Node-B (Bottom): 2x Raid-6 - 11 disks each + 2 hotspairs (24x2TB disks)
Raw/Usable: 36TB/32.74TB

SAN is setup in an active/active HA Cluster with 1:1 replication and automatic iSCSI failover. Total SAN Cluster Usable space: 32.74TB

Synology DS1511+
Raid-5 (5x3TB disks)
Raw/Usable: 12TB/10.91TB

Dell R620 (esxi 5.5)
Raid-5 7-disk +1 hotspair (8x 146GB 10K)
Raw: 1022GB/816.75GB

Total Usable Storage: ~44TB

Additional Hardware:
2x Supermicro (esxi 5.5) -1 one is unfortunately down with motherboard failure atm.
1x MacMini - (1 of 2 Domain Controllers, the other is virtual)
1x Cisco ASA
4x Netgear Switches
2x SmartUPS (2k Watt)

This is my lab and it is inside my house. It runs various services/hardware some to include:

- 2x Firewalls
- LoadBalancer/Reverse proxy
- Exchange Server
- VMware Horizon View (Virtual Desktops and applications)
- Remote Desktop Services Hosts (Application virtualization)
- File Server
- Plex Server
- System Center
- Private Cloud Storage / Mobile Device Management
- Active Directory
- Various other things depending on what I am working on....

http://i.imgur.com/PJbTFgb.jpg
http://i.imgur.com/jlBygKN.jpg

Nice, clean cable management. I like it. I usually don't have the patience to do that part of the job well. I try, and get it to be decent and functional, but never good and clean.

I usually go with many different colored Ethernet cables though, and then color code things, to make everything easier to follow.
 
Oh right you had mentioned that in network pics thread, got it for cheap/free if I recall? Personally I like having stuff at home though. I wish I could even host my online stuff at home but ISP does not allow servers. Nice to have physical access.

Do they actually enforce it?

My experience has been that unless you flaunt it, or use excessive bandwidth, they turn a blind eye to some limited home serving.
 
Zarathustra[H];1041220577 said:
Nice, clean cable management. I like it. I usually don't have the patience to do that part of the job well. I try, and get it to be decent and functional, but never good and clean.

I usually go with many different colored Ethernet cables though, and then color code things, to make everything easier to follow.

Thank you. The cable management is kind of a habit (although having the right kind of rack and the hardware for it can help immensely).

My previous job I would travel around the globe doing server farm refreshes for our company. I mostly specialized in setting up the SANs, virtualization, and disaster recover hardware/software. Although the first few days were always rack-n-stack and cable management.
 
damn dude you have a datacenter in your house, i am in awe. nice and clean too.
 
SAN

-Node-A (Top): 2x Raid-6 - 11 disks each + 2 hotspairs (24x2TB disks)
Raw/Usable: 36TB/32.74TB

-Node-B (Bottom): 2x Raid-6 - 11 disks each + 2 hotspairs (24x2TB disks)
Raw/Usable: 36TB/32.74TB

SAN is setup in an active/active HA Cluster with 1:1 replication and automatic iSCSI failover. Total SAN Cluster Usable space: 32.74TB

Synology DS1511+
Raid-5 (5x3TB disks)
Raw/Usable: 12TB/10.91TB

Dell R620 (esxi 5.5)
Raid-5 7-disk +1 hotspair (8x 146GB 10K)
Raw: 1022GB/816.75GB

Total Usable Storage: ~44TB

Additional Hardware:
2x Supermicro (esxi 5.5) -1 one is unfortunately down with motherboard failure atm.
1x MacMini - (1 of 2 Domain Controllers, the other is virtual)
1x Cisco ASA
4x Netgear Switches
2x SmartUPS (2k Watt)

This is my lab and it is inside my house. It runs various services/hardware some to include:

- 2x Firewalls
- LoadBalancer/Reverse proxy
- Exchange Server
- VMware Horizon View (Virtual Desktops and applications)
- Remote Desktop Services Hosts (Application virtualization)
- File Server
- Plex Server
- System Center
- Private Cloud Storage / Mobile Device Management
- Active Directory
- Various other things depending on what I am working on....

http://i.imgur.com/PJbTFgb.jpg
http://i.imgur.com/jlBygKN.jpg

Love the cabinet. Is that a custom job or bought off the shelf?
 
Love the cabinet. Is that a custom job or bought off the shelf?

Was interested in that myself.

I currently have a bunch of rack-mountable stuff, but no rack.

Need to start learning about rack mounts so I can select something used on the cheap that doesn't suck.
 
Where is this 42U bay from ?

Can you be pleased to post a link of the specs ? :)
 
It's an APC rack. They're not cheap. List on that 38U one is nearly $7000, so don't get your hopes up.
 
Love the cabinet. Is that a custom job or bought off the shelf?

Rack: APC Netshelter CX 38U

http://youtu.be/G2-geG9XRFA (overview of rack)
http://youtu.be/NfTP3yhTybc?t=2m55s (sound damping demonstration)

I only wish I could build something at that level. Maybe someone much more talented than me could build such a monster, but they would have to be extremely careful with designing adequate airflow for a closed/sound dampened system.

The rack is the newest edition, and I am in love with it. I very recently just relocated from living in Japan for the last 10 years to southern California. I could never get the type of rack I wanted in Japan due to the sheer size and weight of shipping it (and expense).

Not exactly a cheap unfortunately, but its damn near sound proof which was a requirement of mine after living with a datacenter a couple feet from my head for the past couple years (sound was starting to drive me crazy).
 
Sorry images are old and kind of crappy, but this shows some of the evolution of my in home datacenter.


The first ghetto rack....
UDDjUaM.jpg



The second not so ghetto rack. Sides and doors became a new requirement because of the damn cats :rolleyes:
U0PNSpu.jpg
 
Rack: APC Netshelter CX 38U

http://youtu.be/G2-geG9XRFA (overview of rack)
http://youtu.be/NfTP3yhTybc?t=2m55s (sound damping demonstration)

I only wish I could build something at that level. Maybe someone much more talented than me could build such a monster, but they would have to be extremely careful with designing adequate airflow for a closed/sound dampened system.

The rack is the newest edition, and I am in love with it. I very recently just relocated from living in Japan for the last 10 years to southern California. I could never get the type of rack I wanted in Japan due to the sheer size and weight of shipping it (and expense).

Not exactly a cheap unfortunately, but its damn near sound proof which was a requirement of mine after living with a datacenter a couple feet from my head for the past couple years (sound was starting to drive me crazy).

I built a small one for my purpose from broken telcom rack in 2012 and still running as today.
Rack controller(Arduino) with Automatic exhaust fan speeds and warnings.
I plan to make "nice" exterior , but not hurry aka lazy since it achieve my goal for "low" noise closed force air rack case .

KwAZO0n.jpg

Angled Front view

JPBHRDx.jpg

side view with front cover

sQDmzBU.jpg

front cover top view showing intake air (without filter)

pPes6IE.jpg

angled back view showung 4 exhaust fans

T9JuAV8l.png

Graphical rack controller with GUI web-based. Powered by Arduino Mega 2560+sensors+I/O+ Ethernet Shield.

QrM6C1n.png

Text web-based.
 
I built a small one for my purpose from broken telcom rack in 2012 and still running as today.
Rack controller(Arduino) with Automatic exhaust fan speeds and warnings.
I plan to make "nice" exterior , but not hurry aka lazy since it achieve my goal for "low" noise closed force air rack case .

Very nice. You could definitely build something something exactly like this pre-built APC rack if you have the skills and tools to do so. No idea how much it would cost you, but it is essentially a rack frame attached to painted plywood box (maybe an oversimplification but still...).

It would certainly be cool to try and build something like it. Although for me living in a tiny place in Japan for the last decade I've never had a place to store big powertools. Now that I've moved stateside and live in a ridiculously big house (or atleast big compared to the last place), maybe one of these days....
 
Zarathustra[H];1041220585 said:
Do they actually enforce it?

My experience has been that unless you flaunt it, or use excessive bandwidth, they turn a blind eye to some limited home serving.

Hard to tell but it would not be a risk I'd be willing to take. It takes a whole day if more to migrate stuff to a new server so it would mean down time if I was told on short notice to take it offline. Have to reconfigure everything etc. They also don't offer static IPs. I don't get these policies, they would actually make more money if they offered static IPs, and multiple IPs that you pay extra for. Heck I'd pay 100 bucks more just to be allowed to run a server if they had the option but they don't.

Mind you I recently moved my web stuff to OVH/Soyoustart for about 49/mo so that's not too bad of a price to pay and it's a half decent server. 32GB of ram, 2TB drive and quad core xeon. Not as good as something I could build at home but it's more than what I need for my own sites/apps.
 
Very nice. You could definitely build something something exactly like this pre-built APC rack if you have the skills and tools to do so. No idea how much it would cost you, but it is essentially a rack frame attached to painted plywood box (maybe an oversimplification but still...).

It would certainly be cool to try and build something like it. Although for me living in a tiny place in Japan for the last decade I've never had a place to store big powertools. Now that I've moved stateside and live in a ridiculously big house (or atleast big compared to the last place), maybe one of these days....

you can build with a fraction of $$$$:D.
exterior finishes are $$$ compare with others material. You know, why I am not done with exterior finishes since finished building in 2012 :p.
complete old telcom rack is a good candidate since side panels, and front panel are solid
I found incomplete(missing side, front and back panels) old telcom rack near dumpster when lived at apartment before moved to my house,
is not just rack and slap with plywood. you would be surprise when building it. rule of thumbs: how to move air from intake through exhaust fans.. rack case controller is the key :D.
APC netshelter has automated fan controller based on temperature, and you can add more features too, but much $$$.
Actually, I built based on APC netshelter, I have all measurements and parts# and assembly in PDF. why reinvent the wheel...

painted plywood box is not "nice", "faked" vinyl finish roll is the best: measure, cut, glue, and stick it! :D.

Arduino is easy for making automated controller with a fraction of $$$$ since many arduino clones and shields on ebay cheaply from China. 10% of total was broken:p. I did not bother to ask refund since not much money involved and a bit hazzle dealing with ebay.
 
Last edited:
you can build with a fraction of $$$$:D.
exterior finishes are $$$ compare with others material. You know, why I am not done with exterior finishes since finished building in 2012 :p.
complete old telcom rack is a good candidate since side panels, and front panel are solid
I found incomplete(missing side, front and back panels) old telcom rack near dumpster when lived at apartment before moved to my house,
is not just rack and slap with plywood. you would be surprise when building it. rule of thumbs: how to move air from intake through exhaust fans.. rack case controller is the key :D.
APC netshelter has automated fan controller based on temperature, and you can add more features too, but much $$$.
Actually, I built based on APC netshelter, I have all measurements and parts# and assembly in PDF. why reinvent the wheel...

painted plywood box is not "nice", "faked" vinyl finish roll is the best: measure, cut, glue, and stick it! :D.

Arduino is easy for making automated controller with a fraction of $$$$ since many arduino clones and shields on ebay cheaply from China. 10% of total was broken:p. I did not bother to ask refund since not much money involved and a bit hazzle dealing with ebay.


I have no doubt that someone with the skills could pull it off for considerably cheaper than buying from the manufacturer.

For me personally, I don't have those skills (or tools) yet to even make the attempt. I put money away for the last year knowing I was going to buy a nice sound dampened enclosure when we moved stateside. When it came time to make the purchase, it was either buckle down and spend the money that I already saved up for it, or buy another standard perforated rack that would drive me nuts with the sound while I tried to come up with some other kind of solution (I didn't even ship my previous rack back to the states, so I needed something).

In the end, my aural sanity demanded I just buy what I really wanted.
 
Zarathustra[H];1041220585 said:
Do they actually enforce it?

My experience has been that unless you flaunt it, or use excessive bandwidth, they turn a blind eye to some limited home serving.

I had that problem with BellSouth ADSL back in the day, some asshat bellsouth employee on DSLReports got my account suspended because I had my personal website hosted on my dsl line... last time I ever used bellsouth or now att
 
I have no doubt that someone with the skills could pull it off for considerably cheaper than buying from the manufacturer.

For me personally, I don't have those skills (or tools) yet to even make the attempt. I put money away for the last year knowing I was going to buy a nice sound dampened enclosure when we moved stateside. When it came time to make the purchase, it was either buckle down and spend the money that I already saved up for it, or buy another standard perforated rack that would drive me nuts with the sound while I tried to come up with some other kind of solution (I didn't even ship my previous rack back to the states, so I needed something).

In the end, my aural sanity demanded I just buy what I really wanted.

I admire you APC netshelter :D, so beautiful!

I wished had $$$ money to spend on it, but put $$$ money on my first new house at that time was a priority;).

exactly.
Once we got spoiled with "low" noise closed force air rack , never ever go back to non closed rack :D.
 
Status
Not open for further replies.
Back
Top