67 TB raw storage for 7.8k

T

Thuleman

Supreme [H]ardness
Joined
Apr 13, 2004
Messages
5,833
So I am prowling the web, still being hurt by the 40k Dell wants for 16 1TB drives, as I come across these guys: Petabytes on a budget: How to build cheap cloud storage

Basically they took commodity hardware and build a 4U server that holds 45 drives at a cost of less than 8k. Yes, it doesn't do most of what enterprise arrays do, BUT it doesn't have to if all you need is some storage. It's still limited to a "single controller" if you want to call the motherboard that but if you consider that you can buy a second 67 TB storage array for the price of a 2nd controller for an enterprise array the Backblaze starts to look pretty sweet.

If you don't need 67TB raw you can buy fewer drives and further decrease the cost of the setup.

Then there's Coraid's EtherDrive which is also an open hardware platform and actually does what enterprise arrays do at a fraction of the cost. As sweet as that VNXe looked, considering the options I may just buy a couple EtherDrives instead.
 
Their implementation is a total POS. Plenty of threads on this forum that explain why. You can buy a proper setup with actual server hardware that won't be horrendously slow for less. You can do 108TB in 4U for the same price with Supermicro hardware (so, 16 cores, 32gb of RAM, redundant power supplies, IPMI, etc) too.
 
Has anyone had experience with the Etherdrive device or devices with ATAoE technology?
 
Blue Fox said:
Their implementation is a total POS. Plenty of threads on this forum that explain why. You can buy a proper setup with actual server hardware that won't be horrendously slow for less. You can do 108TB in 4U for the same price with Supermicro hardware (so, 16 cores, 32gb of RAM, redundant power supplies, IPMI, etc) too.
Click to expand...

Well so I got a bit overenthusiastic about it. The main problem I see with the Blazeback is that there are no redundant power supplies. However, I don't see the use of 16 cores and 32GB of RAM for a box who's only purpose it is to provide storage (either slow stor for backups, or NFS/iSCSI targets (which Backblaze can't do)).
 
That's pretty cool, but must be a pita to swap out drives. I guess you technically can slowly slide out the server, open the case up and do it, but I'd be scared to bump something or have the power cord pull out etc.

But it is an efficient use of space. The traditional SAN enclosure has a lot of wasted space. Guess you can technically treat it as a "storage device" and not a SAN. Have at least two of them in a redundant matter, and just take one offline if it needs maintenance.
 
I'm sure the backblaze pod works for them and they do seem to cram about as many drives into 4U as is humanly possible but it's rather a hack and something I would only recommend if you have replication at a higher level such that you can deal with the death of an entire pod with minimal disruption.

I'd agree that the supermicro route seems a better bet for most uses. They have a case that can put 36 (a bit less than the 45 of the backblaze but not hugely so and hard drives have got bigger since backblaze designed their pod) hotswap bays with easy access (assuming you have good access to the rear of your rack and neat cable management) plus space for a motherboard in 4U. Further it can be purchased with SAS expanders so you can hook all those drives up to one SAS controller.

A supermicro 847E1-R1400LPB (4U, 36 drives, expander backplanes, redundant PSUs) seems to cost arround $1500 which seems comparable to the price of the backblaze case/PSUs/backplanes.

However, I don't see the use of 16 cores and 32GB of RAM for a box who's only purpose it is to provide storage
Click to expand...
So fit a lower spec board something like the x8SI6-F.
 
Last edited:
Blue Fox said:
As already mentioned, SAS expanders make that all possible.
Click to expand...

So yeah, I was confused, looked this up and it's a lot more clear now. Priced out a 48TB raw system for 9.5k plus shipping, doubt you could get it much lower. I don't see how you could end up with 108TB for less than the 7.8k on the BackBlaze.

Either way though the BackBlaze works well for them because they have dozens if not hundreds of them and they are running some proprietary software that creates redundancy among their units which is why non-redundant power supplies are not an issue to them.

jenuster said:
Has anyone had experience with the Etherdrive device or devices with ATAoE technology?
Click to expand...

The biggest issue I see is that you need at least two AoE HBAs for your host(s). That adds inconvenience at the very least, some hosts may not even have space to fit them. A 16TB raw Coraid unit came back priced at 14k through their quick quote process though that's guaranteed to have SATA drives (the quick quote doesn't say) and you still need the HBAs. Sure, it's lower than a 16TB from the enterprise vendors, and it's open so you can replace the drives yourself and that's really the main competitive advantage of Coraid.

However, what is cool, if I understand it correctly, is that you don't need a vSphere Enterprise license to achieve multipathing. That's very attractive from where I am looking. Multipathing is done at the HBA level and vSphere is basically not aware that it is happening. See: http://www.youtube.com/watch?v=Jaqsrm_eOTE
 
Last edited:
Simple...
3TB drives go on sale for $130 a pop now: 36 x $130 = $4680
Supermicro 847E1-R1400LPB chassis: $1553
Supermicro H8DG6-F motherboard: $580
Opteron 6128: 2 x $275 = $550
4gb sticks of 1333mhz DDR3: 8 x $40 = $320

Total: $7683. Should be less than $7800 shipped. See how easy that was? That's with 16 cores, 32gb of RAM, and server board too. If you go to something lower end (single socket), you can bring it down below $7000. With 2TB drives (72TB), you can knock off another ~$2000, so that brings it below $5000. Can probably make it all happen for ~$4500 if you wanted to.
 
Wouldn't you need a AOC-USASLP-H8iR or similar to go along with it, plus a battery, plus cable, that's another $500. I didn't price mine with consumer grade drives but used the WD2003FYYS instead, but I see your point.

I'll give the Coraid guys a call tomorrow and see what's what though. The Supermicro will make a good cheapo storage solution but the Coraid stuff could actually work as primary storage.
 
If you want hardware RAID, yeah, something like that would be necessary. If you're looking at buying/building a system, send me a PM with your requirements/needs and I can give you a quote myself. I've done quite a number large storage systems for a few companies and individuals (even on this very forum) and have references I can provide if that's something you're after. Coraid actually uses Supermicro hardware by the way (or at least every single Coraid system I've seen in person was).
 
Blue Fox said:
Simple...
3TB drives go on sale for $130 a pop now: 36 x $130 = $4680
Supermicro 847E1-R1400LPB chassis: $1553
Supermicro H8DG6-F motherboard: $580
Opteron 6128: 2 x $275 = $550
4gb sticks of 1333mhz DDR3: 8 x $40 = $320

Total: $7683. Should be less than $7800 shipped. See how easy that was? That's with 16 cores, 32gb of RAM, and server board too. If you go to something lower end (single socket), you can bring it down below $7000. With 2TB drives (72TB), you can knock off another ~$2000, so that brings it below $5000. Can probably make it all happen for ~$4500 if you wanted to.
Click to expand...

Apples and Oranges, my friend. The Backblaze design was priced out at $7.2k based on prices in mid-2009 when the article was written. Your using current pricing of higher density disks, etc, and to top it off you started with "wait 'till it goes on sale"...and for all that you didn't even beat their cost, you just came close - but slightly higher.

Price out the Backblaze design now and you come close to $4.5k. Or use the same 3TB drives you did and you come to 135TB, and at current pricing they are still paying far below what you just spec'd.

Your "I can do it cheaper" claim still rings hollow.
 
PigLover said:
Apples and Oranges, my friend. The Backblaze design was priced out at $7.2k based on prices in mid-2009 when the article was written. Your using current pricing of higher density disks, etc, and to top it off you started with "wait 'till it goes on sale"...and for all that you didn't even beat their cost, you just came close - but slightly higher.

Price out the Backblaze design now and you come close to $4.5k. Or use the same 3TB drives you did and you come to 135TB, and at current pricing they are still paying far below what you just spec'd.

Your "I can do it cheaper" claim still rings hollow.
Click to expand...
I used the price example because Fry's and newegg have 3TB drives for that price right now. My point was that you can use proper hardware for the same cost. I even stated that if you go to single socket (only 8 cores then) and used 2TB drives, you'd be at $4500, so the same cost with far better hardware.
 
I should have taken more care in looking at the dates before making a "sensational" topic like this.

I think both solutions have merit depending on what ones needs are. When only considering raw storage then it is difficult (if not impossible) to beat the Backblaze design in a price per TB consideration. However, it is also clear that the Backblaze was designed to run in a setup with many other pods like it, which may not work for most other users.

The Supermicro solution provides the advantage of hardware redundancy for certain components at slightly higher price per TB.

The Coraid wraps Supermicro hardware into a package that can compete with the enterprise storage vendors in some areas but still doesn't provide full hardware redundancy and has some issues with scaling but makes up for that with the ability for users to use commodity hardware to upgrade storage as well as pricing in general.
 
I had started typing this and apparently got sidetracked and never posted it when after finishing writing it up. I see Thuleman beat me to the punch with the comment about CORAID<->Supermicro connection.

I have some exposure to the CORAID SR series arrays. Essentially CORAID is reselling Supermicro hardware. In the case of the the SR2421 it is using a Supermicro SC846 chassis. Inside the SR2421 that I was digging around it, I found an X7 series supermicro motherboard with a single 5000 series Xeon processor, 2GB of RAM and three Supermicro AOC-SAT2-MV8 PCI-X Sata cards hooked up to the 24 ports of the backplane. The motherboard had a white PATA connector that was designed for use with compact PATA flash fobs. They were running their OS from that. From the console it appeared they were using Plan 9. Given that there were essentially three 8 port non-RAID SATA cards, the RAID'ing provided by the array has to be running in software. Essentially it was a direct attached storage appliance using off the shelf components backed by an embedded os running the equivalent of linux's mdraid and exporting the raided volume as AOE targets.

Overall I don't have a very high opinion of the arrays. The supermicro hardware itself is great but the performance leaves much to be desired. Rebuild times of a degraded LUNs were miserable and performance suffers terribly during the rebuild period. Many RAID schemes take a hit while degraded (RAID 5 I'm looking at you) but this was well beyond what I'd experienced with using a RAID HBA like a PERC or SmartArray controller or even just MDRAID for that matter. When breaking down the numbers you could build out a supermicro with better specs(ie include a 24 port RAID HBA) for the less than or equal to the price as a CORAID array. The hardware itself should be more than up to the task of either block level exports (ie iSCSI and/or AOE) or file shares (ie SMB or NFS) but I feel falls short the way CORAID has things put together.
 
Actually, I mentioned Coraid using Supermicro hardware first. :p I've not dealt with their software, but I do agree that the hardware they were using was a bit lacking. The units I've encountered also had 2 extra dual-port NICs in them on top of what you described. I would be really surprised if they were using Plan 9 though. That hasn't been updated in nearly 10 years and I have yet to see it in production.
 
oh right silly me you were the second post. I'll see if I can get a screen grab of the that show the Plan 9 copyright info. I don't recall if it was when the unit is booting or just you can get it to display on the console.
 
lopoetve said:
Shite.
Click to expand...

There's got to be a bit more info than that. ;)

At a first look it actually looks quite enticing for some applications. No mess/fuss to set up, just plug in and good to go. No "enterprise" switch requirement to achieve LACP (802.3ad) like link aggregation, in fact it is my understanding that you can easily aggregate beyond the theoretical throughput of your disk array to eliminate the network bottleneck without having to pay for 10GbE hardware.

Another thing that looks interesting is that in order to do any kind of MPIO on iSCSI, NFS, FC, you need a vSphere Enterprise and above license. Since AoE does aggregation and multipathing on the transfer protocol level it is removed from the vSphere licensing requirement and will do MPIO all by itself regardless of whether or not you have an Enterprise license.

The Coraid product may not be the best way to go due to software raid etc, but is AoE really that bad? Yes, it can't be routed, but if you don't need it to be routed then it doesn't matter whether it can be.
 
Blue Fox said:
Actually, I mentioned Coraid using Supermicro hardware first. :p I've not dealt with their software, but I do agree that the hardware they were using was a bit lacking. The units I've encountered also had 2 extra dual-port NICs in them on top of what you described. I would be really surprised if they were using Plan 9 though. That hasn't been updated in nearly 10 years and I have yet to see it in production.
Click to expand...

Here's confirmation that they are using Plan9: https://groups.google.com/group/comp.os.plan9/browse_thread/thread/6f8092815adfcc98/6f975bc968dc37c9

I sat through a 90 minute technology presentation and Q&A for Coraid and I do think that if you need high throughput and you need massive storage on a petabyte level then their solution is decent though paying essentially 10k for the operating system per cage is a bit of a bitter pill. Still, performance wise and price wise they come out way ahead of traditional storage vendors at that scale.

Essentially what they are doing is a step above BackBlaze when it comes to hardware and then they sell the software they use to tie the different cages together whereas BackBlaze keeps that software to themselves.

I would guess that they use P9 instead of Linux due to some Linux or Linux component license not being favorable to what Coraid is doing.
 
Interesting choice on their part. I guess it's still fairly cost effective when compared to a normal SAN/DAS unit from one of the big OEMs.
 
You must log in or register to reply here.
Back
Top