Build a 100 TB Enterprice Class SAN for $50k?

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I am trying to see what options are out there to have near enterprise class storage at a fraction of the cost. From my experience it seems that Tier 1 and Tier 2 storage seems to range from $2,000 to $5,000 per TB and other much more expensive solutions. So I was looking for something closer to $500 per TB. This would need to include software and hardware. It doesn't have to be exactly 100 TB or $500 per TB, but something close and the software should be feature rich possibly like SANsymphony.

I don't want any single points of failure, so it would most likely need to be a cluster of nodes like VSAN that can tolerate multiple node failures. Primary use would most likely be VMware. I was thinking of a multi-tiered hybrid solution with SATA and SSDs with 10 Gb SFP+ ports.

Do you think it is possible or would it be more like $1,000 per TB?
 
archive.org is simple, much like backblaze.

Just a large collection of disk on cheap servers.

Atleast with archive.org all your serving is static content, simple enough to do, you just need to hold and know where that data is.

backblaze is the same idea, just larger files.
 
So does archive.org have a JBOD setup without any backups or RAID0/1/5/6/10/50/60 in place?
 
Some thoughts if you do not want to use NetApp & Co.

A solution must depend on the amount and quality of internal IT personel
(as you do not have a SLA with a few hours of reaction time.)

Avoid complex solutions
Spread the load over several smaller machines. On problems only a part of your installation is affected.

On problems define a timeframe for service restart on any sort of failures.
The answer can be an active/active SAN cluster, async replication or HA options from ESXi.
In any case, you must think about backups (for data and hot spare server)

Regarding Software:

High quality and affordable storage = ZFS storage
I prefer Solaris based ZFS systems (Free OmniOS or Oracle Solaris, both offer commercial support)
Another options are BSD based ZFS solutions. Prefer appliance solutions with a Web-GUI

Think about async replication between appliances (delay down to a minute).
Active/active HA solutions, see http://www.high-availability.com/zfs-ha-plugin/ (more complex)

Regarding hardware,
look at storage servers from SuperMicro

If you can split the load, use some 2,5"-24 bay storare solutions with SSD only storage (up to about 20 TB per unit and 1000$ per TB)
for high performance use and/or 3,5"-24 bay cases with up to 100 TB (with 6 TB disks, less than 500$ per TB) for normal or backup use.
With such cases and 3 HBA controller per unit, you can avoid expander solutions and use Sata disks/SSDs

Use as much RAM (up to 128GB per Node) as possible as this is used as fast readcache and gives performance.
Include fast ZIL devices like a ZeusRAM or Indel S3700 for secure sync write with ESXi. Prefer NFS over iSCSI.

Add a (redundant) 10 GbE network. For ultra low latency connections between storage and ESXi you can also virtualize a SAN OS on every EXSi server (local shared NFS storage). Each ESXi server acts then completely autonomous but with acces to any other storage via NFS and all SAN features like data security with copyonwrite and checksums, snapshots, caching, fast transfers etc active.
 
For the chassis: http://www.45drives.com/products/direct-wired-redundant.php

Backblaze has a blog that goes through how they handle their storage, and they use this exact chassis. We picked up one ourselves and have 127TB on a RAID60 in three spans. Granted this is scale out storage, and not meant for super high performance. But with proper motherboard, memory, and processing you could get these to run pretty fast (we don't need performance so we just run a simple mdraid on basic hardware). I would not expect this to beat out any good SAN though.
 
You are using this for VMWare to hold your VM's? You want fast disk 10k RPMs here. Infortrend, Hitachi, and Nexsan offer large storage volumes for dirt cheap that are PERFECT for VMWare. Some even offer Tiering which is something you are looking for as well.

I am going with Infortrend 16 900GB SAS drives to hold my VM's and then the additional storage controller/SAS connector with big fat slow Near Line-SAS disks for backups.

Don't do backblaze pods for VMWare VM's. That's like buying a Ferarri and putting a moped engine in it. Backblaze is for Data Retention not constantly accessed files.


I did a huge storage pool with SuperMicro 36 bay servers I got from serversdirect and 3TB 7200 RPMS hitachi's running openstack: Swift.

We had 1.2PB RAW and 400TB Useable replacing EMC, Equallogic, Amazon's EBS storage for our Hadoop infrastructure and storage needs.

Cost us close to $350k said and done including the 10GB fibre with a Cisco Nexus 7009. Servers and storage were about $150k or so.

Highly recommend looking at the storage software you plan on using before you start looking at hardware.
 
This is more hypothetical at the moment. I would only propose such a solution if I can come up with a solid design and cost first. So this is more of the recon stage. I think SAS would be way too expensive. I am more looking towards a hyper-converged type of solution just without the commute attached. VSAN is still an option, but I need to review the price and would prefer the storage be separate so it can be used for more than just VMware.

Part of the idea came from this article. http://thehomeserverblog.com/esxi-storage/building-a-homemade-san-on-the-cheap-32tb-for-1500/

I had read somewhere that Microsoft uses this of course, in a cluster setup. I haven't dived really deep into the configuration, but I would think that their cluster setup would allow your data to reside on multiple physical servers so you could sustain a node failure without data loss.
 
That home server setup is good enough for a home server or a SMB the hardware alone is $1500 but he's using licensed software that costs ????.


Answer me this before I go any further:

What do you plan on doing with your Virtual Machines? What are they? Web Servers? Database Servers? Exchange? File Servers? What kind of I/O do you want and expect?


And SAS is not expensive if you go with 10k vs 15k drives. You'll find it's not that unreasonable.
 
That home server setup is good enough for a home server or a SMB the hardware alone is $1500 but he's using licensed software that costs ????.


Answer me this before I go any further:

What do you plan on doing with your Virtual Machines? What are they? Web Servers? Database Servers? Exchange? File Servers? What kind of I/O do you want and expect?


And SAS is not expensive if you go with 10k vs 15k drives. You'll find it's not that unreasonable.

The VMs vary quite a bit, but I would say the majority is Web Servers. Two customers alone have about 160 VMs for their websites. I would say at least 50 of them are running MySQL. Another big use would be a RDS Farm.

There is no real number of IOPS I am looking for. Until recently our NetApps were most likely less that 10,000 IOPS. We went with a hybrid shelf that is 10k SAS with SSD Flash Pool, but I don't know how you really determine the number of IOPS from a hybrid shelf that uses SSD for read/write caching.

I would say the reason for SATA comes from the fact that vendors like Nimble and Nutanix are using SATA with SSD acceleration. Even VSAN is typically SATA + SSD. Unless that type of setup doesn't work well with ZFS.
 
Nutanix is a rebranded C6200/Supermicro chassis with special sause to make it all work. I've spoken with these guys and know their hardware quite well. Ridiculously overpriced for what it does.

Nexsan E series would do what you want and does offer Tiering/Storage pools based on the type of applications you are describing:

https://www.imation.com/en/nexsan/e-series/

They are inexpensive, offer killer support, and great feature set. I was about to purchase one of these for my prior company until I left. Fully 10GB with dual controllers and SAS shelf support.

Honestly I've never seen the need for SSD's in a production farm unless you are running some massively high requiring IOPS. 10k RPM disks will run everything just fine and that's coming from my experience with high volume websites and huge SQL databases.

I would never put that volume of Virtual machines in a SAN environment on SATA disks especially chatty/busy MySQL servers and you want to move into clustering like Amazon's RDS? Yikes! SAS 100%
 
The RDS I was referring to is Microsoft's Remote Desktop Services. :p

I am also familiar with the Nutanix hardware as we build our own Supermicro servers and almost went with the same exact chassis they are using and their price is ridiculous especially since their secret sauce is the software and not the hardware.

Prior to getting a hybrid disk shelf we were running the RDS (Remote Desktop Services) on a disk shelf with 24x 15k SAS drives. Their performance seemed to increased when we were testing VMware's Flash Read Cache, but had to disable it due to issues with Veeam. We then went with the hybrid shelf with 10k SAS and SSDs and performance has been decent.

Although RDS is not VDI, it is similar and from what I have seen there are many people that use all flash shelves for VDI and these tiered solutions of SATA and SSD seem to be popular as well. Although I am sure linked clones and other technologies like what Nutanix uses makes a big difference and the VDI probably runs nearly 100% off of SSD. With our RDS setup the customer is using Office 365 and therefore they have over 1.5 TB of .OST files.

I guess what I am most curious about is Nimble and their CASL file system. They use only SATA and SSD. Although it seems they try to primarily use NVRAM for writes and SSD for reads.
 
I guess what I am most curious about is Nimble and their CASL file system. They use only SATA and SSD. Although it seems they try to primarily use NVRAM for writes and SSD for reads.



I will put money all they are doing is Tiering and doing it in such a way that they can afford to use cheap SATA for non-changing data and put all the volatile ever changing material on the SSD's.

It's not rocket surgery but I haven't found any open source software that offers Auto-Tiering either. Doesn't mean it's not out there.
 
For the chassis: http://www.45drives.com/products/direct-wired-redundant.php

Backblaze has a blog that goes through how they handle their storage, and they use this exact chassis.
As someone who has to administer these systems, PLEASE don't ever use Backblaze or similar systems. They work okay until a drive fails. Then you're stuck reading off the serial number on every drive in the chassis looking for the failed disk - if you were smart and get a disk manufacturer who put the serial # on the end of the disk. If not, you power the system down, and start taking disks out one at a time until you find the failed disk. Heaven help you if you didn't leave enough room above the pod for a human head to fit... There's no rails or cable management arm to pull the chassis out. Oh, and you have to be pretty handy with a screwdriver, because the only way to access the disks is through the top panel, which has about 10 screws - Including four on the side of the chassis.

Get a real enterprise grade solution with a pretty red light that tells your admins "Replace me, stupid! I'm the broken one!"

Manual tiering can be done if you build a "high speed" and "low speed" system, and allocate vmdks based on content.

I'd guess you could assemble a quality enterprise-grade nearline NAS for $20K - 4U/100TB raw.
 
Last edited:
I will put money all they are doing is Tiering and doing it in such a way that they can afford to use cheap SATA for non-changing data and put all the volatile ever changing material on the SSD's.

It's not rocket surgery but I haven't found any open source software that offers Auto-Tiering either. Doesn't mean it's not out there.

Well, I am not looking for free open source software. SANsymphony supports auto-tiering, but I don't know how much it costs. The objective was to build a SAN with SATA and SSD, and also use a decent OS that is not ridiculously overpriced.
 
I would not use data tiering but SSD only storage when you really need performance as good 1 TB SSDs are now in the 500 $/Eur region For the rest use regular 4-6 TB spindles paired with RAM readcaching and/or SSD caching (Regarding performance RAM readcaching is 100 x SSD readcaching so prefer as much RAM as possible).

Only aspect is that I would prefer SAS disks when using expander solutions.
And again, look at ZFS for quality storage (ex OmniOS as a specialized highend storage OS)

see http://www.snia.org/sites/default/f.../monday/JeffBonwickzfs-Basic_and_Advanced.pdf

a pdf from the development of ZFS but shows the principles.
 
I use infortrend and have been very happy with their stuff... I have well over 100TB in my infortrend arrays and I spent a heck of a lot less than $100k, but it's not the latest and greatest, no idea what their arrays cost new, but I am sure it's less than the raping someone like netapp will give you
 
I use infortrend and have been very happy with their stuff... I have well over 100TB in my infortrend arrays and I spent a heck of a lot less than $100k, but it's not the latest and greatest, no idea what their arrays cost new, but I am sure it's less than the raping someone like netapp will give you

I hope you guys suggesting/using Infortrend are talking about their basic EonStor stuff - it's totally fine.

Run away as fast as you can from their ESVA equipment and/or anything you expect good support on from them.

We bought an "HA" ESVA (not cheap, btw) and it was the worst POS unit and POS company I've ever worked with (worse than the telcos, if that tells you anything).

They had zero native English speakers available for support. They used on-call translators who didn't understand the conversation, and that was after we got a hold of them and waited for "Taiwan Business Hours" 6PM+ PST - the only time they would give us more than 'did you power cycle your HA SAN (twice?)' support. Right after we bought the unit they fired their entire US staff that wasn't from Taiwan - we thought they were going out of business.

After several multi-hour downtimes in the first month, months of begging (while using alternate equipment for stability), and finally a threat of a lawsuit they took it back.

Our home-brew ZFS SAN's are infinitely better and more reliable. As is the Nexenta that replaced the ESVA.
 
Well, I am not looking for free open source software. SANsymphony supports auto-tiering, but I don't know how much it costs. The objective was to build a SAN with SATA and SSD, and also use a decent OS that is not ridiculously overpriced.

SANSymphony is so expensive that I don't think you could even come close to license 100TB of HA storage for $50k, and I'm only talking about the software licenses and not the hardware itself. You should just contact them and get a quote though to atleast hear what they say.

It's been a while since I have looked, but something like 4TB of mirrored storage (8TB total) is like $15k. That's just software, I think they may be even more expensive than the traditional SAN companies like NetAPP, HP, EqualLogic, etc.

The absolute cheapest commercial SAN Software with support that provides 2 Node HA that I am aware of is from: Open-E. It is certainly not the most elegant solution, and you are limited to 2-node clusters, but if you follow the HCL, and don't skimp on the hardware, you can build a fast HA SAN. You won't get alot of the nice features of more modern (and expensive) products though.

Most of the software only solutions you may be interested in START at over $10k for the lowest capacity tiers that include mirroring (assuming you are keen on doing the hardware side yourself to save money). I'm also avoiding mentioning any free solutions since I'm assuming this is for a company and you actually want a support contract.

For hardware/software solution you could look into the cost of a Nimble Storage SAN, I honestly don't know what they cost but I have heard positive things about them (assuming they could be more competitively priced). We mostly have EqualLogics, and I have used NetApp in the past. Obviously both out of the price range you provided for that amount of storage and features.


The things you are asking for are not cheap as you know. You say you want redundancy, but where? Everywhere? Networking, Storage, Controllers, etc? For example, I'm personally not aware of any "SAN Software" you can purchase that would allow you to run redundant controllers (if you are going RAID instead of ZFS). With things like storage, and HA solutions in general, you and your company need to really take a hard look at what kind of availability is actually needed. What is the real cost of downtime for your company, etc. Questions like, are you loosing $1k+ dollars every single minute services are not available? Or are things like a 30 minute outage just an inconvenience. These types of questions and their answers should help drive what kind of solution you really need to budget for.
 
Last edited:
SANSymphony is so expensive that I don't think you could even come close to license 100TB of HA storage for $50k, and I'm only talking about the software licenses and not the hardware itself. You should just contact them and get a quote though to atleast hear what they say.

It's been a while since I have looked, but something like 4TB of mirrored storage (8TB total) is like $15k. That's just software, I think they may be even more expensive than the traditional SAN companies like NetAPP, HP, EqualLogic, etc.

The absolute cheapest commercial SAN Software with support that provides 2 Node HA that I am aware of is from: Open-E. It is certainly not the most elegant solution, and you are limited to 2-node clusters, but if you follow the HCL, and don't skimp on the hardware, you can build a fast HA SAN. You won't get alot of the nice features of more modern (and expensive) products though.

Most of the software only solutions you may be interested in START at over $10k for the lowest capacity tiers that include mirroring (assuming you are keen on doing the hardware side yourself to save money). I'm also avoiding mentioning any free solutions since I'm assuming this is for a company and you actually want a support contract.

For hardware/software solution you could look into the cost of a Nimble Storage SAN, I honestly don't know what they cost but I have heard positive things about them (assuming they could be more competitively priced). We mostly have EqualLogics, and I have used NetApp in the past. Obviously both out of the price range you provided for that amount of storage and features.


The things you are asking for are not cheap as you know. You say you want redundancy, but where? Everywhere? Networking, Storage, Controllers, etc? For example, I'm personally not aware of any "SAN Software" you can purchase that would allow you to run redundant controllers (if you are going RAID instead of ZFS). With things like storage, and HA solutions in general, you and your company need to really take a hard look at what kind of availability is actually needed. What is the real cost of downtime for your company, etc. Questions like, are you loosing $1k+ dollars every single minute services are not available? Or are things like a 30 minute outage just an inconvenience. These types of questions and their answers should help drive what kind of solution you really need to budget for.

Currently looking at Nimble. Their price "might" be a little better than NetApp. Kind of hard to say because of the systems they offer. We have a NetApp that does 20k IOPS and another that does 50k IOPS. Nimble goes from 30k IOPS right to 90k IOPS. It would be nice if they offered something in the middle.

I haven't really looked into the Microsoft pricing for a Service Provider, but they have Storage Spaces that you can build some decent Supermicro servers with JBOD and setup resiliency. I just find it a bit odd running a Microsoft server for a SAN. Although with their tiered storage you can get pretty good performance.

Ultimately if I do anything it may just be VSAN. The VMware license is very cheap, I would just really like to see what kind of performance you get using NL-SAS with SSD. Our ESXi hosts are already 100% compatible. Just need to buy the drives which I spec'd out based on VMware's HCL and it isn't too expensive. I would start small to test, but was thinking if we filled up 5 servers to get 2x 800 GB SSD and 6x 2 TB NL-SAS for each server. Or do 3x 400 GB SSD and 3x 4 TB NL-SAS. I am assuming read performance is probably great, but I am wondering how write performance is. I know part of Nutanix's secret sauce is with how the writes are done compared to VSAN. I don't remember all the specifics off the top of my head.
 
Back
Top