custom SAN with infiniband

[sdadept]

I'm building a SAN for our QA environment which will have some 100 or so virtual machines back ending to the system. I intend to use infiniband (although DDR or QDR to save $). I've googled around a bit and I haven't found very many build guides for this sort of thing.

anyone have suggestions on the best way to build this sort of SAN? I'm looking for pretty heavy multi-user access and about 50tb of storage. I've got about 20 servers that I also got from ebay that are running vm's now but all with local storage which is a beast to manage. Budget is $20k to $40k or so.

my early thinking is

DDR infiniband which I can get super cheap parts for on ebay that will get me easily to 20gbps bandwidth.

10tb worth of SSD's
50tb worth of SATA drives

thoughts?

 

[_Gea]

I can only tell about my decision.
While you can buy really cheap older IB equipment and IB may have some advantages regarding latency, the mainstream was FC and is now iSCSI.

With your budget, price of an adapter is not relevant.
I would go mainstream with up to date hardware and driver support like Intel 520 + iSCSI

For the data, use ZFS (BSD, Linux, Solaris best as a Web managed storage appliance).
My favourite is Solaris or a free fork like OmniOS (supports IB, FC and iSCSI)
http://docs.oracle.com/cd/E23824_01/html/821-1459/fmvcd.html

 

[patrickdk]

I attempted this also.

While tempting, and it does work good, it is normally not practical, unless you want to babysit it a lot.

Generally you have nothing but driver support issues, when using infiniband, as you are stuck, with no place to turn. This will generally mean mellenox drivers, and they are very slow to support new os releases, if you use vmware.

If you are fine using older vmware, windows, or linux, you will likely be ok.

You will also need likely need to pay attention to if you use connected mode or not on all your nodes. and then see if what you have supports srp or iser over all your platforms.

If your running a pure linux eviroment, with only linux storage systems, it could work out ok, I just found, through testing, and how many years it takes to get new drivers for vmware, it just took too much work, for the extra gains it had.

Plus you are seriously limited on distance, if you can't mount it all in the same rack or adjacent racks.

 

[sdadept]

we are going to run a hyper v environment with windows servers (I hope). We've bought some QDR equipment so far and had some good luck with drivers. We'll see if it works out that way once we really push this stuff.

we are looking at freenas for the tiering support (does zfs have that?).

so far we bought

2x of these

http://www.newegg.com/Product/Produ...yMark=False&IsFeedbackTab=true#scrollFullInfo


4x of these

http://www.newegg.com/Product/Produ...yMark=False&IsFeedbackTab=true#scrollFullInfo

4x of these

http://www.newegg.com/Product/Product.aspx?Item=N82E16820147374

 

[sdadept]

Just a followup on this. We went with QDR infiniband and 40gbs. I bought the switch on ebay for $1300. The infiniband cards cost us about $220 a piece new. We are using windows 2012 r2 with tiering turned on. We filled out 12 drives with 6tb SAS drives in raid 6 (12 drives x 6 = 72 - 12tb for raid level). We also bought 12 1tb ssd's and are running them in raid 1+0 for the tier 1 layer. So we now have a 60tb SAN with tiering.

We bought 10 one generation old dell servers with 16 logical procs per box and 96 gigs of ram per box. With infiniband we back ended those to the SAN server. Total was $1k per dell server + $120 for the boot drives we put in them so $11k.

With existing windows licenses we already had we have spent about $36k total for the entire thing and now have a 60tb san, infiniband running 40gbps, and 960gb of ram, 160 logical processors. This has made for a screaming fast testing/dev virtual machine environment where we have close to 200 vm's running.