Gigabit iSCSI VMware Farm

C

chilliblast

n00b
Joined
Feb 15, 2014
Messages
7
Hi all

I'm looking for some second opinions on a solution idea I am putting together.

Essentially I have a need to host the following VMs:

14 x Windows VMs 3 of which are running SQL Server
10 x Linux VMs mostly network orientated rather than IO
1 x Physical backup server via local disks and IBM LTO6 tape device

Total IOPs approx 5,000

My first thought was to go with 10GbE this is what I loosely specced out:

5-6 Dell R620's 2 x 8C 2.60GHz Procs, 64GB RAM, 2 x DP 10GbE NICS (Broadcom 5718 BT), 1 x QP Gigabit NIC

VMware vSphere Standard / vCenter Server Standard Appliance

HP StoreVirtual 4335 SAN
3x400GB SSD
6x900GB 10k
4xGbE
2x10GbE

HP 6250 LTO drive

2 x Dell 8132's (10GbE)



However that is a little over budget so I may well be pushed to go with gigabit iSCSI, I am worried around latency, throughput (and requirement for 5k IOPs) and frame drops with only gigabit and I cannot afford the luxury of LACP / port channel in VMware Enterprise (i realise LACP does not shuttle IO across multiple GbE iSCSI paths at the sametime, that is the job of MPIO) I can only afford vSphere Standard with HA / vmotion etc.

My aim is to find a suitable gigabit SAN that can handle the IOPs with active / active dual controllers support and aggregate IOs over multiple iSCSI paths (for example 4 x ports per controller). What do you think? Any recommendation on SANs?

Now the question is of suitable switches, heres my list:

Cisco 4948-10GE (but these are EOL i believe??)
Dell PC 5524
Dell PC 7024 / 7048
Cisco 4500-X
Force10 S25N
Force10 S50N
Etc.....

Any thoughts?

With the preferred 10GbE solution the networking and SAN blows out my budget just slightly but worried gigabit will not have low enough latency and high enough throughput but maybe it would be ok with aggregated links?

Thanks all
 
Th 6x 10k disks in that array probably have a theoretical peak of 900 (150*6) IOPS. Maybe I am just way off here but if you are pushing 5k sustained IOPS then the array you picked will not be able to support that regardless of the flash hybrid because the flash cannot write it to spinning disks fast enough if it needs to be written to spinning disk. If all of your workload fits on flash and it only migrates old data to spinning disk then it may work ok.

As far as SAN recommendations go, I don't have any for what you need within your budget. I'd buy different servers that accept 8 or 16 disks, and I'd buy them without disks and stuff them with my own Constellation ES drives. Then I'd buy a storage appliance to tie all of those servers together (DataCore? big bucks though, maybe there's something less expensive) into a distributed cluster and will provide you with redundancy and the IOPS you are looking for. You will no longer need 5k IOPS on the array, you'll only need ~1k IOPS per host and that's easily done even without flash.
 
Last edited:
What is the budget? I'm tempted to recommend the cheapest Nimble but it may be out of your price range. You can probably get away with 1Gb networking, MPIO works very well and throughput is rarely the bottleneck. I'd rather have a nice 1Gb SAN over the risk of cheap 10Gb.
 
Last edited:
If you just want a SAN then the Jetstor 716iS ( http://www.acnc.com/p/JetStor_SAS_716iS_10G_iSCSI_IP_SAN ) will set you back like maybe 13-14k (diskless), you can stuff it with your own disks, comes with 2x GigE and 2x 10 GigE, has VAAI support, does not have an annual maintenance fee. Even at 16 spindles getting to 5k sustained iops isn't going to happen unless you fill it with SSDs. Distributing the IOPS among local storage on the hosts is the better option imho.
 
Hey guys

Much appreciate your input thanks!

5000 iops is not sustained apologies that would be peak / burst more than likely around half that figure maybe slightly less at a constant purr.

I was using 150-200 as a general guide for IOPs for a 10k disk plus the SSDs.

Budget is approximately $175k.
 
Thuleman said:
If you just want a SAN then the Jetstor 716iS ( http://www.acnc.com/p/JetStor_SAS_716iS_10G_iSCSI_IP_SAN ) will set you back like maybe 13-14k (diskless), you can stuff it with your own disks, comes with 2x GigE and 2x 10 GigE, has VAAI support, does not have an annual maintenance fee. Even at 16 spindles getting to 5k sustained iops isn't going to happen unless you fill it with SSDs. Distributing the IOPS among local storage on the hosts is the better option imho.
Click to expand...

That looks pretty good! Have you used one in production yourself? What do you think to the performance / features / management compared to the spec sheets online?

Thanks
 
If you're going to spend $175k I would talk to a VAR you can trust to help you choose the right hardware. They can help gather performance metrics for what you plan to virtualize, document your workloads, size what you're purchasing appropriately, and help set it all up. They can help as much or as little as you want.

For that kind of money you could probably get a Cisco UCS starter bundle with 4 blades, Nimble SAN, and vSphere licenses.
 
chilliblast said:
That looks pretty good! Have you used one in production yourself? What do you think to the performance / features / management compared to the spec sheets online?
Click to expand...

I currently have 3x 616iS and one expander in production and had no troubles with it. The management interface is very simple but gets the job done. Any performance stats need to be retrieved through vSphere. I am considering to pick up a 716iS as I do need to add another 25 TB.

If you want to run SSDs in addition to SAS you'll need some sort of 3rd party software that makes the auto-tiering work (that's what you pay Nimble for, it's the logic, not the hardware), if you want to have it migrate data to/from SSD as needed.

What I like about the Jetstor is that there's no annual maintenance (that's 1-2k I can put toward other things), and that I can put my own disks in it and am not vendor-locked buying 300-400% marked up disks from EQL, EMC, HP, etc.

If there is a problem with hardware you are expected to send in the broken part before they ship the replacement. The array comes with a 3 year warranty which you can renew for I am not sure how much. If you buy it with drives then the drives come with a 5 year warranty.

This type of deal isn't for every deployment out there, but I have nothing but good things to say about JetStor and I can't imagine that I will ever buy another EMC, EQL, HP, etc. array again for my current place of employment.

Having said that, at $175k you have quite a huge budget. I am reasonably sure that I paid like $6k per R620 with 96GB RAM last year. That's 36k if you buy 6, that leaves you with lots and lots of money to buy switches, storage, and vSphere licenses. Remember that you will need to pay Cisco big bucks for annual support, ios upgrades. You could try to buy used from http://www.networkhardware.com/ which will also support those switches for a fraction of what Cisco is charging.

I agree that a VAR would be a good option if you have a good one. I disagree on the Cisco UCS. Buying blades at a time when SMB moves to more local storage is imho not a good idea. Blades solve a problem SMBs don't have, density isn't an issue for most SMBs. I'd buy servers that hold a bunch of local disks and then I'd pay for a couple of DataCore nodes with all that money you have.
 
Thuleman said:
I agree that a VAR would be a good option if you have a good one. I disagree on the Cisco UCS. Buying blades at a time when SMB moves to more local storage is imho not a good idea. Blades solve a problem SMBs don't have, density isn't an issue for most SMBs. I'd buy servers that hold a bunch of local disks and then I'd pay for a couple of DataCore nodes with all that money you have.
Click to expand...

With how cheap UCS bundles are, they make a pretty good play even for SMB. There are also advancements coming for UCS around storage, too, but that's down the road.

Still, even just a few decent rack mount servers, basic SAN, vSphere licenses, and a backup device is very doable with $175k. OP has a nice budget to play with.
 
Drop down to 3-4 hosts and respec with 128 or 256GB of RAM.

Get a quote for a Nimble CS220.

Look at the Force10 S55's for 1g, or the S4810 if you are going 10G.

I would also swap out the broadcom for intel on the daughtercard.
 
Last edited:
I would first defne:

- what is the max downtime until you need to regain service
- how many VMs are affercted on problems
- what complexity can you manage yourself
- how fast and effective is your external support

If you say, no downtime you need an all-redundant setup with hot-switchover on any failure (VM-Server, Storage, Network, Power). This is very complex, you should have very experienced redundant staff, and last it is very expensive.

As a contrast, if you can allow a max downtime of say 30 min on any failure you can go very cheap if you split your demands over smaller machines.

Ex: use several VM server each with 5 -10 smaller VMs, connect them over a redundant switch and use several smaller SAN devices with a mirrored Enterprise SSD config each like with a pair of Intels new S3700 (or a 3way mirror). If you add a dedicated small SAN device to every VM Server, you can reduce any problem to a single VM-Server-SAN combination.

I do this with a next step and integrate the SAN part into my VM-Servers as a virtualized SAN with NFS. This can be done nearly without performance problems if you virtualize only the SAN OS but not the storage part due to hardware pass-through. You only need enough RAM for your SAN VM like 16-32 GB or more to use the RAM as readcache. CPU is never a problem. Seveval thousand I/O per machine is achievable without problem.

Maintenance:
- complete hardware failure: put SSDs into next machine, import the storage and VMs and start or start from backupserver: 10min- max 30 min downtime
- planned maintenance: hotmove VMs with hotmove storage
- manual clone/ move/backup: Copy your VMs via NFS or SMB to next datastore
- backup: use ZFS snapshots and replication every few minutes or hours optionally paired with a ESXi solution to backup hot states

- use pairs of such in-one-configs where you can insert the disk from the counterpart
- async-replicate all boxes to a internal and external backup location.
- add a coldspare system from the saved money

Can all be done with some Dells, HPs or SuperMicro boxes, a 10 GB network (can be redundant with a 1 Gb network) You only need boxes with at least 8 better 16 x 2,5" bays. If you can allow OpenSource SAN software, you can go with a ZFS storage. I prefer Solaris based ones. Such boxes offer in many aspects netapp similar features without the cost and with a very low complexity.

I introduced the concept many years ago and use it since then.
I call it napp-in-one, see http://www.napp-it.org/doc/downloads/napp-in-one.pdf


All a matter of philosophy as an option to Nimble or netapp.
 
Last edited:
A one or two way Nutanix setup would be pretty sweet, and you could scale out linearly later.

I almost bought two nodes for a project a couple months back but it was delayed, really nice product.
 
shiznit said:
A one or two way Nutanix setup would be pretty sweet, and you could scale out linearly later.

I almost bought two nodes for a project a couple months back but it was delayed, really nice product.
Click to expand...

Do tell what your quote for 2x Nutanix was.
 
3 nodes of nutanix is around 90-130K. Depends on the memory and CPU of the nodes.
 
My numbers were for the NX-3000 series. Looks like the NX-1000 comes in at a lower price point.
 
Sorry, I wasn't trying to prove you wrong or anything. Just saw that and figured it was worthwhile to post it.
 
No biggie, the 1000 series wasn't out yet when we quoted ours. Good information to have if we need to quote out a satellite site.
 
Your budget of $175k is quite nice, but I would definitely reduce the number of hosts for this project.
Up to 6 hosts for only 24 VMs seems a bit oversized. You pay VMware per socket, so that's 12 licenses to purchase. In this case you would use 1 vShpere license per 2 VMs - not very cost-efficient.
Depending on the VMware features you want/need, maybe even a 3 host essentials kit could be sufficient. All for a fraction of your original price.

3 or max. 4 hosts should give you enough CPU power for those VMs. But you really should crank up the RAM to 128 GB at least - more is better. In many environments RAM is the first barrier and CPU is hardly maxed out ever.
 
Hey guys

Sorry for the delay, work is crazy this week.

The issue with reducing hosts is one of resource reservations, the VMs that will be hosted are not your usual VM's with relaxed resource requirements (when I say relaxed I mean no enforced reservations at a per VM level) - they WILL have reservations set on each VM that must be adhered to.

I know what you are going to say and I agree, that goes against the whole principles of virtualisation - but it is unfortunately the requirement hence the number of VM hypervisors which will exceed this reservation requirement (mostly CPU I might add) and provide a level of redundancy.

Now the IOPs issue I've managed to reduce this to around 3500 with a bit of tweaking the spec, but as you have said above, we are still looking at potentially a full 24 drive array of at least 10K's to meet this without considering RAID penalties and array meta data.

Pricing up a 10GbE it just goes over budget or target budget, GbE ISCSI however is just in line - this would be 6 servers each with 4 aggregated storage paths (round robin / 1 x IO per path setup) to a dual controller (active / active) array with 2 x GbE ISCSI ports per controller.

At the moment I am quite liking the NetApp range (I've taken on board what you guys have recommended too and they all look good) mainly because I like the idea that I can drop in a mezz card and have 8GB FC or 10GbE functionality right away (from Gigabit). Also the lower end NetApp arrays can take 24 drives and provide CIFS/NFS protocol out of the box (FAS2240-2 etc).

The area that is of most concern is using 1GbE for such a high IO requirement, obviously switches are important and at the moment I'm favoring the Force10 550N at least for Gigabit ISCSI/NFS.

Does anyone have any experience with the 550N? (with similar IO requirements).

Really appreciate your input guys.

Thanks again
 
Does your end client even understand how virtualization works re: cpu reservations?

And don't do 1xIO per path - that's pointless.
 
You'd hope so but there is still the requirement for enforced reservations, I totally agree with you that is against the whole point of virtualizations but it is what it is unfortunately (which makes it more expensive).

Why is round robin across multiple paths with controlled IO rates pointless? I'm basing that on not being able to afford MPIO / LACP (Enterprise / Enterprise Plus VMware) btw.

What would your alternate recommendation be?

Thanks
 
Doesn't buy you anything past 1-2 hosts and 1 vm per lun, that's why :) Round robin with the defaults does equally well once you exceed those numbers by design. :) We did a good job with it.
 
chilliblast said:
You'd hope so but there is still the requirement for enforced reservations, I totally agree with you that is against the whole point of virtualizations but it is what it is unfortunately (which makes it more expensive).

Why is round robin across multiple paths with controlled IO rates pointless? I'm basing that on not being able to afford MPIO / LACP (Enterprise / Enterprise Plus VMware) btw.

What would your alternate recommendation be?

Thanks
Click to expand...

What is the end users' justification for setting high CPU and memory reservations? Sounds to me like your first battle is refusing to go forward with the project until they change their tune. If not, tell them you need a bigger budget or just stick with physical servers.

And I wish people would stop thinking 1Gb ethernet will be an IOPs bottleneck. It may be a throughput bottleneck, but not IOPs. IIRC, the theoretical maximum IOPs on a 1Gb link is into the tens of thousands.
 
Thanks lopoetve, what kind of infrastructure do you have in place if you don't mind me asking? :) I need to do some further testing I think!

CoW - I am worried about latency across the gigabit links, if something starts churning away that throughput it will push latency up, at least with 10GbE I have a little more room to maneuver, but if I could get away with gigabit that would be a winner.

Are you using ISCSI gigabit heavily yourself in a VM environment? Is anyone here - just interested to hear your experiences in the field (for my benefit) but seriously pushing the boundaries rather than the odd file server, web host VM etc. (thinking more along the lines of SQL / DB2 / Postgres - DB / Windows hosts which notoriously eat up throughput).

Unfortunately the resource reservation issue is non negotiable but as I have said I agree with you and those above 110%, it goes against the basic principles and advantages that virtualization provides.
 
Nonnegotiable=not understanding

You need to show them. I mean, in reality, if you really wanted to you could add reservations later..etc...but I doubt you will even need to do that.

If you're that worried about latency then I would want to look at a couple of things:
1. Move your storage as close to compute as possible, others have suggested Nutanix..etc. It doesn't get closer than that other than VSAN which is yet to be released, Q1, however.
2. Possibly move to Fibre Channel to address latency concerns if not going with option 1, expensive unless you go with something like UCS and zone within the FI's.

CoW mentioned UCS bundle, you can certainly achieve low latency there with the fact that you can collapse your storage fabric in the Fabric Interconnects and it supports all protocols to include FC Zoning if you wanted to go that route.
 
chilliblast said:
Thanks lopoetve, what kind of infrastructure do you have in place if you don't mind me asking? :) I need to do some further testing I think!

CoW - I am worried about latency across the gigabit links, if something starts churning away that throughput it will push latency up, at least with 10GbE I have a little more room to maneuver, but if I could get away with gigabit that would be a winner.

Are you using ISCSI gigabit heavily yourself in a VM environment? Is anyone here - just interested to hear your experiences in the field (for my benefit) but seriously pushing the boundaries rather than the odd file server, web host VM etc. (thinking more along the lines of SQL / DB2 / Postgres - DB / Windows hosts which notoriously eat up throughput).

Unfortunately the resource reservation issue is non negotiable but as I have said I agree with you and those above 110%, it goes against the basic principles and advantages that virtualization provides.
Click to expand...

Mine doesn't count - I'm in "ludicrous" space ;) (56G IPoIB, 40GbE, and a handful more).

Latency can be just fine on Gig as well - it's not 10G, mind you, or FC for certain - but it'll be just fine for small/mid size commands. If you're trying to run a lot of exchange boxen on there we'll want to do some IO sizing tuning, but even then we can make it work just happy as a clam. Disks are your limiters more than anything.

I've seen gig setups pusing 50k+ with the right setup. You'll be fine.
 
chilliblast said:
I am worried about latency across the gigabit links, if something starts churning away that throughput it will push latency up, at least with 10GbE I have a little more room to maneuver, but if I could get away with gigabit that would be a winner.
Click to expand...

I think that you underestimate the disk latency issues you will run into and you overestimate network latency. Under normal operating conditions you won't see decent switches introduce latency. Our production environment is running on old Cisco Catalyst 2960s and there's a lot of highly variable traffic running across those from dozens of sources yet it has zero impact on measurable latency.

You are guaranteed to see disk latency in your environment even if you buy an array with 24 drives.

Check this out:

cLoL1zi.png


That's a single Microsoft SQL 2008 server on a 16 spindle (albeit SATA 7.2k drive) array in a 3-cell RAID50 datastore. A SAS drive array would lower the latency a little, but it certainly wouldn't bring it down to a level that network latency is anywhere close to being significant over disk latency.

The point is that your concern with network latency is going to be dwarfed by the disk latency you will inevitably experience if you hook VMs from 6 hosts into a single SAN.
 
We run 1gb iSCSI in some of our older sites(Equallogic/Nimble). Unless you're aiming for some crazy backup or file transfer windows the storage network throughput isn't going to be an issue.
 
You guys are awesome - thanks for the info, that graphing screenshot above is very interesting and very useful.

In terms of 10GbE SFP based switches, what would you guys recommend that won't break the bank? I was looking today and noticed that Mellanox do a SFP based switch that looks very well priced but unfortunately I cannot find any information on it in terms of performance.

Any other "cheap" recommendations without sacrificing performance? (the holy grail)

Once again thanks!
 
One of the biggest cost cutters for 10G I've run into would be Twinax cables. If your cable distances are within the 1-5M range these can be significantly cheaper than SFP+ and fiber.
 
The link for Network Hardware that Thuleman posted would be a good bet. You could get a Nexus 10GB switch from them super cheap. There are quite a few places that Cisco authorizes the sale of their old equipment that's also SmartNet capable. Look at http://www.vology.com/ as well. If you want to go the Cisco route.
 
geiger said:
One of the biggest cost cutters for 10G I've run into would be Twinax cables. If your cable distances are within the 1-5M range these can be significantly cheaper than SFP+ and fiber.
Click to expand...

Just be careful with ISLs and multi-vendor fabrics. Most switch vendors don't take kindly to non-branded transceivers, and if they hard-block them there's no such thing as a multi-vendor cable.
 
Might be worth looking at the StoreVirtual VSA.

You can put together something seriously nice and seriously redundant at box and node level using a couple of servers stuffed with a tier of SSD and a tier of enterprise SAS and a RAID controller with SSD cache behind it.

I'm looking at options now and can't help but think software defined is a better bet then sinking money in traditional big iron arrays.
 
You must log in or register to reply here.
Back
Top