VMware proper cloud datastores

L

loudsnare

n00b
Joined
Sep 15, 2009
Messages
36
I was wondering, do most of the higher-end vCenter cloud deployments have totally separate storage servers from the hypervisors? For example multiple iSCSI storage units, and separate ESXi's. I assume that would be the proper way.

However, is it OK to use one large 24TB datastore that exists on one of the multiple ESXi machine as both MAIN storage and hypervisor, 8-10 Gigabit ports. Or strictly a bad idea.
 
Sorry. In a professional high-end VMware cloud setup, for total redundancy, is it ok to use the datastore on an ESXi machine? I know it works. Just curious in a real cloud environment if it's not recommended. Instead, just to use iSCSI servers separately.
 
You mean a local datastore on a vSphere host? Never see that in a real prod environment. It's always shared storage via iSCSI/NFS/FC because that is required for the full cluster functions like vMotion, HA, etc.
 
Local data stores are generally a bad idea. If the physical machine with the data stores dies then the machines can't HA like they need to.

I boot all my hosts off flash drives and just map LUNs off the SAN for storage.
 
Definitely. I have a bunch of ESX's and they all load off 8GB usb's.

Built a couple of open-e machines today, so that should do the trick.

Fun stuff. VMware has changed a lot since 2004. Different world it seems.
 
You guys think open-e iSCSI is better, or a default CentOS NFS install for storage devices?

16TB open-e is roughly $1,000 per machine.

They're both on a 10.0 network direct to the hypervisors.
 
Vader said:
Is this for a production vSphere environment?
Click to expand...

yes. I had a testbed set up at the datacenter with 8 hosts and 1 iSCSI via open-e. I figured I should ask if NFS is a viable option as well. The storage is a 3U server with 16 bays in a raid 10 for 16TB (2 of those). The license for open-e is approximately $1K for 16TB. If NFS does the job too, I can spend that money on RAM.
 
yes. I had a testbed set up at the datacenter with 8 hosts and 1 iSCSI via open-e. I figured I should ask if NFS is a viable option as well. The storage is a 3U server with 16 bays in a raid 10 for 16TB (2 of those). The license for open-e is approximately $1K for 16TB. If NFS does the job too, I can spend that money on RAM.
Click to expand...

There are so many things to consider with architecting a vSphere environment, especially on the storage side. If you talk to most people that support vSphere they will tell you that a lot of issues stem from storage problems, insufficient storage performance caused by incorrect configuration and/or insufficient storage achitecture.

I'll assume you did your homework on that front. I would certainly agree with Netjunkie and look into storage solutions that are on the HCL. Keep in mind using NFS, you do not get MPIO..etc so you'll have to manually manage the load accross multiple interfaces and shares.

Also, VMware has added some very useful support for offloading some functions to supported storage arrays and if you think you'll see benefits from that you'll want to find out if the storage solutions supports VAAI, VADP, VASA...etc.

What's your budget? I'm guessing not that much based on what you were looking at.
 
I guess iSCSI is the way to go here. I've gotten great performance with bonding 8x GigE ports into a good switch, to the hosts. If I remember properly, I got near 220MB/s on VM's with my last test with 2 ports on the hosts, simultaneously on 4 hosts.

Assuming you guys are VCP?
 
Why don't you tell us exactly what you are trying to do. You are all over the place and it seems like your guessing at what is best for your environment.

Give us some information, like how many hosts, what version, what are your storage needs..etc..and i'm not just talking about capacity here. While certainly capacity is important, you need to make sure that you meet IOP's and latency requirements of the VM's you'll be running in your environment.

What exactly are you trying to accomplish here?
 
loudsnare said:
I guess iSCSI is the way to go here. I've gotten great performance with bonding 8x GigE ports into a good switch, to the hosts. If I remember properly, I got near 220MB/s on VM's with my last test with 2 ports on the hosts, simultaneously on 4 hosts.

Assuming you guys are VCP?
Click to expand...

This sounds very weird.

I'm way past VCP, most of the others here are VCP at least.
 
Vader, I didn't want to sit here picking everyone's brain when I should probably be hiring a VCP and doing things properly (just out of respect for you guys, and me not looking so moronic).

I'm trying to accomplish a dual datacenter cloud, maximum redundancy, within means. Company is finishing up the VSPP contract with VMware.

At the moment, these are the hardware details. Everything is at cost and servers built in-house.

All one datacenter now, but will be spread across two via 100Mbit VLAN.

ESXi:
4x 12 core Xeon's 64GB RAM each, quad Gigabit
2x 8 core Opteron's 48GB RAM each, quad Gigabit
1x 16 core E5-2690, 192GB RAM, 8 Gigabit ports

Datastore:
2x iSCSI servers, running open-e (one per datacenter?)
8x core Xeon, 32GB RAM
16x 2TB 7200RPM drives, raid 10 on an Areca 1880
2x quad gigabit cards

Switches:
2x GSM7352S

Needs more gear I'm sure. Missing backup solution and potentially other things.

Best to hire a VCP.
 
Best to also stay with hardware on the HCL. Using servers built in house that are not on the list is just asking for problems.
 
Most of the servers are supermicro and on the HCL. And if they're not, I'll make sure not to use them. Can't mess around with this stuff.

I'm reconsidering the storage options as you guys mentioned it's the most essential.

Better sit down with someone who's set this stuff up and make a plan. Way more advanced than I thought. I appreciate the wake up call back to reality.
 
Rather than going a build-your-own for production level storage stuff, I'd look at some of the lower-cost options on the HCL - much better to have a vendor backing up the entire unit (vs just the software, like some of them are) if something goes down.

Don't try to bridge storage over a 100mbit connection - each datacenter will have to have its own storage and kept locally (replication is fine, but don't try to mount over the 100BE connection).
 
Do you have the option of bringing in a constultant from a VAR? Don't take this the wrong way, i'm not here to make you look "moronic." We're trying to assist you but understand we have limited information. I would see if you could work with a local VMware VAR in your area and talk design, etc.

I have a VCP but many others on here do as well, and some I couldn't even compare too in the 'architecture, design' area. You need someone to come in who does this for a living like a Technical Consultant from a VMware VAR.
 
lopoetve said:
Rather than going a build-your-own for production level storage stuff, I'd look at some of the lower-cost options on the HCL - much better to have a vendor backing up the entire unit (vs just the software, like some of them are) if something goes down.

Don't try to bridge storage over a 100mbit connection - each datacenter will have to have its own storage and kept locally (replication is fine, but don't try to mount over the 100BE connection).
Click to expand...

The 100Mbit was just the link between datacenters. I try to type and post fast and can't get my thoughts across sometimes. Definitely not bridging storage across the datacenters. And you're right about the low cost storage options on the HCL, instead of just the software. In this case I better have vendor support done properly.
 
Vader said:
Do you have the option of bringing in a constultant from a VAR? Don't take this the wrong way, i'm not here to make you look "moronic." We're trying to assist you but understand we have limited information. I would see if you could work with a local VMware VAR in your area and talk design, etc.

I have a VCP but many others on here do as well, and some I couldn't even compare too in the 'architecture, design' area. You need someone to come in who does this for a living like a Technical Consultant from a VMware VAR.
Click to expand...

Thanks for your kindness. Never read anything in a negative was. I know you guys are all being cool, and I'm grateful for that.

I have a few friends who are VCP, but I hate asking them for favors. Some people just make you feel like you owe them for life, even when you frickin' pay.

Agreed, I better find a consultant in the architecture/design who does this daily. Do you guys know where to post such a job request? Even though some of you guys work for other companies already, do you do consulting on the side?
 
sure it does. you just need to use dns round robin to make it work.
Click to expand...

That's not quite MPIO, but it's certainly better than manually setting it up. All you are doing it automating the manual process. Pretty fugly though.

I would also say that a post with minimal input on VMware's site does not validate this working accross the board or supported. I'd like to see others in a prod environment using this.
 
well, it really depends on what you're expecting from MPIO. if you're one of those folks that thinks 4 x 1gb interfaces means you now have 4gbps of throughput you need to read over some documentation again.

that said, round robin from within vsphere for iscsi targets does have some performance advantages as well as better load balancing. however, gigabit is terrible for performance anyway and you should be doing everything in your power to move away from it as fast as budgets allow.

also, do note with vsphere 5 there is full support for dns round robin and nfs ... sadly no nfsv4 yet but there is a lot of folks doing global namespace with NFS nad using DNS to load balance on the back end. granted, it is 'dumb' load balancing in that there aren't any latency checks unless you're using F5's acopia line .. well at least i assume their product line has that built in, it better as they charge a boat load for it ... i digress. the point is with 10gbit you're not likely to bottleneck any single interface as you can transfer 1.1GB/s so you leverage an easy to manage global NFS namespace with simple yet effective load balancing (and HA) via round robin DNS for general purpose low to medium IOP workloads and you use SRP or block iscsi for any workload that requires more consistent resources and or you have a specific SLA requirement.
 
well, it really depends on what you're expecting from MPIO. if you're one of those folks that thinks 4 x 1gb interfaces means you now have 4gbps of throughput you need to read over some documentation again.

that said, round robin from within vsphere for iscsi targets does have some performance advantages as well as better load balancing. however, gigabit is terrible for performance anyway and you should be doing everything in your power to move away from it as fast as budgets allow.

also, do note with vsphere 5 there is full support for dns round robin and nfs ... sadly no nfsv4 yet but there is a lot of folks doing global namespace with NFS nad using DNS to load balance on the back end. granted, it is 'dumb' load balancing in that there aren't any latency checks unless you're using F5's acopia line .. well at least i assume their product line has that built in, it better as they charge a boat load for it ... i digress. the point is with 10gbit you're not likely to bottleneck any single interface as you can transfer 1.1GB/s so you leverage an easy to manage global NFS namespace with simple yet effective load balancing (and HA) via round robin DNS for general purpose low to medium IOP workloads and you use SRP or block iscsi for any workload that requires more consistent resources and or you have a specific SLA requirement.
Click to expand...

Who said anything about throughput? Multipath is not just about a single connection to a single nfs share or datastore and then going to another connection to another datastore or in this case share. It's also about multiple paths to that same datastore, which DNS round robin doesn't take into account in the NFS case. Again, all you are doing is simplifying the manual process of setting up NFS shares per interface, that is all. There aren't additional paths to the same share, and there aren't any advance features or policies.

Having said that, I would be fully onboard using this method, it certainly automates some of the manual bs that you have to do to effectively "load balance."
 
Last edited:
Vader said:
Who said anything about throughput?
Click to expand...
clearly you understand, but how many folks have you dealt with that think 802.3ad = more performance? was just stating to clarify.

Multipath is not just about a single connection to a single nfs share or datastore and then going to another connection to another datastore or in this case share. It's also about multiple paths to that same datastore, which DNS round robin doesn't take into account in the NFS case. Again, all you are doing is simplifying the manual process of setting up NFS shares per interface, that is all. There aren't additional paths to the same share, and there aren't any advance features or policies.
Click to expand...
sure there are

nfs IN A 192.168.0.4
nfs IN A 192.168.1.4
nfs IN A 192.168.2.4

or all on the same subnet if you prefer.
 
I guess i'm missing something here, maybe you can shed some light. Say I have two interface on the vSphere side dedicated for NFS traffic. In your example, how would both interfaces be setup, in a teaming policy and then just add the NFS shares by using the FQDN, RR is done on the NFS Server side and load balancing on the vSphere side is achieved by the teaming policy?
 
Vader said:
That's not quite MPIO, but it's certainly better than manually setting it up. All you are doing it automating the manual process. Pretty fugly though.

I would also say that a post with minimal input on VMware's site does not validate this working accross the board or supported. I'd like to see others in a prod environment using this.
Click to expand...

It's fully supported. NetApp ran it for a bit as an idea, but it doesn't matter at all, because interconnect is a pointless concern in the real world. It's been dropped as a common configuration method as a result, although you do still see it in the CNA world some for those on the bleeding edge.

madrebel said:
well, it really depends on what you're expecting from MPIO. if you're one of those folks that thinks 4 x 1gb interfaces means you now have 4gbps of throughput you need to read over some documentation again.

that said, round robin from within vsphere for iscsi targets does have some performance advantages as well as better load balancing. however, gigabit is terrible for performance anyway and you should be doing everything in your power to move away from it as fast as budgets allow.

also, do note with vsphere 5 there is full support for dns round robin and nfs ... sadly no nfsv4 yet but there is a lot of folks doing global namespace with NFS nad using DNS to load balance on the back end. granted, it is 'dumb' load balancing in that there aren't any latency checks unless you're using F5's acopia line .. well at least i assume their product line has that built in, it better as they charge a boat load for it ... i digress. the point is with 10gbit you're not likely to bottleneck any single interface as you can transfer 1.1GB/s so you leverage an easy to manage global NFS namespace with simple yet effective load balancing (and HA) via round robin DNS for general purpose low to medium IOP workloads and you use SRP or block iscsi for any workload that requires more consistent resources and or you have a specific SLA requirement.
Click to expand...

Repeat after me.

"Interconnect doesn't matter at all in a virtualized environment"

Congrats, now you won't make the number 1 mistake that our customers make. :) IOPS IOPS IOPS IOPS IOPS. IOPS > *. I've seen ONE customer in almost 5 years of doing this that has needed to be concerned about interconnects, and their environment was a masterpiece (with 4 connections dedicated to SQL / Exchange load, and they actually used up every drop of the connection on 4 gig FC and now 8 gig FC). All the rest? Run out of disk first. WAY first. Even with 1GBE (assuming appropriate failover capacity/design). The reality is that no disks can keep up with randomized sizes, data, and order, and that's what a virtualized environment sends. Your interconnects won't matter just because the disks are crying in their sleep.

Oh, and DNS and NFS is nubtarded unless you've got a DNS server non-virtualized - please think of the chickens and the eggs ;) If you do, then cool, but otherwise, imagine the total outage. I get a kick out of the whole "our storage is down because our DNS servers are down because our storage is down beca..." It's hilarious, but only afterwards.
 
Didn't mean to hijack this thread...thanks for the info lopoetve, I didn't realize that it was a VMware supported method of NFS loadbalancing. I didn't find much else on it doing search on google other than the blog that madrebel linked.
 
lopoetve said:
Oh, and DNS and NFS is nubtarded unless you've got a DNS server non-virtualized - please think of the chickens and the eggs ;) If you do, then cool, but otherwise, imagine the total outage.
Click to expand...
i always recommend at least one 'hard' host. i prefer two BSD boxes running jails and virtual box for at least one AD server too.
 
madrebel said:
i always recommend at least one 'hard' host. i prefer two BSD boxes running jails and virtual box for at least one AD server too.
Click to expand...

Why? I see customers still doing the "one physical DC" thing. I see no reason.

And as usual, lopoetve is right about disk. It's almost ALWAYS about the IOPS yet people go through crazy things to try and increase bandwidth..needlessly.
 
You must log in or register to reply here.
Back
Top