• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Comment on this setup please

R

RyteSyde

n00b
Joined
Dec 6, 2012
Messages
15
Greetings fellow VMers. I wanted to see if I could pick your experience and comment on our soon to be setup as well as enlighten me on a few items.

We are going to migrate many of our servers into VMs and we are currently leaning towards the following

2 x Intel® Xeon® E5-2690 2.90GHz, 20M Cache
192GB MT/s RDIMM RAM
2 x 146GB 15K SAS 6Gbps in RAID 1 config (just for ESXi hosts, .iso and template storage)
2 x 2x10Gb DA/SFP+ + 2x1Gb BT Network Daughter Card (4 x 10Gb and 4 x 1Gb)
Additional 1GB NIC's for
Redundant power supplies etc

For the SAN we are planning on:
12 x 900GB 10K SAS in RAID 6 (can't do 10) for 9TB usable
1 x 2x10Gb DA/SFP+ + 2x1Gb BT Network Daughter Card

VMware Standard Accel Kit

We'd like to use the HA and Fault Tolerance so we'll order two of the servers. The host will run 9-10Vm's, all 2008 Server R2 Datacenter. Nothing super fancy on the guest side, PDC, BDC, 1 Exchange (50 users), 1 SQL, Print, other 3rd party, Terminal Server.

1) First question: thoughts about the setup, CPU's, Memory, etc. Should be a beast for 10VM's but would like to hear your thoughts.

2) My second question has to do with the connectivity between the ESX hosts and the SAN. I am getting different information on this so I'd like to hear your input. We will be using SFP+ and I don't see why I can't cable directly from the hosts to the SAN like so:

ESXSAN02.gif


granted the failover lines (blue) will be on secondary cards but I didn't draw it that way, so imagine the blue lines as a second physical card.

I'd have a 10Gb for the SAN, a 10Gb for the management network and additional cards not depicted here for the guests/LAN. Is there a reason other than expandability I can't do this instead of using an overly expensive 10Gb switch? Is this segmented enough? I'd think the vSwitches would handle that. Seems to me if I added a physical switch, it woud bring in an additional single point of failure so then I'd want two. I'd much rather spend that $20K or so on making the SAN redundant.

That leads to question 3
3) the SAN is really a NAS, 2008 Sotrage Server R2. I'm not so happy about the SPoF with having just 1 NAS so what would be a good way to fault tolerant the storage array? I'd have the funds to duplicate the NAS but what's needed on the VMware side? I'd like to have as close to 0 downtime hardware/VM as possible (excluding software issues)

Thanks your your time and experienced input!
 
Buy a switch for the SAN. That direct connect using diff speed NICs is very ugly and will bite you.
 
And if you say "But a 10Gb switch costs too much!" I'm going to tell you to skip 10Gb on that SAN. You don't need it for two hosts. Better off with quad 1Gb NICs in that SAN and going that way.
 
You may also want to reconsider using RAID 6. There's a hefty write penalty with it and if your VMs are going to be doing much writing, you'll see high latency and low write speeds on your datastores because of it. If your VMs will be doing very little writing, then you can get away with it.

EDIT: Now that I see you're running SQL and Exchange on here you may be better off doing 12 drives in RAID 50 but I would highly recommend getting a hot spare or two in there, too.

Netjunkie is absolutely right about the switch. Also, why are you setting up the hosts with 10Gb management? Use your 10Gb ports for storage and VM traffic only while your 1Gb are management.

vSwitch0 - 2x1Gb Management
vSwitch1 - 2x1Gb FT
vSwitch2 - 2x10Gb Storage
vSwitch3 - 2x10Gb VM traffic primary on NIC1, standby NIC2, VMotion primary NIC2, standby NIC1

If you're going to use FT to protect very busy VMs, then move VMotion to vSwitch1 and use two vmkernel ports for 2Gb of VMotion throughput and move FT to vSwitch3 in place of VMotion.
 
Last edited:
And don't forget...you can only use one vCPU with FT. And that isn't changing any time soon.
 
Plus the disks on the VM's protected by FT must be eager zero thick provisioned in case you were planning on having them thin provisioned.

FT really comes down to "are the huge amounts of restrictions really worth avoiding the 2 minutes of downtime it takes for HA to restart the VM on the other host?"
 
Thanks for your input so far, great food for thought.

I thought FT/vMotion traffic went across the management network and having 10 VM's replicating would be better to have them on 10Gb.

It's not that a $10K switch is too expensive, it's that I didn't want to spend that if it weren't necessary if the direct connect will accomplish the same thing with the same reliability, just not have expandability. It is looking like pay me now or pay me later so perhaps it's best to go ahead with the physical switch and be done with it.

Really we don't need two hosts for the 10 VM's, the second was strictly for manual load balancing and HA/FT. I wasn't aware FT can only use one vCPU and that's not going to cut it on at least half of the VM's. I also cannot do thick provisioning on 2 of the VM's so that's out for FT on those. I'd like to have FT for sure but looks like we're going to need to settle for HA. as CoW stated, not really worth FT on these considering the restrictions.

One of our 3P software providers has sent me performance charts for the SAN we'll be using. It's a Dell NX3200 which is basically a PE R720 server chassis. He says they get better IO performance out of that with 12 7.2 NL-SAS than say a $40K Equilogic. I'll trade the 20TB capacity on 7.2's for 9TB on the 10K spindles. Charting our current IO, we'd infrequently peak at about 90% of the 7.2K setup IO capacity so I like the 10K IO padding. The only options for RAID are 5 or 6. So you'd rather see RAID 5 w/ hot spare?
 
If you're going to be using an actual server as your disk box you might want to use an HP instead, their raid controllers are much faster than Dell's.
 
FT and vMotion go over whichever vmkernels and NICs you specify. Sure, it's nice to do vMotion over 10Gb but given your requirements it's not a requirement. I'd rather see you use 1Gb with a good pair of switches for redundancy.
 
RyteSyde said:
Thanks for your input so far, great food for thought.

I thought FT/vMotion traffic went across the management network and having 10 VM's replicating would be better to have them on 10Gb.
Click to expand...

No, you have to create separate vmkernel ports for VMotion and FT. They can go over the same physical NICs as management but you have to create a logical vmkernel port to sit on that NIC. You can't use FT to replicate 10 VM's on 2 hosts anyway, the limit is 4 per host.

It's not that a $10K switch is too expensive, it's that I didn't want to spend that if it weren't necessary if the direct connect will accomplish the same thing with the same reliability, just not have expandability. It is looking like pay me now or pay me later so perhaps it's best to go ahead with the physical switch and be done with it.
Click to expand...

Yes, I would. If you're going to spend the money to build a virtual environment that will support such important pieces of your infrastructure and will serve as the foundation point for future expansion you should do it the right way.

Really we don't need two hosts for the 10 VM's, the second was strictly for manual load balancing and HA/FT. I wasn't aware FT can only use one vCPU and that's not going to cut it on at least half of the VM's. I also cannot do thick provisioning on 2 of the VM's so that's out for FT on those. I'd like to have FT for sure but looks like we're going to need to settle for HA. as CoW stated, not really worth FT on these considering the restrictions.
Click to expand...

Yes, I rarely suggest FT and rarer still see anyone using it except for the most business critical applications that absolutely can not go down.

One of our 3P software providers has sent me performance charts for the SAN we'll be using. It's a Dell NX3200 which is basically a PE R720 server chassis. He says they get better IO performance out of that with 12 7.2 NL-SAS than say a $40K Equilogic. I'll trade the 20TB capacity on 7.2's for 9TB on the 10K spindles. Charting our current IO, we'd infrequently peak at about 90% of the 7.2K setup IO capacity so I like the 10K IO padding. The only options for RAID are 5 or 6. So you'd rather see RAID 5 w/ hot spare?
Click to expand...

RAID 6 has to write two parity blocks for every write so the overhead is pretty significant. You could go RAID 5 with 11 disks but that's an awfully big RAID set to use RAID 5. A 900GB drive could take a couple hours for the hot spare to rebuild and during that time if you lost another drive for any reason you've just lost everything (hope you had backups!).

Granted, the odds of this are slim but as I tell my customers you don't want to be the guy who used a huge RAID 5 set and lost the entire thing during a rebuild. I typically don't like to use larger than 7 disks in RAID 5 but some vendors are supporting more. For example, EMC will now allow storage pools in 9 disk RAID 5 increments.

If your only options are RAID 5 and 6 then I guess go with your big RAID 6 idea. A big RAID 5 would be a tad risky but not unheard of if you wanted to go that way. But don't try RAID 5 without a hot spare. Or perhaps use 8 drives in RAID 5 and 4 in RAID 10 just for SQL plus a hot spare if that would work.
 
VM and it's capabilities are simply amazing but man oh man there are so many options and potential configurations it can get mind boggling.

Okay so I'm over FT. We already use other devices for HA/Backup and DR/backup of that HA on a remote site so if one of our servers die now, I can bring it back up in a VM running it's last shadow copy and be back online in as little time as it takes to spin the VM up. If that datacenter HA goes down, I can bring it up on the DR site. The VMware will take over the initial HA and our current devices will become a secondary HA/offsite DR so lots of redundancy there already. These are not absolute mission critical servers and 10 minutes or so of downtime for a VM is okay. After business hours and weekends are normal maintenance windows so FYI. Just thought hey if the capability for FT is there and it's part of the Standard Accel kit, why not use it but it's limitations have negated it's use.

So, FT out of the picture now, I'm still not sold on the 10Gb switch between the SAN and hosts. I have a lot of flexibility as far as NIC's in the hosts/SAN and my brain just isn't seeing the benefit of two dedicated 24 port switches that are only using 2 ports each ports to connect just the SAN. I mean, the vSwitches can do the same thing no? Seems like adding a switch in this setup is just asking for another piece to fail. If I had 2 x 2 x 10Gb NIC's in each host and SAN, you could direct connect ESX1 to the SAN by 10Gb sfp+ and failover with another 10Gb sfp+ from different NIC's on both host and SAN, then duplicate that from ESX2. 4 sfp+ cables and boom, done, no wasted 22 10Gb ports on a pSwitch x 2. Just seems in our environment that money would be better spent elsewhere. Then I can have additional 6-12 x 1Gb NIC's in each host for management and VM's. If ever a time comes we for some now unknown reason need to add an additional host to the same SAN we would pSwitch then, but that would be 3-4 years off anyway. This is how the aforementioned 3P vendor sets up their VM systems for many many bank clients. Instead of them buying the hardware for us as they typically do for clients, I am doing so so I can expand the specs to host more than their required 4. What am I missing?
 
RyteSyde said:
So, FT out of the picture now, I'm still not sold on the 10Gb switch between the SAN and hosts.
Click to expand...

Your NAS is going to present NFS datastores I take it? If you direct connect your two hosts to the NAS you're not going to be able to do HA, FT, or VMotion. Since you'll have 1 physical port on the NAS connected directly to host 1 and another directly connected to host 2, each of those physical ports on the NAS will have to have a different IP address.

However, if you mount the same NFS datastore on two hosts each with a different IP then that same NFS share won't be seen by the hosts as shared storage but as two separate datastores unique to each host. Then you won't be able to use HA, FT, or VMotion.

With a network switch between your hosts and NAS, then you can create an LACP aggregate or fail safe aggregate (assuming your NAS can do this) between your two physical NAS ports and the switch and that aggregate will have a single IP address. Then you're mounting the same datastore to each host with the same IP address and HA, FT, and VMotion will work.
 
You can direct connect the storage and still do HA/FT. You'd export the datastore out over both NICs/IPs and from the vSphere hosts you'd need to connect by a hostname that is the same on both hosts...not by IP since that would be different on each host. That's where it gets real ugly. (This assumes NFS.)

You don't need 10Gb. Just go 1Gb and get a switch. Do it the right way. People try to get clever and then pay me a lot of money to come back and do a design re-evaluation of their environment to get it right.
 
And to be honest, I'm not sure VMware would even support direct attached shared storage like this. If something odd starts happening that's a troubleshooting nightmare.
 
NetJunkie said:
You can direct connect the storage and still do HA/FT. You'd export the datastore out over both NICs/IPs and from the vSphere hosts you'd need to connect by a hostname that is the same on both hosts...not by IP since that would be different on each host. That's where it gets real ugly. (This assumes NFS.)
Click to expand...

Then you'd need some sort of /etc/hosts entry on each host so each is pointing to the same name but a different IP. And I don't even think editing that will persist across reboots in ESXi 5.1 will it? Ick. Definitely not something I would attempt in production.
 
So really direct attaching SAN is not recommended ?
We've got 2 Dells R720 connected to Netapp 2040 and all to HP 2 stacked 5500 switches - bad thing with this is that when 'we've got problems on network - iSCSI lose connection.
We use Hyper-V instead of ESXi
 
It just blows my mind that a 10Gb switch will cost more than the NAS itself. So what specific switches are you guys using? If I look at the Dell PowerConnects for example, it looks like I'd need the 8024 and it comes with an $18K price tag for one, plus the sfp modules/cables. Granted my cost would be about half of that but still...

http://configure.us.dell.com/dellst...l_id=powerconnect-8024&c=us&l=en&s=biz&cs=555

the frustrating thing for me is I do not have the time to do all of the research I feel comfortable in needing before I have to pull the trigger on this. I talk to the 3P provider who has a lot of hands on experience with setups like this and get one direction. Then my Dell reps give me something different. Then I come to guys like you that after reading convince me you know what you're doing and get even more different info. Between CoW and NJ you two just conflicted each other saying you can/can't vMotion/HA etc over direct connections and it makes my brain want to explode :) It's like trying to learn to work on a car engine without having a physical engine to mess with. I have as much of a lab environment setup as I can in the short time I have but it's not nearly what I would like.

In order to get the approval for this, one caveat was this needs to be done correctly now. I will not be able to come back in January and tell them we need another $10K switch etc. I have workstations, software licensing, and so many other parts to this project, the VM stuff is just a part of the budget I have to work with. Shoot, 60-70 scanners and thermal printers alone are taking $55K of that. In order to incorporate the switches, I have to modify other aspects of all of this so that's why I am pushing to understand if I really must use the switches.
 
RyteSyde said:
It just blows my mind that a 10Gb switch will cost more than the NAS itself. So what specific switches are you guys using? If I look at the Dell PowerConnects for example, it looks like I'd need the 8024 and it comes with an $18K price tag for one, plus the sfp modules/cables. Granted my cost would be about half of that but still...

http://configure.us.dell.com/dellst...l_id=powerconnect-8024&c=us&l=en&s=biz&cs=555

the frustrating thing for me is I do not have the time to do all of the research I feel comfortable in needing before I have to pull the trigger on this. I talk to the 3P provider who has a lot of hands on experience with setups like this and get one direction. Then my Dell reps give me something different. Then I come to guys like you that after reading convince me you know what you're doing and get even more different info. Between CoW and NJ you two just conflicted each other saying you can/can't vMotion/HA etc over direct connections and it makes my brain want to explode :) It's like trying to learn to work on a car engine without having a physical engine to mess with. I have as much of a lab environment setup as I can in the short time I have but it's not nearly what I would like.

In order to get the approval for this, one caveat was this needs to be done correctly now. I will not be able to come back in January and tell them we need another $10K switch etc. I have workstations, software licensing, and so many other parts to this project, the VM stuff is just a part of the budget I have to work with. Shoot, 60-70 scanners and thermal printers alone are taking $55K of that. In order to incorporate the switches, I have to modify other aspects of all of this so that's why I am pushing to understand if I really must use the switches.
Click to expand...

There's a way to make HA, FT, and VMotion work over direct connect but I wouldn't consider using it in a production environment nor would it be supported if you gave VMware a call for help.

NetJunkie is right though, why do you need 10Gb? I've set up bigger environments than yours with only 1Gb and it works perfectly well. Do you anticipate needing more than 125MB/s of read/write throughput to any of your VMs? I know 10Gb is sexy but if you don't truly need it, that money can be better spent elsewhere.

Perhaps the better investment would be to drop 10Gb altogether and use the extra money to get yourself a pair of 1Gb switches to connect your hosts and SAN to. 8x1Gb per host should work just fine.

2x1Gb per vSwitch
vSwitch0 - Management
vSwitch1 - VMotion
vSwitch2 - NFS
vSwitch3 - VM Traffic
 
oh I can go with 1Gb right now without anything extra as I've already got 1Gb, both L2 and L3 in place.

The 125MB/s to a VM is not the issue, it's the aggregate of multiple (10) VM's accessing the datastore at the same time.

Perhaps what I'll do is still spec the hosts with 4 x 10Gb and fill the rest with 1Gb's and see how it goes with that. They'll be happy with the lower cost but they will have to accept an additional purchase in the future if we need to jump to 10.

Can you aggregate/team more than 1 1Gb NIC to the NAS? I'd be a lot more comfortable with 2Gb. I know that's an entry level question but my mind is so cluster-mucked right now I'm not thinking straight. I need a good sleep to help clear the noise.
 
scratch that, you already answered it
2x1Gb per vSwitch
vSwitch0 - Management
vSwitch1 - VMotion
vSwitch2 - NFS
vSwitch3 - VM Traffic

see what I mean :)
 
You'll be amazed at how little actual throughput you need for storage with those VMs. The vast majority of vSphere deployments using iSCSI and NFS are *NOT* on 10Gb...instead they use a couple of 1Gb links and work just fine with a lot more load than what you are going to do.
 
So there's really no point in having more than 2 1Gb's in the VM traffic vSwitch correct? I'll have the capability but sounds like adding 12-16 x 1Gb is wasted overkill.

Here's a screencap of an aggregate perfmon report for 3 current main production servers.
1. 1003 Server - PDC, file server, exchange server, print server
Read Write ratio: 99%/1%
2. 2003 Server - terminal server
Read Write ratio: 60%/40%
3. 2003 Server - imaging server
Read Write ratio: 88%/12%


Just these 3 servers are going to be split into 9 VM's but the actual IO won't be much different, just moving to different guests.

This is a 24 hour period for a typical day, report starting at 16:30 or so, running through the backups, rest period and through the next day. Servers are currently all on 1Gb. The spikes around 21:00 are when the VSC/backups start to the HA device. I currently have them all set to all start VSC/backup at the same time to see it all at it's most intensive point and it doesn't look bad at all. Even if the backup is where I took the hit, not a big deal really. I can provide individual server charts for the same period if it helps but what say ye?

From what I'm seeing, it may even be better to go with 12 x 2TB 7.2K near-line SAS drives rather than the 12 x 900GB 10K's as well just for the significant storage capacity.

Also looks like 1Gb is looking a lot better than 10Gb eh

ts24dpak.gif
 
RyteSyde said:
oh I can go with 1Gb right now without anything extra as I've already got 1Gb, both L2 and L3 in place.

The 125MB/s to a VM is not the issue, it's the aggregate of multiple (10) VM's accessing the datastore at the same time.

Perhaps what I'll do is still spec the hosts with 4 x 10Gb and fill the rest with 1Gb's and see how it goes with that. They'll be happy with the lower cost but they will have to accept an additional purchase in the future if we need to jump to 10.

Can you aggregate/team more than 1 1Gb NIC to the NAS? I'd be a lot more comfortable with 2Gb. I know that's an entry level question but my mind is so cluster-mucked right now I'm not thinking straight. I need a good sleep to help clear the noise.
Click to expand...

Simply adding the two NICs to the NFS vSwitch won't give more throughput though. All you're ever going to get is 125MB/s max to each datastore from each host. There are ways to get your host to use both NICs for connecting to NFS but the throughput you'll get to each datastore is a single 1Gb link.

However, you'd be surprised that this ends up being just fine for your 10 VMs. Trust us.

Skip the 10Gb adapters entirely and put that money towards a better SAN. From what I see the Dell NX3200 is going to run you $17k list. Frankly there are better options out there for that price point. I have experience with the EMC VNXe but can't recommend it. I believe the HP Lefthand P4300 and Netapp FAS2220 fall in that price range. Better still, they support VAAI and can be scaled out as you grow.
 
RyteSyde said:
So there's really no point in having more than 2 1Gb's in the VM traffic vSwitch correct? I'll have the capability but sounds like adding 12-16 x 1Gb is wasted overkill.

Here's a screencap of an aggregate perfmon

<snip>
Click to expand...

Yeah, that's nothing. 1Gb is going to be fine.
 
I was originally looking into the EqualLogic but my 3P contact swears by the NX3200, has put many of them out and sent me the following from his 3200 in his lab.

You're pretty close on list, cost to me is a bit over $10

Here is some performance numbers for it

3200stats.gif


So since I'm saving a buttload from the 10Gb, how do I take care of the SPOF with 1 NAS? I mean I already have two HA/DR devices but I'd like to know how I can do it with VMware. Don't want to go to enterprise accel kit just for sdrs
 
RyteSyde said:
I was originally looking into the EqualLogic but my 3P contact swears by the NX3200, has put many of them out and sent me the following from his 3200 in his lab.

You're pretty close on list, cost to me is a bit over $10

Here is some performance numbers for it

3200stats.gif


So since I'm saving a buttload from the 10Gb, how do I take care of the SPOF with 1 NAS? I mean I already have two HA/DR devices but I'd like to know how I can do it with VMware. Don't want to go to enterprise accel kit just for sdrs
Click to expand...

First up, decide how you're going to back up your VMs. VMware offers their own appliance called VMware Data Protection but it's probably not a good option for your AD, SQL, and Exchange VMs because it's not application aware. Veeam Backup and Recovery is a good product and I like using it plus it seems to play better with backing up the aforementioned applications. You can also have it backup to a CIFS share off your SAN where VDP has to be on a datastore.

To eliminate the SAN as a SPOF you're going to either want to get a SAN with dual controllers (Netapp, EMC, etc.) or a clustered SAN solution like HP Lefthand which can spread your datastores across nodes.

So long as you have a good backup solution in place and your SAN has redundant controllers or is cluster based plus you have redundant power circuits to connect them to you're going to be in pretty good shape.

If the SAN blows up in your face obviously you're going to have extended downtime but with good backups at least you can restore everything when you get it back up and running. Beyond that you'd have to look at SAN to SAN replication if you want more protection.
 
Since I won't be using FT, I'm not going to be able to SLA 0 downtime so the SAN becomes less of an issue. As stated earlier, we have two HA/DR devices that are application aware and backup via VSC. I can bring any of the servers online in a matter of minutes in the HA VM (or DR VM if the HA blows too - say earthquake at HA site) and users are none the wiser outside of downtime. It's pretty cool as it'll continue to backup it's own VM so users can still change files etc. When the hardware is fixed I just perform one more self shadow of the HA's VM then restore it to the server. I can create a network share directly from the HA or DR device directly to any of the backups and recover files down to individual mail items in Exchange or restore entire volumes retaining all meta data etc. As long as the VM guest sees the data as local, it'll work just fine with VMw and as far as pure data protection, no absolute need to backup the VM datastore itself. I like redundancy, a lot, so anything else is more icing on the cake. Still a bit to do and will look into clustering the 3200...not sure if I can. I know it only has 1 controller though :(

Choices choices
 
These guys are giving you some good technical advice, so I just want to give you the step back thinking statements...

First, you are acting like this is a much bigger build out than it is. In reality you have a fairly straightforward and small setup you are virtualizing. Dont take this as a bad thing, take it for what it is. Follow all of the best practices for setting it up, even if that is not the latest and greatest technologies (ex. 10gb), use what is tried and true and you will be so much better off in the long run. I run some cobbled together setups at home that are great for labs and are very effective, but I would never run a 24/7 business off of them.

Second, re-think your SLA. There is ALWAYS a chance of a failure or problem, ALWAYS. Redundancy options just each reduce that risk by a little bit. The best thing to do is again follow best practices and put in redundancy where it really counts, some of the redundancy really gets to be overkill and you have other weaker points in installations. You can sit there all day trying to find failure points, and then the roof will cave in and take out your power, hosts, and SAN in one fellow swoop.

I give more credit to businesses that think of SLAs from the actual application impact perspective, for example, if exchange goes down the issue beyond users being able to use email is potential lost email that is floating around in the ether waiting to be delivered. Our email SLAs take this into account and we have RTOs that get us up and running in time to not lose any mail.

Trust me I get what its like to work in a business that does not want to have any downtime, but in reality people can deal with it for a little bit while you spin up another VM or do a quick restore. The trick is having flexibility and plans in place.
 
Grentz you are quite right. This is much bigger than my lab here but I also realize this setup is quite the entry level VM environment for production and you guys are probably accustomed to dealing with much grander setups. Just trying to get my ducks in a row :)

So, how's everyone feeling about this setup?
VMNetwork001.gif
 
Looks good. Would be even better if you could spread your network connections across redundant switches to eliminate them as a SPOF. If you can't do that, then this is good.
 
Agreed, see if you can separate out those redundant pairs across those switches.
 
I think this should cover the SAN yes?

As far as the 48p L3 Gb Switch, I may end up just having a spare on standby, ready with the loaded config. That's the plan for the other 5 24p L3's; to have saved configs for each site and load it to the spare and deliver if needed. If I put in a second (just below the SAN in diagram) I can failover the ESX/SAN to the site users but I'd have to manually switch the ELAN port and the "Misc 1 Gb" devices. If we'll physically need to change patches, probably easier and less confusing just to have a standby and do them all. Thoughts?



VMnetwork002.gif
 
a bit more info...

I had a discussion with the aforementioned 3P vendor about his recommended 10Gb vs y'alls recommendation of 1Gb. I sent his the same aggregate perfmon charts I posted earlier and this was his nutshell response.

"I have seen a number of systems with very fast drives and very little throughput which offer very poor performance. 1GB ISCSI is not much easier than 10Gb ISCSI but the difference is speed is night and day. With 1GB ISCSI you can try and squeeze as much bandwidth as possible but it is a clear bottleneck."

I've taken what you guys have said to heart and I agree that having a 10 lane highway is nice and all but if you're the only car, you only need one lane :)

So to put this into practice I first tested the current network. We have no speed complaints or obvious bottlenecks as is. I first ran an iperf from our oldest server to our imaging server as the host. They are currently connected via 1Gb. The results:

iperf.gif


So I'm confirming a bit over 100MB/s potential /s

Then I ran a sqlio from the same old server to a mapped 1GB dat on the same imaging server which is 2TB, 7.2K RAID 5 (4 spindle [vs our upcoming 12 spindle NAS]). The results:

sqlio.gif


That's about 68% of the iperf bandwidth.

If we refer to the aggregate perfmon charts I posted earlier, average usage (outside of the backup spikes at 9pm) is what, 10-20MB/s, so I don't see where the bottleneck will be in actuality.

Am I missing something or is he just being a 10gb fanboy?
 
10Gb equipment is expensive so I'm sure they want you to spend the extra money. If money weren't an option then I'd tell you to go ahead with 10Gb. It certainly won't hurt. What NetJunkie and I are saying, and you seem to understand, is that 10Gb isn't necessary.

With a NFS datastore you've got 125MB/s as your theoretical maximum throughput over a single 1Gb link. With a bonded interface on the NAS assigned two IPs and two links on each host, you'll be load balancing your total NFS traffic across two 1Gb links. So you'll get a combined 250MB/s theoretical maximum throughput to your NAS, just that a single TCP stream cannot exceed 125MB/s.

That combined bandwidth is what's important, not the speed of a single link. Except under certain circumstances, you'll find your servers typically (and your graphs show this) hum along using only a few MB/s. Even when you multiply that by 10 servers, you're still seeing very little actual average disk throughput.

Sure, there may be peaks once in a while where having more than 125MB/s burst available would be nice but how often does that really happen? Unless it causes you to fail to meet your SLAs with your business units, it shouldn't be an issue.

But here's the beauty of your virtualization journey -- all your servers and the applications that run on them are now portable. The hardware they run on is merely a commodity now; a vessel used to provide the performance, availability, and manageability you deem your applications need.

1Gb ends up being too slow? No big deal. Get a 10Gb switch and some 10Gb adapters, configure the switch, put a host in maintenance mode, install the 10Gb adapter, cable it up, configure it in VMware, test it, repeat on host 2, and VIOLA! You've got 10Gb with zero downtime to your applications.

Find your SAN is too slow and can't provide the IOPs you need? Install a new one and live Storage VMotion your guests to it with no downtime at all.

In short, yes, I think he's being a 10Gb fanboy. Just by going virtual you're opening up a lot of possibilities for yourself and if you find 1Gb doesn't cut it, it's not hard to move up to 10Gb. :)

If you've got a tight budget 1Gb will work fine to start with. Work with your management to ensure future budgets allow for expansion into 10Gb ethernet down the road because you're going to find your 10 VMs will turn into 20 and then 30 and then 40 before you know it.
 
Thank you for the excellent response CoW. I'm ready to pull the trigger on this and get this vm environment up and running!
 
Good thread and excellent final explanation by CoW.

RyteSyde, what is your final config out of interest?

Thanks,

G.
 
gabi, pretty much as depicted in the last illustration. I'll have 10Gb NIC's in the hosts/SAN but will use as 1Gb. I'm sure I'll direct connect via 10Gb before we go into production just to get some benchmarking and put some theory into practice but we're holding off on purchasing 10Gb switches until if/when they are needed. For now production will be switched 1Gb. I see no reason not to trust the experience of NJ and CoW on this.
 
You must log in or register to reply here.
Back
Top