• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

vSphere Multipathing Issue?

P

Paladin21

Gawd
Joined
Jun 22, 2004
Messages
529
I'm trying to squeeze some more performance out of my DroboElite iSCSI system that I am currently using. I have two Dell R410's w/4 nics hooked to a ProCurve 2824 as the host systems. The Drobo has 2 gigabit ports hooked in as well. Each server has 3 vSwitches set up. Two of these have one nic each that are on a vLan with one of th Drobo ports. The other vSwitch is set up with the other two nics and used for console and VM operations. All ports/vSwitches are set up for Jumbo Frames and pathing is set to Round Robin.

The problem I am having is that even though each server has two independent paths to the drive array, I only get the performance of a single port. Transfers to the individual VM's (made with a variety of tests such as HDTach in Windows and dd in Linux) top out at ~120MB/s. The switch only shows <50% utilization on either switch port. If I set the paths to MRU or Fixed instead of RR, port utilization goes up to ~90% (on one port and disappears on the other), and transfer speed stays more-or-less the same.

Is there some way to actually utilize two links worth of bandwidth that I'm missing?
 
I've read through both that one and the original one from a while back, but it doesn't address my issue. I have multi-path setup on the hosts, and can observe in the switch that it is sending packets down both pathways when set to RR pathing. My problem is that I can either disable multi-path and get one gigabit link worth of data across, or enable it and get two 500 mb/s links (performance wise). It seems like when I go multipath, it just farms one link worth of traffic across two links instead of actually using the full bandwidth of both links.
 
I started testing with one just to do the performance testing, I'm now up to 6 with no change in behavior. The Drobo itself has 8 WD1501FASS drives in it and is set to single-drive redundancy. I tried dual-drive redundancy out of curiosity a couple of weeks ago and noticed no significant change in performance (other than space available).
 
Paladin21 said:
I'm trying to squeeze some more performance out of my DroboElite iSCSI system that I am currently using. I have two Dell R410's w/4 nics hooked to a ProCurve 2824 as the host systems. The Drobo has 2 gigabit ports hooked in as well. Each server has 3 vSwitches set up. Two of these have one nic each that are on a vLan with one of th Drobo ports. The other vSwitch is set up with the other two nics and used for console and VM operations. All ports/vSwitches are set up for Jumbo Frames and pathing is set to Round Robin.

The problem I am having is that even though each server has two independent paths to the drive array, I only get the performance of a single port. Transfers to the individual VM's (made with a variety of tests such as HDTach in Windows and dd in Linux) top out at ~120MB/s. The switch only shows <50% utilization on either switch port. If I set the paths to MRU or Fixed instead of RR, port utilization goes up to ~90% (on one port and disappears on the other), and transfer speed stays more-or-less the same.

Is there some way to actually utilize two links worth of bandwidth that I'm missing?
Click to expand...

No. ESX does not do MPIO.
 
Okay, I guess I totally missed the point of having multiple paths if no combination of paths is significantly faster than a single one. In any case, I guess the next plan would be something like make a *bunch* of targets on the array and split the VM's in between them so that each individual target has a chance to get its own session. The current setup is arranged logically with 20-25 VMs in each target (arranged by class, at a University). If I split the array such that there are only 4-5 VMs on each target, would each individual target get the ~120mb/s?
 
Paladin21 said:
Okay, I guess I totally missed the point of having multiple paths if no combination of paths is significantly faster than a single one. In any case, I guess the next plan would be something like make a *bunch* of targets on the array and split the VM's in between them so that each individual target has a chance to get its own session. The current setup is arranged logically with 20-25 VMs in each target (arranged by class, at a University). If I split the array such that there are only 4-5 VMs on each target, would each individual target get the ~120mb/s?
Click to expand...

Failover, and just as you surmised here, load balancing/port balancing.

It's about shared resources - once you get a few servers going, interconnect speeds will ~not~ be the limitation - disks will be. :) 4-5 servers using 1gbE links can easily overload several SAS disks, let alone SATA - you won't be concerned with link speed then, only with disk speed.

Round Robin is there for port contention / balancing automatically every 1000 iops. ESX currently can only use one path at a time per lun, so if you have more luns (or targets) you can divide the paths they're using much like you suggest, and utilize the links. But you'll find again that you run out of disk speed once you get things up well before you run out of link speed.
 
Yep. That's why I asked about targets. Multiple targets using multiple datastores across them. It's a bit of manual load balancing, but it'll spread the throughput over multiple links. Also like lopoetve said, you'll hit IOP contention before throughput contention, in most cases, with ESX and VMs.
 
Wasn't there a load balancing technique you can initiate on a Cisco switch to balance traffic going to the host while round robin balances traffic from the host? Does that do anything for speed?
 
Child of Wonder said:
Wasn't there a load balancing technique you can initiate on a Cisco switch to balance traffic going to the host while round robin balances traffic from the host? Does that do anything for speed?
Click to expand...

Not as far as ESX is concerned, from what I know.
 
So I made some more iSCSI targets and moved things around so that any one target was only running 4-6 VM's at most, and did some more testing. Now, the performance on my Linux VM's is great. The system will happily run 24 OpenSUSE VM's with them installing/compiling Apache and other similar tasks without a problem.

HOWEVER, when I try to start up the Windows XP VM's, all hell breaks loose. While I can allow a full class of students to use Linux, even a *very* small amount of Windows VM's (5-6 total) causes all of the instances to lag out and basically breaks iSCSI. I've gone through and turned off all the services that can be safely turned off, disabled AV software, turned of System Restore, killed all themes, and reduced startup programs to nothing but the VM Tools entries. The only other thing that I could think of would be to change to disk system from IDE to SCSI (as the Linux ones use SCSI). I've looked around the web and haven't had any luck in digging up any comparision benchmarks with a recent setup. Has anyone done any testing on this? I'm going to set up a few more VM's to do some preliminary work on, but I'm not starting in on converting ~100 XP VM's over if it isn't going to help.

Thanks!
 
Install tools, add a scsi disk (itty bitty) to the XP vm so that the driver is installed, and then switch the boot disk to scsi and remove the second, see if that helps.

However, I'll be honest - the Drobo is not exactly a powerful array, and windows generates plenty of extraneous IO. What RAID level are you using with the SATA disks it comes with?
 
It's thier "BeyondRAID" stuff. I don't have any real details on it because their advertising point is that you don't have to manage it (so they make it where you can't, haha). I'm just stumped that the array can handle something like 20+ OpenSUSE VM's actively compiling Apache at the same time without an issue, but at the same time only 4-5 Windows VM's causes it to completely run out of performance when doing nothing other than booting up to the desktop.

I should add that the Drobo was never my first choice (I wanted to build an OpenFiler system for about the same price with 20 spindles and a much better controller), but I was required to obtain a certified solution. The Drobo was the only one in our budget range for the project at all, so comparison shopping wasn't exactly an option.

I read an article on swapping existing IDE drives to SCSI, and the steps involved generating duplicate disks and using the geometry information to re-write part of the configuration manually. Is that actually necessary, or can you just add the controller, install the driver, change ide to scsi in the configuration file and call it a day?
 
Are you booting all 5 VMs at the same time?

You ~should~ be able to swap controllers. One moment, verifying.
 
I've been restricting it to booting one at a time due to the performance constraints. However, after the ~5th one or so comes up, when starting another one all of the previously started VM's become non-responsive until the new one is up and running. I've killed everything I can think of that's causing disk access and it's only marginally better than it was before.
 
ok, cancel that. Havent' tinkered with XP on ESX in a while. looks like we pulled the drivers totally from the base image. You can get them from LSI (they're the LSI20320-R - -you want the driver prior to 1.28.03.
 
Paladin21 said:
I've been restricting it to booting one at a time due to the performance constraints. However, after the ~5th one or so comes up, when starting another one all of the previously started VM's become non-responsive until the new one is up and running. I've killed everything I can think of that's causing disk access and it's only marginally better than it was before.
Click to expand...

Look at esxtop - under d for disk, see how many commands a second you're sending to the array.

What happens if you ahve the OF vms down, and then boot XP?
 
Not that this helps the OP at all, but doesn't PowerPath in ESX 4.x offer true MPIO as well? I haven't looked into it, but I have a feeling.
 
Kachunkachunk said:
Not that this helps the OP at all, but doesn't PowerPath in ESX 4.x offer true MPIO as well? I haven't looked into it, but I have a feeling.
Click to expand...

very very close. It's an intelligent algorithm that starts at the equivalent of RR with IOPS=1, but is intelligent and capable of adjusting to certain conditions. :)
 
You must log in or register to reply here.
Back
Top