Slow Read Speed - Storage, ESXi, or Network Issue

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
Brief details of setup and description of issue:

Single ESXi server running 5.1u1 with Intel X520-DA2 10Gb NIC
Two Arista 7050s 10gb switches
NetApp FAS8040 with hybrid shelf (SSD for Flash Pool and 10k SAS)

I setup the ESXi server over the weekend with 2x10Gb Twinax cables to the Arista switches using “Route based on physical NIC load” for load balancing. NetApp is configured using static multimode vif with 4x10Gb connections going to the Arista switches with Twinax. Two connections per head. Created a couple of LUNs and mounted to ESXi server using iSCSI. Doing a fresh install of Windows Server 2012 R2 took less than 4 minutes. Used Parkdale for a disk speed test. Sequential write was 372 MBps and sequential read was 653 MBps. This was last night. Did speed test this morning and write is about the same but read dropped to 72 MBps and is very inconsistent. As in while it is doing a sequential read, the MBps jumps around a lot. Last night it was smooth and consistent.

WTF, right? Nothing changed.

Now for the backstory:

The NetApp was originally shipped to CA so it could be setup to seed a 24 TB SnapMirror from our other NetApp. Day 1 of the install, Head1a had a bad motherboard and needed to be replaced. Once replaced and configured, seeding occurred without any issues. Seeding was through Head1b as that is where the SATA shelves are located. No real testing was done on Head1a as far as storage while in CA. This was a few months ago. After the seeding, the NetApp was shipped to NJ and setup, but only with 1Gb connectivity to keep the SnapMirror running.

A couple of weeks ago we received the Arista 10Gb switches and installed them in the cab next to the NetApp. Had to order some optics and Twinax, plus we didn’t have an ESXi server yet to setup. So it wasn’t until last Friday that I got to finally get things going.

On Friday we connected the two switches together with 4x10Gb Twinax in LAG. I brought an ESXi server from another datacenter to this one. The server was already running ESXi 5.1u1. I installed brand new Intel X520-DA2 NICs. These are two port NICs and I installed three of them for a total of 6x10Gb. The first thing I wanted to do was get the storage networking going so I could test out our NetApp. Had some issues with the LLDP on the Arista. Not sure if this is always the case, but it seems VMware doesn’t read any of the LLDP information until you actually add the NICs to a port group. Even the Arista didn’t learn the MAC addresses until I configured all the Distributed Switches and ports. Because I thought LLDP wasn't reading properly, I went to VMware’s website and downloaded the Intel drivers for that version of ESXi.

This was the install:

Code: 
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
     Reboot Required: true
     VIBs Installed: Intel_bootbank_net-ixgbe_3.21.5-1OEM.500.0.0.472560
     VIBs Removed: VMware_bootbank_net-ixgbe_3.7.13.6iov-10vmw.510.1.12.1065491
I am not sure if this makes a difference at all, but just putting it out there as a variable. I setup two dvPortGroups for the storage. Each dvPortGroup has one Active uplink and the other set to Unused. For Load Balancing I went with “Route based on physical NIC load”.

For the NetApp I setup two 2 TB LUNs on our hybrid shelf and a 200 GB LUN on our SATA shelf just to store ISO’s, etc. I installed a iSCSI Software Adapter on the ESXi server. Added the IP addresses for both vifs from the NetApp, rescanned and added the LUNs. Also enabled Storage I/O Control with default settings. I changed the Path Selection for all LUNs to Round Robin.

Now that storage was setup, I installed Windows Server 2012 R2 and did some disk speed tests to see how fast our new storage is. Everything seemed good and called it a night. That part was last night. This morning I did another disk speed test and that is when I saw the read test jumping around and running at under 100 MBps. Although both 2 TB LUNs are on the same shelf, I migrated the VM to the other 2 TB LUN to test again. Same results, it was reading slow.

For the hell of it, I decided to migrate the VM to the SATA storage on Head1b. During the migration, the LUN on Head1a disconnected briefly and crashed the VM. After it re-connected I did the migration again, this time successful. Powered up the VM on the SATA shelf and ran the same disk speed test. Write was good at 309 MBps, but read was also jumping around and inconsistent and ended at 65 MBps.

Being that the issue is on two different shelves and two different heads, I don’t think it would have anything to do with the motherboard that was replaced in Head1a. I should mention, both shelves are actually hybrid and have SSD Flash Pool. The Flash Pool is 558 GB in size and is used for reads and writes. Not sure it has anything to do with the issue either.

Unfortunately I only have the one ESXi server there right now, so I can’t test on another server. I was going to upgrade to ESXi 5.5 and test again. I am kind of learning towards maybe an iSCSI setting. Not sure what are the recommendations for NetApp. But still weird that it was working fine last night and today not so much.

Any ideas what may be causing the issue or what I can look for?
 
Just to add to it, I also installed the NetApp VSC and used that to set the recommended values. Everything in the overview shows as normal. Status, Adapter Settings, MPIO, etc. But still has the same read issue.
 
I was planning on upgrading this server to ESXi 5.5, but instead I will do a clean install of 5.5 and see if it makes a difference. Unless someone else has an idea of what this might be?
 
lopoetve said:
did you bind the iSCSI nics?
Click to expand...

Yes. I didn't think you could even add an iSCSI LUN without binding the NICs. Both NICs are bound and compliant. Round Robin is setup, all paths are Active.

Is it fine to use “Route based on physical NIC load” for the load balancing?
 
If you don't, it'll just do its damndest to "find" a way to the array. :)

For iSCSI to be done properly, you either need two separate switches each with a single nic in it, or each vmkernel port set to use only one nic (and the other set to unused), either of which makes the load balancing on the network side moot.
 
lopoetve said:
If you don't, it'll just do its damndest to "find" a way to the array. :)

For iSCSI to be done properly, you either need two separate switches each with a single nic in it, or each vmkernel port set to use only one nic (and the other set to unused), either of which makes the load balancing on the network side moot.
Click to expand...

Yup, that is how I have it setup. I am going to try the new ESXi 5.5u2 that just came out. If there is still an issue, I guess I will setup a different server and test with that.
 
I could really use some help on this one if someone knows the proper config.

I have narrowed it down to the network setup on the NetApp. That part I am not overly familiar with. Here is the whole network setup for the storage network.

Server1 > Switch1 - VLAN 500
Server1 > Switch2 - VLAN 500

Head1a > Switch1 - VLAN 500
Head1a > Switch2 - VLAN 500
Head1b > Switch1 - VLAN 500
Head1b > Switch2 - VLAN 500

Switch1 > Switch2 - 4x10Gb connections in LAG - VLAN 500

So with this setup everything should be redundant. The ESXi host is setup correctly and multi-pathing is working. When I migrate a VM to a datastore connected to a different Head, that is when connectivity to the datastore drops. If I disable both ports on say Switch1 for the NetApp, then it works fine. Multi-pathing still even works. The one vmnic goes over the port channel to Switch2 and back to the NetApp. No packet loss. I tried it in reverse and it also works fine. So all the ports, cables, and NICs are working fine.

It only doesn't work when the NetApp is connected to both switches. I even disabled the LAG between the two switches to see if that was causing a problem, but it didn't help.

The NetApp is setup with a Static MultiMode VIF. (I didn't configure this part.)

This article is good. https://communities.netapp.com/blogs/ethernetstorageguy/2009/04/04/multimode-vif-survival-guide

What I don't understand is that it seems you DO need to setup port-channel when using Static MultiMode VIF. I was under the impression that LAG was not used with this setup. We actually had an issue at one of our sites after a NetApp was upgraded to 8.2.1. The iSCSI connections kept dropping like I am seeing now. The engineer that worked with NetApp ended up removing the LAG and using the Static MultiMode VIF like what we have on this NetApp. That resolved the issue at that site.

So what is the correct setup? Being that our switches aren't stacked, we can't even LAG each Head as they only have a single connection to each switch.
 
OK, now I am mad. The person here that fixed the issue with NetApp at our other location that setup Static MultiMode said that the ports were not setup with port-channel. I just logged into that switch and checked the config and look what I see. Two port-channel groups. One for head1 and another for head2. Don't ask why they are all on one switch. :mad:

Well, supposedly the Arista switches have MLAG so you can use port-channel across multiple switches. I guess I will need to configure that tomorrow. Just out of curiosity, is there a way to setup a NetApp with multiple connections and not use LAG and it actually works?

Second, since I will be setting up port-channel, does that mean I should change the load balancing from “Route based on physical NIC load” to "IP Hash"?


Code: 
1/1/33  Up      Forward    Full 1G    33    Yes N/A  0   748e.f87f.c120  head1
1/1/34  Up      Forward    Full 1G    33    Yes N/A  0   748e.f87f.c120  head1
1/1/35  Up      Forward    Full 1G    33    Yes N/A  0   748e.f87f.c120  head1
1/1/36  Up      Forward    Full 100M  None  No  900  0   748e.f87f.c123  head2
1/1/37  Up      Forward    Full 1G    37    Yes N/A  0   748e.f87f.c124  head2
1/1/38  Up      Forward    Full 1G    37    Yes N/A  0   748e.f87f.c124  head2
1/1/39  Up      Forward    Full 1G    37    Yes N/A  0   748e.f87f.c124  head2
 
technically, load balancing doesn't matter - with each nic bound to a single vmkernel port, that's going to get ignored entirely - the SCSI layer handles it instead.
 
I can't seem to get the port channel working on these Arista switches. I was able to setup the MLAG between them successully, but not the port channel for each head.

Is there a way to setup the NetApp without port channel?
 
Finally got it setup. Never used this MLAG stuff before. Basically had to create port channel groups and bind them to the mlag. So far everything appears to be working good now.
 
You must log in or register to reply here.
Back
Top