So, we were approved to update one of our locations next year to 10GbE. Is two 10GbE NICs per ESXi host adequate? One for production and the other for storage? We will still be using 1Gb for management and our NetApp is setup using NFS. We have about 20 ESXi 5.1 hosts setup in a cluster using DRS.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
10GbE Configuration with ESXi and NetApp
- Thread starter KapsZ28
- Start date
I configure them with 2 10GbE and then another 2 1Gb for management redundancy, you should be fine with your config.
On thing I noticed with the NetApp, we set each head up with an IP for NFS and an IP for iSCSI on different storage vlans, the hosts would not connect if I didn't have vmkernels on both vlans. So even though some hosts only needed NFS, if I didn't put a vmkernel on the iSCSI vlan as well, it wouldn't let me mount any NFS mounts.
On thing I noticed with the NetApp, we set each head up with an IP for NFS and an IP for iSCSI on different storage vlans, the hosts would not connect if I didn't have vmkernels on both vlans. So even though some hosts only needed NFS, if I didn't put a vmkernel on the iSCSI vlan as well, it wouldn't let me mount any NFS mounts.
- Joined
- Sep 14, 2004
- Messages
- 14,512
When you say 2 10GbE nics, do you mean 4 ports total? or just 1 NIC with 2 ports?
We run 2x NICs with 4 ports total on all hosts and on each netapp head unit. We have them in a LAG/LACP with Jumbo Frames enabled. We are also all NFS to our esxi hosts. It has worked fairly well minus a few Ontap bugs we hit being early adopters on some technology. We are also running esxi 5.5 on our production clusters as well.
We configured our storage IPs to be non routable so that only the esxi hosts will talk to the netapp on the backend and nothign else will interfere.
Biggest got ya we had with this setup was making sure FlowControl was completely turned off on the switches and ESXi hosts. The Netapp(ours at least) has a qlogic CNA that you cant enable/disable Flowcontrol but as long as the other end is set you should be good.
We experienced microbursting issues and pause frames that would cause cascading latency issues and start crashing VMs.
We run 2x NICs with 4 ports total on all hosts and on each netapp head unit. We have them in a LAG/LACP with Jumbo Frames enabled. We are also all NFS to our esxi hosts. It has worked fairly well minus a few Ontap bugs we hit being early adopters on some technology. We are also running esxi 5.5 on our production clusters as well.
We configured our storage IPs to be non routable so that only the esxi hosts will talk to the netapp on the backend and nothign else will interfere.
Biggest got ya we had with this setup was making sure FlowControl was completely turned off on the switches and ESXi hosts. The Netapp(ours at least) has a qlogic CNA that you cant enable/disable Flowcontrol but as long as the other end is set you should be good.
We experienced microbursting issues and pause frames that would cause cascading latency issues and start crashing VMs.
So are you running both NFS and iSCSI through a single 10GbE connection from your NetApp and using the second 10GbE for Production traffic?
Nothing has been ordered yet, but I would say a single NIC with two 10GbE ports on it. Although it may be safer to get two single port 10GbE NICs.
Child of Wonder
2[H]4U
- Joined
- May 22, 2006
- Messages
- 3,270
I would. A single 2x10Gb port NIC would be a single point of failure.
Do you have Enterprise+ licensing? If so, make sure you configure NIOC since you'll have so many types of traffic sharing those 2x10Gb ports (NFS, iSCSI, VMs, VMotion).
We're using Nexus 1000v virtual switches, so both nics are in the uplink which has basically everything using it except management traffic.
Don't split them physically like that or you'll obviously lose redundancy. If you're worried about overrunning bandwidth or causing I/O starvation use NIOC (if you have Ent+ licensing).
So, you are saying stick with a single two port 10GbE nic?
We have enterprise plus. I guess I will start looking into NIOC.
He's saying don't split them to only perform one a singular function. Use one as a standby for the other. Your comment sounds like you were going to use one for each function.
Yeah, Exchange was the only item I noticed that is supposed to be on iSCSI, but we don't have many Exchange servers. Anything else that we would really need to use iSCSI for?
Yeah..you can do NFS just not supported yet.
And thanks. Mod here and in the Networking forum.