Recommend my ESX Setup

R

RiDDLeRThC

2[H]4U
Joined
Jun 13, 2002
Messages
3,963
Ok so heres the deal, we have two ESX Servers both have two network cards for iSCSI, one iSCSI Array with one dual-port controller card, 2 network switches.

What would be the best way to build out the iscsi network? two subnets one for each switch? Should I do nic teaming on the ESX server?
 
one switch for regular "production" traffic, one for the iSCSI array. are you hoping to boot from SAN on the iSCSI array? which array are we talking about, here? do you plan to use the hardware iSCSI interfaces only for the iSCSI access, or will you also be using software initiators built into ESX?
 
ooh. I'll get to this tomorrow. I'm gonna have fun here, cause I found a good way to do this :D Yay load balancing!

What SAN, just so I know?
 
We already have a seperate network for normal private and public traffic.

Each ESX Box has two onboard ports (Private/Public vSwitches) then have TWO additional dual port nic cards (6 ports total per server) 4 are usable for iscsi.

I planned on running iSCSI/VMotion traffic across 2 of the 4 ports and just leaving the other two offline. We don't VMotion often at all so its traffic isn't too big of a deal.

right now theres a service network also setup on the iscsi switch. also one setup on our private network switch.

I have a total of 4 switches that can be used
one dell web managed 24 port (Powerconnect 2724) this has VLAN's setup on it for the public network traffic.
two dell L2 switches 24 ports (Powerconnect 5424)

Array is an MD3000i
 
Your the master, I'm just the puppet :) Hit me up when you get a chance tomorrow.

Right now the servers are in production just only using one network card and one switch. the other switch is configured and racked just waiting.
 
ok. the basic concept here is that you are going to be creating two "iscsi HBAs"

Each HBA is formed from a service console port and a vmkernel port.
IPs must be on different subnets.

For instance -
vSwitch1 =
COS 192.168.153.25
vmkernel 192.168.153.26

vSwitch2 =
COS 192.168.154.25
vmkernel 192.168.154.26

Capiche? :)

on the MD3000, you need to configure one set of ports on the first subnet, and one set on the second. You CANNOT use the group IP with this configuration that I know if.

For networking, you need each port on the host connected to the right ports on the san, through the switches, with ZERO crossover between switches.


In VC, enable iSCSI, and set dynamic discovery to all portals on the SAN.

Rescan. Congrats, you should see 2 paths to your luns now. You can manually balance paths to balance load through the gui, and as of U4 we will restore that config on boot :)

You will get normal, FC-esque failover, where we show dead paths and everything.
 
lopoetve said:
ok. the basic concept here is that you are going to be creating two "iscsi HBAs"

Each HBA is formed from a service console port and a vmkernel port.
IPs must be on different subnets.

For instance -
vSwitch1 =
COS 192.168.153.25
vmkernel 192.168.153.26

vSwitch2 =
COS 192.168.154.25
vmkernel 192.168.154.26

Capiche? :)

on the MD3000, you need to configure one set of ports on the first subnet, and one set on the second. You CANNOT use the group IP with this configuration that I know if.

For networking, you need each port on the host connected to the right ports on the san, through the switches, with ZERO crossover between switches.


In VC, enable iSCSI, and set dynamic discovery to all portals on the SAN.

Rescan. Congrats, you should see 2 paths to your luns now. You can manually balance paths to balance load through the gui, and as of U4 we will restore that config on boot :)

You will get normal, FC-esque failover, where we show dead paths and everything.
Click to expand...

All sounds good, think i will enable this in the morning when i get in.

Only difference is that my MD3000i only has one controller card so only a total of 2 ports.

That won't change things though too much? just one each port on a different subnet.
 
Last edited:
Correct :)

Note - this is a significant change, you'll need it all down to switch to this config :)
 
oh yeah - and once done, then you need to balance the paths. Go in to the properties page of the datastore and set differing preferred paths.
 
Alright looks like everything works correctly. Here are some screenshots of the config.

iscsi01.PNG

iscsi02.PNG

iscsi03.PNG

iscsi04.PNG

iscsi05.PNG
 
Ta da!

now test failover - fire up a test vm, on a lun with a path, and then yank the cable. Just to make sure :)
 
yep i actually took it one step further and powered down an entire battery backup killing one of the everything in every server/switch/storage array.

test went well. my 2k8 install didn't miss a beat during it.

Thanks so much lopoetve.
 
lopoetve - Whats the best way to test the iscsi performance? Not the redundant part but just transfer speeds in general?
 
If you look at what lopoetve is doing you'll see he makes two iSCSI networks with different IP subnets. Why? Because the software iSCSI initiator will NOT load balance two connections if they are on the same IP subnet. You have to do two. For many things this means you do two iSCSI networks and two iSCSI targets on the storage array.

But he's exactly right..that's how you do it.
 
NetJunkie said:
If you look at what lopoetve is doing you'll see he makes two iSCSI networks with different IP subnets. Why? Because the software iSCSI initiator will NOT load balance two connections if they are on the same IP subnet. You have to do two. For many things this means you do two iSCSI networks and two iSCSI targets on the storage array.

But he's exactly right..that's how you do it.
Click to expand...

:)

Pretty much dead on -more than that htough, we can't have 2 vmkernel ports on the same subnet. We're creating two ports for the same initiator. It still won't load balance, but you can set preferred paths to balance load on the SPs yourself. (don't try round robin with this yet).
 
You must log in or register to reply here.
Back
Top