Upgrade journey to vSphere

V

Vader

Supreme [H]ardness
2FA
Joined
Dec 22, 2002
Messages
5,139
Well, I finally received approval to upgrade our current VI3.5 infrastructure to vSphere 4.1 U1.

Although, it's a minimal step...phase one is complete. We upgraded our Dell m1000e this morning with the latest firmaware. Love the new interface.

m1000e.jpg


We also removed 2 ESX hosts and I will be completing a fresh install of ESXi this week on those as well as upgrading our 16pt passthrough switches for larger capacity, then installing a couple of Cisco Nexus 1000v's.

Next phase is updating our EMC CX4-240 to the latest Flare release and adding additional capacity to include EFD's for FAST Cache to start, and FAST tiering with a large DB project coming up. Mostly this update will be to take advantage of all the storage features with vSphere and to utilize the EMC plugins to provision within vCenter similiarly to my lab:

emcvcenter.jpg


I will update this post regularly on my progress. This is going to be an exciting year!:)
 
Last edited:
Keep in mind to use FAST auto tiering you have to create storage pools. I figure you know that since you said you are adding capacity including EFDs. It's a hassle to convert an existing array to FAST without adding additional spindles to seed the new storage pool(s) with. Some people miss that.

Let me know if you hit any issues.
 
Keep in mind to use FAST auto tiering you have to create storage pools. I figure you know that since you said you are adding capacity including EFDs. It's a hassle to convert an existing array to FAST without adding additional spindles to seed the new storage pool(s) with. Some people miss that.

Let me know if you hit any issues.
Click to expand...

Thanks! The FAST auto tier will come later with another project which we will be purchasing new DAE's..etc for. The only thing i'm really concerned about with this upgrade is implementing the Nexus 1000v's. I have lot's of reading to do..lol.
 
Let me know if you need help on the 1Kv. Read my blog. I write a good bit on it and I did the session at VMworld last year.
 
I read it already..but that was a while ago..i'll definately go back and look it over.

BTW, how did Varrow madness go? Wish I could've been there..you guys had a killer guest list on the topics I was really interested in and it looked like much fun.

Go UCONN!;)
 
The event was GREAT. It far exceeded our expectations. Was told a few times it was like a mini EMC World. Presenters, sponsors, my own people..we all did our job very well and it went off without a hitch. VCE brought in a real Vblock for people to look at..the labs ran great. Very pleased so it'll be an annual event for sure now. Here is a slightly skewed panoramic pic I took. The tables were actually more like 2/3 of the view but due to my vantage point they look jammed together. :)

http://yfrog.com/h7iatulj

EDIT: Blog post by Chad. http://virtualgeek.typepad.com/virtual_geek/2011/03/3-big-ideas-and-a-low-level-vnx-discussion.html
 
Gahhh...i'm really frustrated. My Nexus 1000v's and PowerPath/VE licenses were removed from the capital budget as "unnecessary items." I wasn't that mad about the Nexus but the PowerPath really got me. While it's true that both aren't really a necessity, I at least wanted PowerPath for "smart" mutlipathing to our Clariion.

I'm going to put it in the next reforecast. I have a new environment budget for our non-gov't clients in June so hopefully i'll be able to get at least PowerPath in there, it is a larger budget so i'm thinking we'll be on track.
 
What are you upgrading your passthroughs to? I am guessing you are talking about the 16port PS in the back of the chassis. We currently run the 10GB passthroughs and the only issue we have had so far is a compatibility issue with the Brocade CNA's that is being resolved by Dell right now
 
I orginally wanted to go 10GBe but once again, no one wanted to spend the dough. Right now i'm just stuck with 4pts per ESXi host.

For now, i'm just going to implement 1 vSwitch with all 4 and configure VLANS for the Management, VM, vMotion/FT traffic.

My initial goal was to implement w/Capital costs with a lot of the extras...ie PowerPath/VE, Nexus 1000v's..etc.

Now it's completely stripped down, so my focus is now on upgrading to vSphere 4.1 U1 and reforecast some of the extras and spread them out a bit over the course of this year.
 
Last edited:
ESXi is installed on the two blades we removed from the VI3.5 infrastructure.

I'm in the process of setting up the network and have some questions.

Here is what i'm working with and from researching this is how I think this should be setup:

1. We have 4 passthrough NICS per blade slot so 4 Uplinks.
2. We have two FC HBA's passthroughs per blade slot as well.

First, switch setup, which I have questions about. It's a single 6509 and i'm going to use the Native VLAN for the VM/Management traffic and I have created a different VLAN for vMotion/FT.
Here is where i'm a bit confused. Since I will be setting up all nics in an Active/Active configuration, using the Load Balancing Policy "Route Based on IP Hash" I need to setup Link Aggregation for this to work correctly. Then I see in the next paragraph that LACP is not supported? So here is my confusion, in order for this to work, LACP needs to be setup but it's not supported in ESX/ESXi?

Pending the answer to the first questions, my ESX/ESXi Network configuration will be 4 Adapters in an ACTIVE/ACTIVE w/VM Network-Management IP in the native VLAN. All adpaters will be ACTIVE for this network. VLAN 24 in VMkernal Network for FT/vMotion. NIC4 will be primary and all others standby for this network.

Any input would be appreciated.
 
I wouldn't put all your traffic on a single set of pNICs. I've had issues in the past with heartbeat traffic being lost during heavy load on the NICs when the Mgmt port group is living on a vswitch with VM traffic.

Why can't you create one vswitch for Mgmt and VMotion with 2 pNICs, then a second vswitch with 2 pNICs for VM traffic? That would be the better configuration with 4 pNICs and if you want to create aggregates you still can so each vswitch is utilizing it's two pNICs.

I'd also VLAN the Mgmt traffic away from the VM traffic. You want to tightly control where the Mgmt traffic goes and who can get to it. Then create a single VLAN for VMotion and another VLAN for FT (only if you actually plan on using FT, otherwise don't bother with it).

You may have already seen this, but this KB article will help with implementing link aggregates on your network.

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1004048
 
If you have Enterprise Plus (I assume you do, since you mentioned the 1kv), then you could use a dvSwitch for the VM traffic, then the new load based teaming policy which in my opinion is a better route to go than using link aggregation in a lot of cases. I still think using a standard vSwitch is best for the management network/vmotion/etc.
 
Let me get this straight:

1. Use a standard vSwitch and segment two of the four ports in an Active/Active config and create an additional VMkernal port for vMotion-FT and use two separate VLAN's...the vMotion can be a non-routeable VLAN, but the Mangement will have to be routeable as we won't sit on the same segment.

2. Create a dvSwitch for the VM network in the current prod VLAN and utilize the last two uplinks.

I've seen so many different recommendations, here and in my books i'm reading...lol.

I'm trying to go "best practices" here..that's huge for me..following vendor best practices and guidelines.
 
Vader said:
Let me get this straight:

1. Use a standard vSwitch and segment two of the four ports in an Active/Active config and create an additional VMkernal port for vMotion-FT and use two separate VLAN's...the vMotion can be a non-routeable VLAN, but the Mangement will have to be routeable as we won't sit on the same segment.

2. Create a dvSwitch for the VM network in the current prod VLAN and utilize the last two uplinks.

I've seen so many different recommendations, here and in my books i'm reading...lol.

I'm trying to go "best practices" here..that's huge for me..following vendor best practices and guidelines.
Click to expand...


None of what we're really talking about violates any best practices. Its definitely BEST to split the networks up, management from VM traffic. It is BEST to have the management on its own VLAN, and vmotion and FT on their own. It is BEST if you have the management stuff on a standard vSwitch on that host directly. You don't have to, but it does make things easier to manage -- ie configuring the management network info is still doable through the DCUI rather than having to use vCenter in a situation where vCenter isn't available.

The rest of it, vSwitches or dvSwitches for VMs, which load balancing policy you use, how many vlans you need, etc, all of that is situational. Not really a best practice. The one rule of thumb I go with is whatever is EASIEST for you but meets all the goals. If you're comfortable with dvSwitches, then the new load based teaming policy is awesome. There is aboslutely nothing wrong with doing a standard vSwitch with route based on the originating virtual port, or using link aggregation and the IP hash policy. You need to do what is best for your environment, but understand the impacts of your decisions.
 
Ok..sounds good..thanks for the input. I'll have an update as soon as the SAN Flare update is completed and we have some additional storage then I can install vCenter and get my cluster setup. I really should be looking into the Fiber setup and not talking about networking..lol.
 
I just wanted to update this post to say that the upgrade is going along nicely. I've been working on the Server cluster, after will be the VDI cluster.

The past couple of days i've implemented the network changes and attached some storage..setup vCenter...working on View and the Wyse terminal manager. I only had one hiccup which was user error on my part...but here is some screenshots to give you an idea of implementing by best practices and great input from the experts here.

NETWORK Setup- Single vSwitch for Management and vMotion/FT - dvSwitch for VM traffic - All logically segmented - Port Channels for both/trunks - LoadBalancing IP HASH







STORAGE Setup- ALUA (Failover Mode 4 EMC Clariion) - Round Robin ACTIVE/ACTIVE - Storage I/O






Of course I have a lot more to do.....but more updates will follow. I have to say after planning and mapping it out..etc...it's fairly stright forward. I think my next big hurdle will be setting up View Manager and Composer..we shall see! Still have another node to add to this and another 5 node VDI cluster...so much work..but setting up all the backend took a lot time to even get to this point. I also still have the Flare 30 update to complete which is happening in the next week or so so I can start provisioning and getting the benefits from our SAN integration w/vSphere...
 
Last edited:
Sweet :).

I have a question: I see you have resource pools, so do you know what the resource pool pie paradox is?

I don't know that I'm all cozy around the idea of using ip addresses for the hosts in the inventory, but that is just because I'm OCD, it should work like that without problems. HA used to be pickier than it is now.

Very nice job so far though, I dig it.
 
I have a question: I see you have resource pools, so do you know what the resource pool pie paradox is?
Click to expand...

Yeah..i've read about it..so many vm's have so many pieces of the pie in terms of resources..etc.I have to really calculate what i require...VMware's design of resource pools isn't the best..that's for sure. I just set up three basic for now so if they don't work out..I can wax them...but i'll have something to test with first.
 
Vader said:
Yeah..i've read about it..so many vm's have so many pieces of the pie in terms of resources..etc.I have to really calculate what i require...VMware's design of resource pools isn't the best..that's for sure. I just set up three basic for now so if they don't work out..I can wax them...but i'll have something to test with first.
Click to expand...

You're the man, you'd be amazed how far you are ahead of most people in your shoes. Kudos to you for all the research you've done. Duncan posted a pretty sweet powershell script that can run nightly to adjust the shares for the resource pools given a specific weight you specify to each. I'm just too lazy to search yellow-bricks for it :p. For such a small environment it probably isn't a big deal, but around here when literally many 10s can fire up in a given day it becomes a bigger issue.
 
Last edited:
Ducan posted a pretty sweet powershell script that can run nightly to adjust the shares for the resource pools given a specific weight you specify to each. I'm just too lazy to search yellow-bricks for it :p. For such a small environment it probably isn't a big deal, but around here when literally many 10s can fire up in a given day it becomes a bigger issue. ]
Click to expand...

I appreciate it...i'll have to look into that if the need arises....I have to say may main learning sources are:
Scott Lowes Mastering vSphere Book
Virtual Geek Blog
Yellow Bricks (of course..lol)
Jason Nash's blog..thanks Netjunkie!
Nickapedia
Some other sites..oh yeah..hardforum!..lol
My home lab of course...running an EMC VSA and two ESXi hosts...unfortunately dont have the required network switches for VLANs..etc as of yet...will get there.

And of course, I am going over the material for the blueprints for the exam and the thousands of pages of .pdf's from VMware's site. I know I still have quite a bit to learn..hoping to move to a consulting position after my 3rd interview with a company similair to Varrow..small, quickly growing..etc.working their Cloud infrastructure with Cisco UCS-EMC-VMware. That's where I want to be so..hopefully tomorrow..I get a solid offer as i'm visiting the home office and getting the tour in Rochester.
 
Last edited:
You must log in or register to reply here.
Back
Top