Building a scalable VM deployment

The Spyder

2[H]4U
Joined
Jun 18, 2002
Messages
2,628
I was asked last week to virualize a server environment for a larger client of mine. They currently have 6-8 servers. 60 active users. They want to add 4 more servers and have expansion plans to grow to over 150 people in the next 2 years. There current storage and database servers are insufficient. The plan is to move them to a redundant VM solution and use the newest of the existing servers as part of the project. I need to spec out the hardware this weekend for a report Monday. My first attempt was more of out of fun then anything, but within their budget.

(3)
Poweredge R710
(2) Intel Xeon 5550 Quad 2.66 1333fsb 8mg cache
32gb RDIMMS 1333fsb (8x4gb)
Rails with Management arms
Perc6i
Perc6e
2 Intel Pro 1000PT Dual Port adapters
2x450gb 15k SAS drives
HO Redundant powersupplies
iDrac6 cards
3yr 4hr/365 same day response.

(1)
MD3000i
(12) 1tb SAS nearline drives
2 Controller Cards
3yr 4hr/365 same day response.

Now this is completely over kill at just over $50k. I still need to squeeze in switches, battery backup, and make sure the existing backup hardware can handle the new storage.

What have you guys seen in a environment like this, or rather what would you like to see?

HP is next on my list... :)
 
look at sun, I'm presently buying x4450's lately, they're great. look at the 4170 (dual nehalem) for what you're doing. The density in our 4450's is great, 4x 6-core intel CPU's and 128GB RAM (up to 256 per host).
on the storage I'm using a NetApp 3140, but that'd be very overkill, maybe something like a netapp2050? a lot of people seem to like dell md3000's or whatever, although I've never used them myself.
 
Drop the 1TB drives and get faster spindles. Remember, disk IO is the biggest hurdle in the VM world.

Also, big drives on the hosts are wasted space if you expect the VMs to be highly available (meaning VMs must be on the san). With ESX(i), 72GB drives are even overkill...
 
Also (a little OT), I just noticed the Dell MD3000i uses the same firmware/management agent as the IBM DS3400...
 
Drop the 1TB drives and get faster spindles. Remember, disk IO is the biggest hurdle in the VM world.

Also, big drives on the hosts are wasted space if you expect the VMs to be highly available (meaning VMs must be on the san). With ESX(i), 72GB drives are even overkill...

I need 6tb of HDD space for the fileserver. 1tb SAS isnt good enough? Would I be better using the 1tb as a storage pool and changing the main server drives to 72gb + adding 4 350gb SAS drives to the MD3000i?
 
I need 6tb of HDD space for the fileserver. 1tb SAS isnt good enough? Would I be better using the 1tb as a storage pool and changing the main server drives to 72gb + adding 4 350gb SAS drives to the MD3000i?

SAN is good, but the problem is you won't get extremly good IO from so few spindles. Another thing to take into consideration is rebuild time which will be quite high with such large disks. To look at it objectively you would need to list all IO heavy virtuals that will use the SAN, and what type of IO it will be. Something like a very busy mail server can cause havoc in a scenario like yours.
 
From my reading... I need to change quite a few things. Bear with me as I am going from small scale, budget restrictive environments to full scale deployment.

It looks like I want 2 servers. Dual Quads (Same architecture for easy future cluster expansion). 32gb Ram each. Internal Storage can be small (<36gb), but fast for the core install. Servers are connected to the iSCSI SAN, configured with 2 arrays, the first being a single array with multiple LUN's mapped to it for the VM's. Several VMFS3 stores for the servers. 5 Disk Raid5 with 15k drives for the first array would give me ~15 VM's (4 disk with 1 spare). A second 10 disk 15k SAS Raid 5 (9+1) would give me the storage needed. I would have to setup a internal guest iscsi connector in a Server 2k8 box to access the second array and let me format it/use it OR setup multiple LUN's, stripe them together with the OS, and go that route to bypass the 2tb cap.(Would I be best to do the latter with RDM's?) I will NOT be able to snapshot this box due to the disk size.

This would leave me with:
2 Servers (Dell or HP)
1 SAN with a 5x15k SAS (How Much gb for 5-10 VM's?) and 10x15k SAS (Whats the largest true SAS? 450gb?) (MD3000i?)
2 Switches (redundancy) (Dell/HP/Cisco? Existing core, if any, will be replaced with these- expecting 24port)
2 new battery backups with network cards

Please feel free to comment, provide feedback, hardware suggestions, ect. I hope I am on the right track.
I found out on Friday- the cheaper the better, meaning my initial impression was wrong as they want to fund this with the last of 2009's budget- time is of the essence.



Some INFO I am reposting so I remember in the AM.
VMFS3 need to have proper block sizing- relevant to the size of the volume.
256GB -1MB blocks
512GB - 2MB blocks
1024GB+ - 4MB blocks
2048- 8mg blocks
 
Dell Configuration:
(2)
Dell Poweredge R710
(2) Intel Xeon 5220 2.26/1066/8mg
32gb RDIMM 1066
2x 146gb SAS 15k (Smallest option)
Perc6i onboard card
iDrac6 Managment card
Rapid Rails with management arms
High Output Redudant Powersupply
3yr PRO 24x7 4hr responce

MD3000i
5x 146gb 15k SAS
10x 450gb 15k SAS
2x2port Controllers
Redundant Powersupply
Rack Rails
3yr 24x7 4HR PRO support

(2)
Powerconnect 5424
Redundant Powersupply
3yr 24x7 4HR PRO support

$34k before discounts, ect ect.
 
Where are their app requirements? What is there I/O load? You can't spec an environment without that information. That's why you perform a Capacity Planner first or atleast PerfMon. We redo a lot of VMware deployments where other consultants come in and guess on these items and the performance is awful. Then virtualization gets a bad name in that organization when it was never properly designed.
 
as NetJunkie said, you need to appropriately size the solution for the current environment, and plan for growth in the future. Getting a handle on what your systems are, what their current load looks like as physical machines, and their CPU/RAM/DISK usage is very important, and you have not specified any of that.

+1 for capacity planner in this case. Many companies (and consultants) incorrectly gauge the performance needed in the areas of CPU/RAM/DISK. Don't spend money now where you don't have to. They know they are growing, so it should be expected that, down the road, they will be spending more $ on IT to facilitate that growth. Make sure that, in this case, you have scale out options on the hosts, and scale up on the storage. failure to do this will pigeon hole them into solutions that work now, but are no longer available in 2 years. Don't recommend tech that has been out for 3 years already...chances are, it's getting ready to go EOL, and a few years after that, will no longer be supportable.

...size it first, price it second...never in reverse.
 
...size it first, price it second...never in reverse.

THIS. ESPECIALLY on storage. And don't just size it for "size", size it for IOPS. It's fine to use a bunch of 1TB SATA drives in a mass store, but when you're talking a max theoretical deep queue IO performance of, say, 100 IOPS, and you add in RAID 5 overhead, you get a lot of storage, but it's a lot slower than you'd expect (especially compared to almost 300 IOPS out of a good SAS 15k drive). It's not hard to exceed your iops budget by a mile and stay WELL within your sizing budget.

Budget both. If you need help with the iops, I have a set of formulas I can help you with.
 
See, this is why I ask you guys!
I never even knew about capacity planner, and honestly I do not believe the other techs I am working with know about it either. There are very view consultants in town specializing in VM. Does anyone know the approximate license cost? Please keep in mind, I have been brought in to this blind with very little info about the existing setup and was asked to spend $XX to deploy a scalable VM system. Hardware specs are based on the provided budget/ limited existing infrastructure information/ planned growth.
 
THIS. ESPECIALLY on storage. And don't just size it for "size", size it for IOPS. It's fine to use a bunch of 1TB SATA drives in a mass store, but when you're talking a max theoretical deep queue IO performance of, say, 100 IOPS, and you add in RAID 5 overhead, you get a lot of storage, but it's a lot slower than you'd expect (especially compared to almost 300 IOPS out of a good SAS 15k drive). It's not hard to exceed your iops budget by a mile and stay WELL within your sizing budget.

Budget both. If you need help with the iops, I have a set of formulas I can help you with.

That would be great.

I shot out a email to the team asking to get me specific information about the servers, existing/planned and suggesting capacity planner.

The second setup I just did had 5x 146gb 15k SAS and 10x 450gb 15k SAS for storage.
 
I would recommend an acceleration pack for you. List pricing is here: http://www.vmware.com/vmwarestore/vsphere_accel_purchaseoptions.html

Find a local business partner using the VMware website. They will be able to secure you better than list pricing. If you are new to VMware, you might want to dig in and find out what features you think you need or want. Many of us here can give you more info on the features, should you need it.

If you're budget is $35k, you're going to need to plan to spend about ~$15k of it on licenses. And FYI, you cannot buy licenses without support @ the initial time of sale. You can let it expire if you choose to, after the first year (but I would not recommend it).
 
$35k is hardware. Licensing will be interesting, as they are a microsoft non profit and may want- dare I say- hyperv :(
 
Hi Spyder,

Are you getting pricing directly from the manufacturer in HP case or you are involved with a VAR?

With some brands, discounts can be noticeable from list price.

I am working for a VAR focusing mainly on Sun Microsystems and I can say that their pricing are quite competitive.

Their servers have some features like integrated Lights out management and quad NICs that comes as base configuration.

I have setup multiple customers with VMware VI3 and VS4 and let me tell you that they are rock solid.

I may be biased but they can be a really good Tier 1 alternative to brands like HP, IBM and Dell.

Also, considering the fact that you are running databases, I would advice you to run 300 or 450GB SAS or FC drives since I/O are the most frequent bottle necks experienced by people getting in virtualization. As a rule of thumb, I suggest figuring at least the equivalent of 1 drive per VM since you want at least as much I/O performance as your physical servers. Let me know if you have any question concerning Sun hardware.
 
I actually use a spreadsheet for my iops calcs. I'll dig the numbers out for you all.
 
Meeting today-

$30k budget for 2 servers, SAN, switches, APC's, and rack managment.
Hyper V (Free for them)
Server 2008 (Free for them)

Current data store is ~500gb, total growth over 5 years expected to be 2.5-5 TB.
Lots of fun... Gotta re do everything and research Hyper-V.
Looking to run 8 virtual machines to start with.

Wish me luck as I would have rather used VMWARE any other day of the week.
 
They are a non profit and the physical hardware budget was put in place before this was decided. I am doing it because the company is looking for a tech like me and has expressed intrest in hiring after the projects complete. Very neat company, nice people.
 
Non-Profits can get Windows 2008 Enterprise for next to nothing which would give you 1 Physical License and 4 Virtual Licenses. Did a virtualization project for a school using Hyper-V earlier this year, 7 TB SAN, 2 Servers, software for $35,000 so it definitely can be done.
 
Excellent to hear. I have deployed Hyper-V twice before, but on small (4 VM's per host) scale.
 
Dell knocked my $30k quote down to just above 20. I am stunned. Yay for non profits!
 
Dell knocked my $30k quote down to just above 20. I am stunned. Yay for non profits!

Don't stop their!!! turn hp against them then turn hp against dell, or whoever else hardware company you might purchase from
 
Haha! Thats the plan today- That and I was asked if you had $x more money to spend, where would you spend it? I may get to "spend the savings" as they do have $30k in their budget and need to spend it. Time to look in to upgrading their backup solution. :D
 
After much bickering and some budgeting, here is what I ended up with:

2) Dell R710 servers
With Dual, Quad Core 2.4gzh 1066 8mg cache, 32gb RAM, 2x146g 15k SAS, Intel Pro 1000 dual port NIC, Redundant powersupplies, rack rails, cable arm, and 3yr 2HR Responce

1) MD3000i
11X 450GB 15K SAS
3yr warranty
Redundant powersupplies and Controllers

2)APC 2200VA + network management cards

2) Dell 5424 Gigabit switches with 3yr warranty

For $24k shipped. Not bad IMO. Going in tomorrow for another meeting.
 
Back
Top