My over-engineered VMware lab

[DracoDan]

So, I have been frustrated by the limited amount of learning I can do with my simple ESXi box, additionally, I have a VM that is using a ton of storage (on an unofficially support Areca ARC-1210 RAID card) that I use for my file server, and I don't want to risk screwing it up, so I decided to come up with a better solution where I have multiple hosts and a separation between my storage and hosts.

Initially, I figured I could just do iSCSI targeting from a Linux system and deal with 1gb/s (at best) disk throughput, but as I started looking, I found a MUCH better and surprisingly cheap solution. You can get PCI-X 2gb/s FC cards for absolutely dirt cheap (two of them I got for $1/ea). I started researching and found out openfiler supports FC targeting, meaning I can build a dirt cheap but TRUE SAN. All my cards are dual port, so I can actually get 4gb/s throughput. Here is my list of parts and prices paid so far, let me know what you think.


Hosts (two hosts, so everything here is doubled) $391/ea

$100 Arima NM461 motherboard with 2x 133mhz PCI-X and 2x x8 PCI-E
$45 ($22.50 x2) Pair of opteron 8350 2ghz quad-core socket F CPUs for a total of 8 cores
$80 ($10 x8) 2GB PC2-5300 ECC/REG for a total of 16GB RAM
$1 QLogic 2312 dual port 2gb/s FC PCI-X card
$50 ($25 x2) Xigmatek LOKI HS/F (currently be modded for front to back cooling layout)
$70 Norco 4U rack mount case
$45 Corsair 430W Builder series single rail PSU (not top quality, but should do fine)

SAN $909

$100 Arima NM461 motherboard with 2x 133mhz PCI-X and 2x x8 PCI-E
$0 (included with boards) Pair of opteron 2214 HE 2.2ghz dual-core socket F CPUs for a total of 4 cores
$60 ($10 x2 + $5 x8) 2x 2GB + 8x 1GB PC2-5300 ECC/REG for a total of 12GB RAM
$15 low profile QLogic 2312 dual port 2gb/s FC PCI-X card
$100 LSI 8888ELP SAS RAID card (minor damage but works perfect) (bracket modified for low profile case)
$0 (recycled) Intel X25-m 80GB (Primary VM store)
$0 (recycled) Seagate Barracuda ES 250GB 7200rpm x2 in RAID1 (250GB usable for openfiler boot, Host boot LUNS and secondary VM store)
$100? (4/6 are recycled, still need to order last two) Seagate 1.5TB 7200rpm x6 in RAID6 (6TB usable for data store)
$20 ($10 x2) SFF-8087 .5M cables
$0 2U low profile heatsinks (included with boards)
$12 ($6 x2) 60mm temperature sensing fans (to replace the screamers on the 2U heatsinks)
$195 Norco 2U case with 8x hotswap bays and SAS backplane
$45 460W Seasonic 2U PSU
$50 Brocade silkworm 3200 with 8x 2gb LC gbics
$36 ($6 x6) 1M LC-LC MM fiber

Software $0
This is a discussion about hardware really. My software will consist of VSphere, Windows Server, and Linux. I will not be using this system in a business environment so I don't feel bad about how I'm obtaining software, but I don't wish to discuss it

Total $1691
This also does NOT including shipping, even though it was free on most items. I would guess I spent another $100-150 on shipping.

(Almost) everything has been bought and a good bit of it is put together, but I got delayed by the fact that the heatsinks are designed to ONLY blow air bottom to top, which doesn't work very well in a rack mount case, so I'm building my own brackets (edit: brackets work perfectly, still need to make them for the second host).

This took a long time to write out, a lot longer than I expected, so PLEASE comment on my system, good or bad.

Thanks,
Dan

 

[DracoDan]

Wow, I got a bunch of hits when I (incorrectly) put this in the storage section, and now that I post here... no comments.

I did manage to get my first host completed and booting last night, everything looks great and it isn't very loud at all (for a server). I think the next step will be to get the first host and SAN rack mounted and cabled, and then I will start into the install. I REALLY wanted to use OpenFiler 3.0, but the OF guys are taking their sweet time getting it released.

 

[demografik]

in your other thread when you say "move everything within corporate IT to VMware" what is "everything?"

 

[piker28]

I keep meaning to come back and post about it. As someone who wants to get into VMware I am debating following your steps.

 

[DracoDan]

First off, don't confuse the corporate stuff with the home lab stuff, they're two seperate unrelated projects. The hardware listed above is only for my home lab and will run:

• Personal/family web server using apache/gallery/mysql to run a website for my rather large family to upload pictures to. Last I checked I currently have like 12k pics on this server.
• Personal file server for storing my insane amount of porn
• VCenter server
• AD/DNS/DHCP server (I need to get experience with locking down VCenter/View access via AD)
• VMware view server (need more experience with this product)
• Anything else I decide I need in the course of my training. I got my RHCSA/RHCT Last September, my next goal is RHCE and then VCP.

The corporate equipment I listed in the post for my work project will run the following services for about 150 people:

• AD server
• Internal DNS server
• exchange server
• Print server
• Sharepoint server
• File server
• CVS repo
• and several internal app/web/database servers.

 

[Red Squirrel]

How did you manage to get everything for so cheap? the 16GB of ram alone would probably set me back a couple grand here in Canada.

 

[DracoDan]

I keep meaning to come back and post about it. As someone who wants to get into VMware I am debating following your steps.

I started in with just a basic ESXi server that had a single Xeon 3060 (Q6600 equivalent), 8GB of ram, and about 4.5TB usable local storage. This worked out great for understanding the basics and getting a feel for how the different components are laid out, and is roughly what I would recommend for people just getting started. My new system is being built to allow me to expand my knowledge by getting into some of the advanced features such as high availability (HA), dynamic resource scheduling (DRS), fault tolerance (FT) as well as the use of vmview and vcenter and just keep me current on the technology.

I have been lucky enough to be able to work on a very high end enterprise vmware configuration on a government contract I am assigned to, but it is currently on hold thanks to the contracting office goofing up the renewal.

 

[piker28]

Hmm well I have about the same system with a core 2 quad lying around. I was not sure that would be compatible with ESXi. I loaded it instead on a spare Dell server I have but the damn thing is heavy and loud so a pain to deal with sometimes.

 

[DracoDan]

You should switch over to the Q6600, it's exactly the same as the Xeon 3060 and will work perfectly, including all the VT instructions.

 

[piker28]

I will look into it since I have the Q9400

 

[lopoetve]

interconnects are never the limitation... you wouldn't have outrun iSCSI, assuming you have more than 1 host. Disks will always run out first.

This is even MORE true in RAID 6, which sucks hard donkey cock for write performance. You're making the most common mistake people make in storage when it comes to virtualization -> IOPS are King, everything else is secondary, and those SATA disks at 7200 RPM in RAID6 are gonna have almost zilch for performance in that regard

In fact, you'll have...

~me does the math...~

About 150 front end IOPS for that array. Which is enough to run one light-load DB, or a couple of desktops under load, or a handful of web servers under light load, plus an AD box or two for any of the above.

In other words: RAID 10, my friend, or SAS 15k disks. Especially since you don't have much in the way of cache.

 

[Stanza33]

I am looking at setting up something similar.

I have questions about fibre... never played with it at all.

I have 2 x ML350 G5's
Got my eye on a few of those cheap dual port 2gb HP (qlogic) fibrechannel cards..

Can you just driect connect them easy enuf.. or is it best to have a switch?

They will connect to a ML350 G4 which will be Opensolaris.... for the Storage side

I also have 6 x dual port 1GB lan cards... I am presuming the fibre cards setup for the storage networking side of things (once I figure how) will easily beat the lan cards as storage network yes?

so

2 x ML350 g5's for Vitrual machines
1 x ML350 G4 as the storage
5 x dual port lan cards
3 x fibre card (still sourcing)
Fibre switch bidding on ebay now.

Best way to configure is the next step?

 

[farscapesg1]

Any idea what the power consumption is on those hosts? Just curious since I'm always wondering what people are spending on electricity for their home labs I've got a Dell PE2950 that I got for free, but will probably sell on craigslist or the FS forum (local only, not going to try to ship it) because it figured it would cost me $20-$25 USD per month to run 24/7...

 

[DracoDan]

interconnects are never the limitation... you wouldn't have outrun iSCSI, assuming you have more than 1 host. Disks will always run out first.

This is even MORE true in RAID 6, which sucks hard donkey cock for write performance. You're making the most common mistake people make in storage when it comes to virtualization -> IOPS are King, everything else is secondary, and those SATA disks at 7200 RPM in RAID6 are gonna have almost zilch for performance in that regard

In fact, you'll have...

~me does the math...~

About 150 front end IOPS for that array. Which is enough to run one light-load DB, or a couple of desktops under load, or a handful of web servers under light load, plus an AD box or two for any of the above.

In other words: RAID 10, my friend, or SAS 15k disks. Especially since you don't have much in the way of cache.

Very good points; you have gotten me to rethink a few things. The major point in the large RAID 6 array is actually my file server, which is one of the systems being hosted. It will be very common for this system to have a steady flow of sequential writes being done to it.

As far as the VMs, I'm now considering putting my old x25-m (80GB Gen2) in the openfiler to use for the actual VM data stores, and then I will just depend on backups for dealing with the lack of redundancy. Worst case scenario is that the SSD dies and I have to run the VMs from the RAID 6 until the SSD is replaced. The only real question is whether I can comfortably run 4-6 Linux and 2 Windows VMs in about 70GB of space. I would really rather not have to buy any more new hardware for this.

 

[DracoDan]

Any idea what the power consumption is on those hosts? Just curious since I'm always wondering what people are spending on electricity for their home labs I've got a Dell PE2950 that I got for free, but will probably sell on craigslist or the FS forum (local only, not going to try to ship it) because it figured it would cost me $20-$25 USD per month to run 24/7...

I should be able to give you at least a rough idea once I have everything up and running. I plan to start loading up the software tonight. Once I have everything running, I believe my APC Smart-UPS provides the ability to view the power draw, but don't quote me on that. If it doesn't have the ability, a guesstimate will have to suffice. Since I plan to write all of this off as a business expense, I will actually need to know the real numbers, so I may look at something like a kill-a-watt. The problem with that is the kill-a-watt is only rated at 15A while my UPS draws up to 20A (and is on an appropriate breaker).

 

[DracoDan]

interconnects are never the limitation... you wouldn't have outrun iSCSI, assuming you have more than 1 host. Disks will always run out first.

This is even MORE true in RAID 6, which sucks hard donkey cock for write performance. You're making the most common mistake people make in storage when it comes to virtualization -> IOPS are King, everything else is secondary, and those SATA disks at 7200 RPM in RAID6 are gonna have almost zilch for performance in that regard

In fact, you'll have...

~me does the math...~

About 150 front end IOPS for that array. Which is enough to run one light-load DB, or a couple of desktops under load, or a handful of web servers under light load, plus an AD box or two for any of the above.

In other words: RAID 10, my friend, or SAS 15k disks. Especially since you don't have much in the way of cache.

After thinking about this a bit, I decided on my solution.

You see, I hadn't actually bought 4 of the drives yet, I still needed to double check the model number on the drives I have, and they are in operation. I also had 4 slightly older seagate enterprise drives, 250GB each. So now my plan is as follows:

4x 250GB in RAID10 = 500GB - VM store
4x 1.5TB in RAID 5 = 4.5TB - Data store

I've also put a healthy bid in on a pair for Seagate 15k.6 300GB SAS drives on ebay, if I win I will be putting those in RAID1 for my VM stores instead, cutting my space slightly in trade for a huge speed boost. I may eventually go to an SSD, but with the current price/GB, it would be difficult to get enough space without spending a ton.

 

[/dev/null]

I did something similar. I use proxmox/KVM with gig-e to NFS on zfs server

14 * 73G raptor in raid10 for vm's (err mirror raid)
then 3-5 disks in raidz for my apt repository.

Runs great with a dozen vm's on a single server that is a phen2 965 with 16G.

 

[dasaint]

I use OpenFiler 2.3 in my home production environment

6x 2TB Hitachi 3K5000 in Raid 6 (Videos, Audio, ISOs, etc) NON-VM Drive
4x 1TB WD Greens in Raid10 (VMs)
1x Revo 240 PCI-Express (Linked Clones for View)

I have been playing with OpenFiler 2.99 on a VM using PCI-Passthrough (LSI1068e) with 3x500GB (Raid5) on an iSCSI SCST lun which has been running pretty well.

i tried upgrading my Prod to 2.99 but came across issues with slow web interface (could be the way i installed it but still working on that part) (but i will mention that you should use 2.99 especially for iSCSI using SCST instead of IET and also they revamped SCST to 2.0

Definatly go Raid 10 even if you do get those 15k SAS drives IOPS are definatly king when it comes to building a proper environment.

i will say be careful when using those PCI-X cards for Fibre, used a few and have recently had a lot of em just go bad (aged). switch to PCI-Express Fibre 2gb cards ~50
/ea on ebay and have had a lot less problems.

Check my Sig for my setup

 

[DracoDan]

I use OpenFiler 2.3 in my home production environment

6x 2TB Hitachi 3K5000 in Raid 6 (Videos, Audio, ISOs, etc) NON-VM Drive
4x 1TB WD Greens in Raid10 (VMs)
1x Revo 240 PCI-Express (Linked Clones for View)

I have been playing with OpenFiler 2.99 on a VM using PCI-Passthrough (LSI1068e) with 3x500GB (Raid5) on an iSCSI SCST lun which has been running pretty well.

i tried upgrading my Prod to 2.99 but came across issues with slow web interface (could be the way i installed it but still working on that part) (but i will mention that you should use 2.99 especially for iSCSI using SCST instead of IET and also they revamped SCST to 2.0

Definatly go Raid 10 even if you do get those 15k SAS drives IOPS are definatly king when it comes to building a proper environment.

i will say be careful when using those PCI-X cards for Fibre, used a few and have recently had a lot of em just go bad (aged). switch to PCI-Express Fibre 2gb cards ~50
/ea on ebay and have had a lot less problems.

Check my Sig for my setup

Funny you should mention the slow web thing in 2.99, I figured it was just something with java/flash/DNS causing it. I quickly forgot about the web interface and started doing all my configuration from the command line. I'm a Linux sysadmin so I usually prefer this method anyway, and there was enough info out there about setting everything up for the CLI that I totally forgot about the slow web stuff.

That's good info to know about the FC stuff, but if I have problems, I will just move to direct connected SAS, I already picked up a dual port PCI-e card for my server for dirt cheap, I would just need to grab a couple more for my two servers. Not sure openfiler supports this type of setup or not, I'm still looking into that. Another cheap option is going with older infiniband stuff, I've seen it ridiculously cheap.

 

[DracoDan]

Numbers updated to reflect changes for the disk setup as well as more RAM I bought. The extra 2GB of ECC/REG ram I had bought for my filer turned out to be non-ECC/REG and the seller gave me a refund. I turned around and found a great deal on 1GB sticks x8 though, so my filer will be up to a healthy 12GB of RAM.

 

[QHalo]

i will say be careful when using those PCI-X cards for Fibre, used a few and have recently had a lot of em just go bad (aged). switch to PCI-Express Fibre 2gb cards ~50
/ea on ebay and have had a lot less problems.

Check my Sig for my setup

I'm having some difficulty finding the cards you're referencing for ~50. I can't find anything PCIe for less than 3x that price really. I'm really considering building a small FC system in my home lab but I'd like to not break the bank to add it in.

 

[DracoDan]

I'm having some difficulty finding the cards you're referencing for ~50. I can't find anything PCIe for less than 3x that price really. I'm really considering building a small FC system in my home lab but I'd like to not break the bank to add it in.

That's the same thing I saw. Had I been able to find PCI-e FC cards for the price, I would have gone with that so I didn't have to find motherboards with PCI-X slots.

 

[DracoDan]

Made another change to my original plans. I just finished upgrading my desktops x25-m G2 80GB disk to a nice Vertex 3 120GB. Now I'm going to put the x25-m into my openfiler and use that disk for the OS datastore. That should blow away anything I could get with traditional hard drives, and I should have enough space to do a single windows VM as well as 5-6 Linux VMs (20GB for Windows and ~8GB for Linux VMs). All real data will be on the RAID 5/6 array. I lose all redundancy doing this, so I will need to figure out some way to back the disk up.

 

[dasaint]

wow i saw that the prices have increased, i got my QLE2360's for 50/ea on ebay a few months back

Draco, be careful using SSDs i would definalty make sure that any reads are done from them but limit the amount of writes. but again IOPS are king so i would goto raid 10 on that real data for the performance gains..

 

[DracoDan]

wow i saw that the prices have increased, i got my QLE2360's for 50/ea on ebay a few months back

Draco, be careful using SSDs i would definalty make sure that any reads are done from them but limit the amount of writes. but again IOPS are king so i would goto raid 10 on that real data for the performance gains..

There's no reason that writes should be an issue. The intel drives are great for wear leveling and with ext4, I will have trim support. As far as the data goes, I'm talking about my actual file server. This storage is used for stuff like movies, porn, music, porn, backups, porn, and raw picture storage, I need space a lot more than I need speed. The VMs will be running on the SSD, which will blow away any spinning disk RAID level for IOPS.

 

[DracoDan]

Been a few months, I lost a lot of motivation over the summer and didn't get much done, but with the cool weather lately I've made some serious progress.

The first host and SAN are 100% complete and rack mounted, I also have my FC switch mounted and I'm waiting on a powerconnect 2716 that will round out the rack. I have all the parts ready to go for the second host, I just need to hack up the case like I did the first one, make my heatsink mounts, and get it all together. I have also once again changed the disk layout slightly, I now have this:

2x 250GB 7200 enterprise drives in RAID 1 (250GB usable)
This array holds openfiler as well as luns for each of the hosts to boot off of. I still have around 200GB free that I can use for VMs if the SSD gets too full

1x intel x25-m 80GB
All my VMs go here until I run out of space

6x Seagate 1.5TB in RAID 6 (6TB usable) - four of these drives are still in my old server and I still need to purchase the remainnig two.
All data for my file server goes here.

The biggest change is how the hosts boot, I ran into nothing but headaches trying to get vsphere 5 to install to my CF cards in both SATA and IDE adapters, and I finally gave up. It turned out booting from the fibre channel was extremely simple to get working, it even tried to connect to the LUNS on all possible paths. At this point, my servers have absolutely no local storage (except DVD drives).