Whitebox ESXI 5.1 Hardware Recommendation

K

Kerbysj

n00b
Joined
Mar 16, 2013
Messages
57
Hi guys,

Looking to build a new server as my dual cpu xeon machine decided to die via means of a faulty voltage regulator.

As i was about ready to migrate to a virtualized environment i thought i might as well do a system overhaul to higher end desktop parts which will also keep some of the heat down in my office!
Looking to host a few machines on here but want the head room to put a couple more without maxing her out.
win 2008 r2 server
Win 2012 server
A NAS distro have yet to decide to which to go with.
Win 7 xbmc sql server and possibly a tvheadend (may put this on a separate linux box)
the odd linux distro for testing.

Probably only have 4-6 running max at the same time.

I'm doing this on a slight budget so i have a couple of options below. Could anyone give their opinion of the two builds which would be better and if there are any problems with hardware i have chosen?

Build 1:

AMD 8350 - 8 Core
Gigabyte GA-990FXA-UD5 - apprently supports passthrough however is the NIC supported?
32gb Ram - Corsair Memory Vengeance Pro Series Blue 16GB DDR3 1600 MHz x 2

Build 2:

I7 4770 Haswell
Asus Z87-PRO
16GB (2x8GB) Corsair DDR3 Vengeance Low Profile Jet Black, PC3-12800 (1600) 16gb moving to 32gb after a month or so.


I have some hardware left from my last server namely 120gb agility 3 ssd, 13x 2tb drives, 2x3tb, a supermicro saslp-mv8 (hoping to use as passthrough to the NAS VM) A case and hot swap bays for above drives. also i have a corsair 850watt so more than enough power.

I also have a HP microserver with 8gb ram. This is currently running my old server build which is my Media server.

Which would be better? im finding it hard to get much info on this perticular setup as Haswell is too new and only a couple of Haswell setups out there documented.

Also would the AMD be sufficent? Would it offer passthrough successfully? Out of interest could i overclock it to squeeze more performance out of it?
 
I have *EXACTLY* Build 1, right down to the RAM. The NIC is supported straight out of the box for ESXi 5.1 - in fact everything on the motherboard (including the extra SATA controller) is identified and set up properly without having to add drivers or build custom installers. I'm even using my Win7-64bit VM to record video from my Hauppauge HD-PVR via NextPVR and I did *NOT* have to use USB passthrough to get the HD-PVR and the IR-blaster (I use a cable box) working.

I can't judge which is better as I don't have a Haswell box, but I've had zero hardware/driver issues with this rig (other than whatever VMware broke like USB passthrough in 5.1). Performance is good and the extra PCI-E lanes means that adding multiple HBAs is no biggie should I build a SAN.
 
Thanks rtangwai! ill order this up on monday morning!

So USB pass through is broken in 5.1? i did read about this breifly but im still just getting into VMware since i helped a client migrate 30 blade workstations to virtuals and ive fallen for vmware in a big way :D

Shame that usb doesn't passthrough as my tvcard is USB.

How many machines are you running? im just interested to how many machines it could handle and what kind of load. I will probably run a couple of game servers here and there and interested how well it will hold up.
 
rtangwai said:
I've had zero hardware/driver issues with this rig.
Click to expand...

IMO that is enough said especially if you are already looking at that specific hardware. I am partial to Supermicro but I use my system for different tasks and I need IPMI.
 
The performance of USB on my ESXi rig is good enough that passthrough isn't needed for my HD-PVR to record 1080i. In fact, under ESXi 5.0 the USB passthrough was much *SLOWER*. Try it with your TV card, it might work.
 
[I have an FX-8120 with the Gigabyte 990 mobo, 32gb RAM and at the moment I'm running 5 machines without breaking a sweat. Here's the list:
  • VM_1 -- Running DD-wrt. 1 CPU, 64mb RAM. This is my router between my two vswitches
  • VM_2 - 1 CPU, 2gb RAM running Win-XP pro. Just for testing stuff
  • VM-3 -- 1 CPU, 2gb RAM, Win 7 Pro--Used for work and online presentations
  • VM-4 -- 1 CPU, 512mb RAM, RH 6.4. this is my DNS and DHCP server.
  • VM-5 -- 4 CPU, 8gb RAM, Running RH 6.4. Used as a development platform for work software
  • VM-6 -- 1 CPU, 4gb RAM, running Win 2012 for work

Will be adding 2 more Win, 1 RH, and 1 Solaris (to simulate a lab environment for work). Based on the performance I've seen so far, adding these will not be a problem. In fact, my system outperforms the development environment they let me use at work. I'm a happy camper.

I can only think that the 8350 would be even better.

Bill W
 
I currently run an asrock xtreme11 z68 board and a 3770 i7, and it supports vt-d and runs great with 24gb of ram. Fortunately this board has an lsi hba that has 8 ports. I assign 2cores to everything, and all servers get 4gb of ram. I can tell you the ivy bridge processors are supported, and from what I've read, the i7 haswell is supported as well
 
Proberbly a stupid question but will doing a mild over clock on the 8350 cause problems? Just interested!
 
If it seams like the system is lagging, then go ahead and apply some light voltage and frequency modifications, but chances are you won't need to over clock. I used to use an i3 running at 3ghz with the same amount of machines that I do now and I never had a problem. Esxi isn't too processor intensive, but it does use more ram since it dedicates a set amount of ram that you assign
 
What video card would you recommend going with build one, since I have been kicking that build around for a bit now?
 
I actually have gone for the i7 build now. Long story! Thought I'll go for a newer CPU. Was also thinking power usage etc. All parts coming tomorrow so hopefully start building over next few days.
 
Nasty_Savage said:
What video card would you recommend going with build one, since I have been kicking that build around for a bit now?
Click to expand...

I have the MSI Radeon 5450. It compatible with IOMMU and is inexpensive. Unless you want something beefier, then I'd check out.
 
All hardware setup! Now just found out the onboard nic is not supported! Grrr! Only went with this as the scan.co.UK tech promised all was supported! Is there no way of adding drivers?
 
@OP

You have not researched enough. I had a box almost the same as you except for RAM. You can add drivers. It's not that hard to google how. The results on the 1st page show you how.

EDIT: There's even a precompiled driver and a utility you can use to make a custom ISO and use that ISO for installing ESXI.
 
Last edited:
metril said:
@OP

You have not researched enough. I had a box almost the same as you except for RAM. You can add drivers. It's not that hard to google how. The results on the 1st page show you how.

EDIT: There's even a precompiled driver and a utility you can use to make a custom ISO and use that ISO for installing ESXI.
Click to expand...


Sorry had the longest day at work and was looking at this at 11:30pm last night. will look this morning. Is the ASUS board definitely supported? Sorry i am a newbie at esxi but is it not based off of redhat? so if a driver exists it is compatible?
 
OK so found the utility and the driver! Hopefully should be good later when I get home from work! Was powered by rock star last night which generally means I was working on muscle memory not brain power!
 
I'd think either would have been ok, but I would have added some form of Intel NIC to the equation. They just seem to be the best fit for ESX, and they always work with vmware stuff.
 
OK so built an image of esxi with the compiled drivers. It installs (however says it doesn't support hardware passthrough ! I have found the correct settings though) when installation completes it appears to be the 60 day trial not just the hypervisor?
Anyway the real problem is its not booting from the USB??
 
@OP

ASUS boards don't have proper IOMMU support. I found the Gigabyte board to work better.

You can register for a free ESXI key that gives a limited feature set.
 
But the passthrough must also be enabled at the BIOS level.

As for booting from USB, did you install on the USB? Silly question I know, but.....
When I did my initial installationi, I just put the CD in, booted from CD and then the install program took over. Gave me a list of places to install to and I chose USB (4gb Kingston). After that, I went back into BIOS, changed the boot order and was off and running.

You can get Intel pro ether cards off ebay. Single or Dual (I got a dual port for $30 a month or so ago). But my RealTek NIC was also recognized by ESXi5 & 5.1 and worked fine. (I went with the dual-port Intel card for other reasons.)

Bill W
 
Well tried again this morning. If I try and install esxi from CD to USB with vt-d and CPU virtualization turned on esxi install crashes. That's using the bog standard disk and my compiled disk. If I turn them off at the bios esxi installs fine albeit warning me there is no pass through. I then let the machine reboot itself and it won't boot to esxi it loops back to bios. Starting to think the board I was recommended by their tech guys was a waste of time. Any ideas before I send it back to them pre-lubes to stick up their @rse? :)
Good morning from the UK BTW!
P.s thanks for all the help so far! Look. Forward to getting this running as have 40+tb of drives sitting on my desk just starring at me!
 
Last edited:
Yeah - use Astroglide X Silicone, it's not waterbased so it won't short out the motherboard :p

When I did my research a couple of months ago on a ESXi whitebox (before Haswell came out) I was immediately struck by how much ASUS avoided supporting VT-d on their Intel Z77 motherboards. I use an ASUS motherboard on my personal workstation and confirmed it, which is why I went Gigabyte for my ESXi host. I also went AMD because of the robust IOMMU support on their CPUs and MB chipsets - it's actually hard to build a new AMD system that *DOESN'T* support IOMMU. I'm not terribly surprised VT-d is broken on your Z87 board, ASUS is probably playing catchup in that particular area.

Basically you got burned because you didn't follow the first rule of software deployment: "Never fuck with version 1.0 of *ANYTHING*!" :D
 
Yep learning that! Im going to try a couple of things when I get in then if still same will be putting the motherboard back in the box for a refund and order the gigabyte for Saturday delivery. Well considering the bios actually has a setting it seems abit ridiculous it doesn't support it?! Even advertises it as the spec? What if I was to install esxi then enable vtd? Would that work?
 
Little update. I managed to actually get esxi running with vt-d enabled. Boot up can take up to 30min as it hangs on usbagregator. Once booted I managed to get a VM running of server 2012. All working fine. I assigned my super micro saslp-mv8 card as passthrough, rebooted the box. Once I assigned the card to my VM the VM hung on boot up. Now here's where the interesting stuff happened!
My esxi host stopped responding I lost all network through the house! My wireless dropped out and was unable to even ping my router! Once I powered down the esxi host my network came back up!!! I then replicated the problem this morning. Had me laughing as was the weirdest thing Ive ever came across.
Needless to say the board is boxed back up ready to be returned in the morning and I'll have the gigabyte delivered for tomorrow!

I'm going to grab an Intel nic as appears that the gigabyte nic is not supported!

Just to note I did have this running albeit with basic features!
 
Are you sure the Gigabyte NIC isn't supported??? It works on my setup no problem - in fact I'm posting this from the Win7 VM. Don't get me wrong, the Intel cards are really good but unless you need multiple NICs aren't you jumping the gun a bit? If I were you I'd spend money instead on a M1015 so you can put a SAN on a VM - that's my next purchase.
 
Well i have done the install on the intel nic.. the gigabyte isn't actually showing prob a driver issue so will sort that in the coming days.

The new install went through without a hitch! had it up and running and a vm deployed within an hour and half!

Next question.. i have a microserver thats running as my temp server but would like to consilodate alot of my storage into my esxi and run my micro server for weekly backups. so if i did lose data its not the end of the world!!

Ive just installed win server 2012 as i have a licence from work to play with and noticed storage pools. Are these reliable enough to use? i have a supermicro saslp-mv8 card i was planning on using as a passthrough.

Otherwise im thinking to have a freenas vm. Any tips or tricks as im spending the day playing with it the misses has given me a free pass as its partially education related :p
 
Just an update. had major instability problems. belive this was due to the supermicro saslp-mv8.

Found a guide for unraid/esxi that shows how to help passthrough the card. Check this Thread. Helped me out. I havent before managed to get it to stay up for more than 16 hours. So far up to 25-26 hours. Hopefully ive cracked it now.

Running two vm's so far. Windows server 2012 for storage and sab/sickbeard.

2 storage arrays
1- 3x 4tb seagates
2 - 2x 3tb seagates + 2x 2tb samsungs.

These are setup with drive spanning. i know its risky but just to keep it simple. the servers crashed a few times and the drives are still showing fine. I doubt will be the case with RAID.

I have a backup solution for this as well. which is a microserver with 8gb ram, this will be setup with 4-5 2tb drives that will run a weekly backup and core file serving services ready for if/when this goes down while playing with it.

About to setup a couple more VM's for other services and learning.

1 will be for testing out plex on my media centres as i like the central server and transcoding for mobile devices.
 
You must log in or register to reply here.
Back
Top