New hypervisor host build

[BSDMan]

Hi All

I am looking to build a new hypervisor host for my job/career/training/lab. I am in an IT role that involves moving between dozens of clients and keeping up with all the new technologies is a must!

Currently I run the following setup with Hyper-V 2012 R2 (Core):


Mobo: Supermicro X10SL7-F
CPU: Intel Xeon E3-1230 V3 Haswell
RAM: 4 x 8GB DDR3 ECC 1600 (32GB in total)
Case: Fractal Design Mini
PSU: Seasonic 550W
Drives: 2 x 128GB Samsung Pro 840
1 x 256GB Samsung Pro 830
1 x 512GB Samsung Pro 850
Fans: 5 BeQuiet fans (120/140mm) for cooling
IcyDock 5.25" bay to house the 4 SSD drives
HDD: Western Digital 4TB Red


This build has been fantastic for HyperV and ESXi but unfortunately I am limited by the 32GB of RAM. I currently run 10 VMs but am unable to start/create anymore VMs. I run (but am not limited to): Exchange, SQL, Citrix, Remote Desktop Services, Veeam, Active Directory, IIS, etc. I need to run at least another 10 VMs (Lync, System Center, PRTG, vCenter etc).

I have been thinking about the following build:

Mobo: Supermicro X10SRA-F
CPU: Intel Xeon E5-2630 V3 Processor Haswell
RAM: 4 x 32GB DDR4 (128GB in total)
Case: Fractal Design Define XL R2 Black Pearl

I was going to reuse the PSU and the SSD/HDD drives I have BUT the questions I have are:

1)I currently use no RAID and considering how much I rely on this build I would like to have some protection against hard drive failures on the SSDs (all my VMs are on SSD storage and the SATA drive is used for backups only). So what sort of RAID setup should I use for VMs? RAID 1? 10? 5? I don't mind buying more drives

2) I'm keen on using the new PCI Express SSD drives for some VM storage as they are wicked fast. Can one use multiple PCIe SSD drives in a RAID setup? I won't be booting off them and will only be storing VMs on them. Could I use 4 in a RAID 5? I want the system to be snappy/fast to use as I test so much software for my job that I can't afford to sit around waiting hours for something to happen! Should I consider the Intel 750? Samsung 950 PCIe? I'm a bit out of touch with storage.

3) I do have an IBM MegaRAID card which I know works with ESXi (I can't remember the exact model as I type this but it has 512MB cache onboard

Ideally this new setup should work with HyperV and ESXi. My main concern is the storage system. I want the IOPS to be good and want to be able to upgrade the machine as time goes by. I can still add more RAM to this motherboard!

Look forward to repsonses. If I have lefy anything important out then let me know!

 

[Netwerkz101]

Just offering food for thought.

I have the X10SL7-F and it makes for a good storage node.
(I previously ran FreeNAS. Just loaded NAS4Free and added 8 SSDs + 10GbE Mellanox ConnectX3 card.)

Have you thought about keeping it as a storage node and building a new compute node?

In your situation (if only a two node setup), you could add a single port 10GbE card in each node and
directly connect them for good throughput.

8 SSDs on the built in LSI and some large spinners on the AHCI SATA ports.

You could still add a PCIe NVMe card in the Gen. 3 x8 PCIe slot and plug the 10GbE card in the Gen 2.0 x4 slot.

 

[BSDMan]

Thanks for the idea but I would prefer to have everything in a single box. From a practical point of view this has to be in my lounge so has to be quiet and tucked away in a corner.

I was thinking of booting off a USB stick for ESXi and then using RAID 10 for the current SSDs (2 x 128GB and 2 x 512GB) for VM storage off the IBM ServeRAID M5015 SAS/SATA Controller and then maybe having a couple PCIe SSDs in RAID 1 for some more VM storage?

Can you do RAID with the Samsung 950 Pro 256/512GB M.2 PCI-e 3.0 x 4 NVMe Solid State Drive?

What adapter card do you need to connect the PCIe M2 drives to the motherboard?

My current host has 10 VMs running on it and this new build will easily have 20 VMs running on it and I would like it to be able to grow to 25-30 VMs for future projects.

Its just the storage I need to sort out in my mind. I need to get the RAID correct from the beginning and have enough IOPS to be able to run so many VMs. Currently I have 10 VMs spread across 4 SSDs and it runs great but there is no RAID involved so I am at risk if a drive fails (yes I have backups but thats not the point).

 

[piker28]

I would also think your PSU will not work on the new setup.

 

[ToddW2]

If you really plan to run 25-30 VMs you should not be using consumer drives, at all.

 

[Netwerkz101]

For M2 to PCIe card ...:
https://www.google.com/?gws_rd=ssl#q=m2+pcie+pcie+adapter

First two links from Amazon show one Addonics and one Bplus.

AFAIK ESXi won't RAID those for you. Will be seen as individual drives.
Do note: RAID is for uptime ... backups are for recovery.

Current Max performance = PCIe NVMe
Good performance + RAID = standard SSD in RAID10 on your controller.

Booting ESXi via USB = yes

 

[Netwerkz101]

I would also think your PSU will not work on the new setup.

Why would you think that it won't work??
550 watts to start shouldn't be an issue.

 

[Netwerkz101]

If you really plan to run 25-30 VMs you should not be using consumer drives, at all.

Why does the number of VMs determine consumer vs. enterprise?
Is that ust for SSD/Flash based drives vs. spinners?
Because of writes?? MTBF/TF?

I think this was his home lab for testing stuff for work.

I'm asking because I settled on 8 EVOs for mine and will be running at least 30+ VMs.
Are you thinking they won't last ??

 

[BSDMan]

Thanks for all the input!

I would also think your PSU will not work on the new setup.

I'm pretty sure this PSU will work with the new setup. The Xeon is about 85W I think and SSD drives use minimal power.

If you really plan to run 25-30 VMs you should not be using consumer drives, at all.

I am running Samsung Pro SSDs. Yes this is for a lab/home environment but I do run my own email server and host a friends email too. At the rate I replace hard drives (and hardware in general) I doubt these drives would wear out. Also there isn't much load on the system. I think consumer drives are just fine for my setup so long as I stick with the Pro drives.

For M2 to PCIe card ...:

Thanks for those links! Is it possible to do hardware RAID with PCIe drives? I want to try and avoid non-RAIDed drives.

I guess I could not bother with PCIe drives and just stick with regular SATA SSD drives in RAID 10?

I really fo want to try and use PCIe drives for maximum IOPS! But RAID is a must.

I'm asking because I settled on 8 EVOs for mine and will be running at least 30+ VMs.
Are you thinking they won't last ??

I love the Samsung SSD drives! The last 850 Pro drive I bought was rated at 300TB before it would die! I have seen people test them as far as 1000GB! I'm not even remotely worried about my SSD drives wearing out. Seriously, I still have my very first OCZ 120GB SSD drive from over 5yrs ago and its still going strong (it gets used every day in my media centre).

My main goal with this new hypervisor is so that I can upgrade it as the system grows. That means being able to add more RAM and more hard drives for capacity and IOPS. I'm really limited by 32GB of RAM currently and it is preventing me from doing what I need to.

 

[ToddW2]

Why does the number of VMs determine consumer vs. enterprise?
Is that ust for SSD/Flash based drives vs. spinners?
Because of writes?? MTBF/TF?

I think this was his home lab for testing stuff for work.

I'm asking because I settled on 8 EVOs for mine and will be running at least 30+ VMs.
Are you thinking they won't last ??

It's not so much wearing out but performance turning to shit, and staying there with large # of VMs accessing the drive(s).

The steady-state performance of consumer SSD drives slow to a crawl, and honestly even in the enterprise space non-Intel drives don't fair as well either. The garbage collection / ssd performance 'regain' from cleanup/etc on consumer are not designed the same way as enterprise drives are and they need a "break". Consumer drives are made to work really fast in bursts not sustained like enterprise drives. Your 8 EVOs may work great for a period of time and then you're going to wonder where performance went because it never came back. ANAND's reviews of SSDs (mostly Ent) show steady-state / consistent performance after x# Seconds for hours on end.... this is where the Intel S3500, S3700, S3510, S3610, S3710 REALLY SHINE.

The reason I mention the # of VMs is because it's very very likely that with that many doing 'very little' there will still be something always going on in the drive, and a consumer drive simply wont be able to 'catch back up' with how the SSD was meant to be performing.

Depending on the work-load the writes may be an issue too in regards to performance. Consumer drives already fair much worse in a mixed enterprise workload (VMs) than their enterprise counter-parts so I would expect them to perform very poorly if a couple VMs are writing and the rest want to read.

The only consumer drive I've seen hang with the ent. are the Sandisk Extreme Pro. I believe ANAND or some other site has reviews showing this.

I got 2 850 EVOs to put through torture test to see how long they'll work as VM Guest OS drives as well as some really cheap OCZ, older SSD, etc... right now I'm working on my benchmark/testing setup and then I'll have more info.


For a high performing VM and something that's easy to install, works with ESXI, etc... I would look at Fusion-IO PCIE storage on ebay. $150-250$ will yield 320-640GB MLC / MLC DUO drives. These drives can push 1GBs+, were made for enterprise and have insanely high endurance.

 

[Netwerkz101]

<snipped for brevity>

Thanks for those links! Is it possible to do hardware RAID with PCIe drives? I want to try and avoid non-RAIDed drives.

I guess I could not bother with PCIe drives and just stick with regular SATA SSD drives in RAID 10?

I really fo want to try and use PCIe drives for maximum IOPS! But RAID is a must.
<snipped for brevity>

For hadware RAID you need a controller to handle the drives and present them to any OS/ESXi.
I know of no hardware based controllers for these.

Yes .. in your case, since you want Hardware RAID, you can just use standard SSDs with your controller.

I think you are on the right track then ...
the only change I might consider for a single, super host is using a dual proc motherboard like the
Supermicro X10DRH-I or similar.
Can use the lanes, PCIe slots, additional DIMM slots for future gowth.
Just have to worry about sticker shock and licensing fees that may result
because of the extra socket/cores.

 

[Netwerkz101]

<snipped for brevity>

I got 2 850 EVOs to put through torture test to see how long they'll work as VM Guest OS drives as well as some really cheap OCZ, older SSD, etc... right now I'm working on my benchmark/testing setup and then I'll have more info.

<snipped for brevity>

I understand now.
I guess I will be performing my own non-scientific, "real world, real-time" testing.
I always have 8 2.5" spinners to fall back on if forced to.

 

[BSDMan]

I think you are on the right track then ...
the only change I might consider for a single, super host is using a dual proc motherboard like the
Supermicro X10DRH-I or similar.
Can use the lanes, PCIe slots, additional DIMM slots for future gowth.
Just have to worry about sticker shock and licensing fees that may result
because of the extra socket/cores.

To be honest I don't think I need a dual CPU motherboard. I find it difficult to use the current Xeon I have with 10 VMs running! Actually I have never seen it go higher than 50% utilsation! The main concern for me is having the option to add more RAM later on AND to have a good storage system so that I have the IOPS I need for the VMs and am protected from a hardware failure.

I had a quick look at enterprise SSDs and they just seem to expensive for a home setup? I have 2 SSDs that have been running 24x7 for 2yrs and the performance is still fantastic.

My setup won't have much load but will be used quite a bit.

How many VMs should I run on one SSD? At the moment the most I have on one SSD is 5. It runs fine considering my utilsation but I was curious what the limit is before the drive starts to choke?

If you can't RAID PCIe SSD drives should I avoid these then? Pity as the performance is insane!!

 

[Netwerkz101]

To be honest I don't think I need a dual CPU motherboard. I find it difficult to use the current Xeon I have with 10 VMs running! Actually I have never seen it go higher than 50% utilsation! The main concern for me is having the option to add more RAM later on AND to have a good storage system so that I have the IOPS I need for the VMs and am protected from a hardware failure.

I had a quick look at enterprise SSDs and they just seem to expensive for a home setup? I have 2 SSDs that have been running 24x7 for 2yrs and the performance is still fantastic.

My setup won't have much load but will be used quite a bit.

How many VMs should I run on one SSD? At the moment the most I have on one SSD is 5. It runs fine considering my utilsation but I was curious what the limit is before the drive starts to choke?

If you can't RAID PCIe SSD drives should I avoid these then? Pity as the performance is insane!!

The idea for a dual processor setup that I mentioned was not to improve CPU performance.
I mentioned it because a dual proc board allows you to use twice the number of PCIe lanes
as a single proc board (80 vs. 40) which means more PCIe devices of your choice (RAID, PCIe NVMe, etc).

You also get twice the memory slots as stated previously.
You can start with using half the board ... 1 CPU, 8 DIMM slots, and 40 lanes and throw in a second CPU
when ready to use the other half.
You said your goal was to have a single, expandable system.....there ya go!


Enterprise drives = I'm with you ... expensive ... can't afford them for my lab right now either.
I plan to backup regularly - which is something you can/should do whether or not you RAID.

How many VMs should you run on one SSD??? No easy answer .. it depends on the
IO/IOPS load. A domain controller VM won't be much demand like SQL/Exchange VMs.
This is where your own testing/benchmarking on your own equipment comes into play.

 

[BSDMan]

The idea for a dual processor setup that I mentioned was not to improve CPU performance.
I mentioned it because a dual proc board allows you to use twice the number of PCIe lanes
as a single proc board (80 vs. 40) which means more PCIe devices of your choice (RAID, PCIe NVMe, etc).

You also get twice the memory slots as stated previously.
You can start with using half the board ... 1 CPU, 8 DIMM slots, and 40 lanes and throw in a second CPU
when ready to use the other half.
You said your goal was to have a single, expandable system.....there ya go!


Enterprise drives = I'm with you ... expensive ... can't afford them for my lab right now either.
I plan to backup regularly - which is something you can/should do whether or not you RAID.

How many VMs should you run on one SSD??? No easy answer .. it depends on the
IO/IOPS load. A domain controller VM won't be much demand like SQL/Exchange VMs.
This is where your own testing/benchmarking on your own equipment comes into play.

You bring up some good points so thank you for that! ;-)

The number of PCIe lanes is not something I have thought about. How many lanes should I be thinking about for my setup? I know this is a vague question but for PCIe I will possibly be using my IBM RAID card, an SSD NVMe or two and maybe a video card (for XenDesktop HDFX Pro 3D).

I really like that motherboard you mentioned and its not much more than the single CPU board I originally mentioned. Do you know if it'll run ESXi 6? Supermicros OS compatability list always seems to be a bit behind! Can you recommend a case? I was thinking of getting another Fractal Design but the big one.

I have never owned a dual CPU board before so considering the board you mentioned has 16 memory slots, does this mean 8 slots get used per physical CPU? If I have 2 CPUs in the board do I have to ensure the slots are used evenly? Or is the memory shared between the two CPUs? Will a 550W PSU be enough for two CPUs? The Intel site says the Xeon I want uses 85W so I think I will be ok (my current E3-1230v3 CPU is rated at 80W and is fine). But I may add a video card too which will eat up some watts...

I do backup regularly so thanks for the tip! I use Veeam currently with an NFR license and it is fantastic. Its runs daily and I have a monthly job too. All the backup files sits on a Windows Server 2012 R2 dedupe drive so I have about 700GB of backups using only 180GB! The savings with dedupe are incredible. I also run a one off Veeam job occassionally to an external USB drive so that I have an "offsite/offline" copy of my backups.

I was just curious what other people found when running multiple VMs on one SSD. My VMs won't be that busy or under load most of the time. I have run up to 5 or 6 VMs on one SSD and it was fine. Is it reasonable to say you could run 30 VMs on 4 512GB Samsung Pro 850 SSDs in RAID 10?

Theres so much I want to do with this server in my lab environment so I want to ensure that the correct hardware is purchased considering the amount being spent. I will say that I am incredibly happy with my current Supermicro board. Been running it 24x7 for 2 years now and it runs flawlessly and have run FreeBSD, HyperV and ESXi 5.5 with no issues.

 

[Netwerkz101]

You bring up some good points so thank you for that! ;-)

The number of PCIe lanes is not something I have thought about. How many lanes should I be thinking about for my setup? I know this is a vague question but for PCIe I will possibly be using my IBM RAID card, an SSD NVMe or two and maybe a video card (for XenDesktop HDFX Pro 3D).

The number of lanes needed depends on the add in cards used.
Example: A x8 RAID card + a x4 PCIe based NVMe card + x16 Video Card = 28 lanes needed.

You are likely to run out of slots before lanes on most setups but Supermicro
has a board with 11 PCIe slots too ...though it's a proprietary form factor.

I really like that motherboard you mentioned and its not much more than the single CPU board I originally mentioned. Do you know if it'll run ESXi 6? Supermicros OS compatability list always seems to be a bit behind! Can you recommend a case? I was thinking of getting another Fractal Design but the big one.

I do not own that board so no first hand experience with it or it's compatibility with ESXi.
I have the X10SRi-F which has same C612 chipset and Intel 350 NICs and ESXi is fine on it.

The board mentioned is an E-ATX form factor and the Fractal Design R2 XL you mention looks to support E-ATX.

I have never owned a dual CPU board before so considering the board you mentioned has 16 memory slots, does this mean 8 slots get used per physical CPU? If I have 2 CPUs in the board do I have to ensure the slots are used evenly? Or is the memory shared between the two CPUs? Will a 550W PSU be enough for two CPUs? The Intel site says the Xeon I want uses 85W so I think I will be ok (my current E3-1230v3 CPU is rated at 80W and is fine). But I may add a video card too which will eat up some watts...

<snipped for brevity>

Yes .. 8 DIMM slots per CPU.
SuperMicro documents how to configure DIMMs based on 1 or 2 CPU setups.
While you can have an odd number off DIMMs, performance is best in pairs.

From what has been mentioned so far..minus the video card...I'd say 550W is fine.
You'd need to research the specs on your video card's power draw to determine if
a higher rated power supply is needed.

I was just curious what other people found when running multiple VMs on one SSD. My VMs won't be that busy or under load most of the time. I have run up to 5 or 6 VMs on one SSD and it was fine. Is it reasonable to say you could run 30 VMs on 4 512GB Samsung Pro 850 SSDs in RAID 10?

<snipped for brevity>

Yeah...I'm still not sure how to figure this out without actual trial/error testing as
it depends on the loads you are putting on them.

 

[BSDMan]

I thought I would throw a spanner in the works! After some more research I found this website:

https://tinkertry.com/

On the website the following is used as an ESXi 6 host:

Supermicro SuperServer SYS-5028D-TN4T Xeon D-1540


This little server (and it is small!) has amazing power for its size like:

1) Xeon CPU with 8 cores
2) Support for up to 128GB of RAM
3) The case can take up to 7 SSD drives (one of which has to be PCIe)...if you want you can use the four 3.5" bays for non-SSD drives
4) Two 10gb NICs and two 1gb NICs (plus IPMI)

So this got me thinking, why not use one of these for my ESXi host? I have the drives I need so after purchasing this server I'll have everything I need. The small size is great and it is VERY energy efficient.

Bu then I thought about it a bit more, why stop at one of these small servers?

Yes, I thought buying one of these would be a great start and then later on I could add a second server and start using the clustering. Two of these servers would be slightly more than the mega build I had in mind that was mentioned earlier in this post but then I would have a cluster with 256GB of RAM and 10gb NICs!

Downside? No RAID built in. Not even sure if my RAID card would fit but that also got me thinking about storage again: Why bother with RAID? So what are the options that have come to mind so far?:

1) Use vSAN: My knowledge of this is minimal and you need a minimum of 3 hosts but the latest version supports 2 hosts with a witness applicance. I'm assuming you could lost an entire host or a disk in the host and the VM(s) would continue running. I need to research this more.

2) Use my current "old" HyperV server as a Nutanitx node running Community Edition. Also don't know much about this so I need to rearch further.

Ideally I would like to have all local storage in the two small Supermicro SuperServers mentioned on TinkerTry and not bother with shared storage PROVIDED the VMs will continue to run in the event of a hardware failure.

Although using my exising server is an option in the new setup going forward I would like to avoid it if possible.

So I'm back to the area I am unsure of: How should I setup/use the storage in the Supermicro SuperServers mentioned on TinkerTry if I could this route?

This option I mention here is so different from what I had in mind originally. Rather than have one mega beefy expensive server, this option allows for many smaller (but still powerful) servers that allow you to scale out should you need more resources (RAM!). Running 3 or 4 of these where my server is currently would be no problem due to their size. And, they are VERY energy efficient.

Appreciate any thoughts or feedback. I'm still struggling with the storage side!

 

[Netwerkz101]

Don't get me wrong ... but what was it that you wanted again???

In post #1 in this thread:

<snipped>

Ideally this new setup should work with HyperV and ESXi. My main concern is the storage system. I want the IOPS to be good and want to be able to upgrade the machine as time goes by. I can still add more RAM to this motherboard!
<snipped>

See post #2 in this thread ...

Then this:

Thanks for the idea but I would prefer to have everything in a single box.

<snipped>

and then see my post in another thread....


See this:
http://hardforum.com/showpost.php?p=1042051906&postcount=1537

 

[Netwerkz101]

I love what the Supermicro X10SDV-TLN4F is, but expandable it is not.
A perfect compute node in a multi-node setup.
A good candidate for an all-in-one nested lab setup.

But...what are you going to do with that RAID card?
The Video Card?
The AIC PCIe NVMe drive (though on-board M.2 slot might suffice).
That CPU is soldered.

It's wrong for "upgradable" / "Expandable".

 

[BSDMan]

Hehe, great post!

I *thought* I wanted a single mega powerful (expensive) single server that I could just keep throwing VMs at.

After reading Tinkertry and giving this some thought, I think having lots of small servers (which are still quite powerful IMHO) is better (and cheaper longer term) as I can scale out. This will allow me to have a cluster with 2 or more nodes and grow the system that way rather than having to replace a big server. The Supermicro Super Server is tiny. VERY tiny. So having 3 or 4 of these in my lounge where my existing server is will take up the same space!

It can take a video card but I still need to find a video card that will work with XenDesktop 7.6 HDFX 3D Pro. I know on Tinkertry he has a video card in his setup and runs 3 monitors from it.

It does have a slot on the motherboard for one PCIe drive.

I know I'm a bit all over the place but I'm just trying to find the right solution for my needs so am willing to consider all possibilities.

The Supermicro Super Server sure is looking like a good bet. I know the CPU is soldered to the motherboard but so what, just add another node in the cluster and so you grow the system. I like the fact that these servers are very energy efficient.

Hopefully I won't need to use my RAID card or even use RAID. If I use vSAN then I can probably get away with using single drives (ie non RAIDed) in each server node. I still need to look into this as I don't know too much about it at the moment.

This projects got me excited about this now. 128GB of RAM is gonna go a long way!

 

[Mackintire]

I read the OPs post and then thought of my VMware setup at work.


Each host is a HP DL380G9:

(2) x 10 core Xeon E5-2660 v3 w/hyperthreading enabled
768GB of RAM
(2) x dual Intel X520 ethernet
(2) x 300GB SAS


Connected via iSCSI to a Dell SC4020 paravirtualized SAN with 120TB of usable storage (20TB usable) of that is SSD

We have 5 host boxes and 300-350VMs running at any one moment.


Run's fairly well.

 

[TheSocialHermit]

I'm definitely interested in this project.

 

[BSDMan]

Can anyone recommend a low profile video card that will work in the Supermicro server AND allow me to share it with XenDesktop 7.7 AND Horizon View?

Ideally I am looking for a video card that can be shared between 2 or 3 VMs. I don't need a powerful video card as I just need it to be able to handle 1080p video playback within a XenDesktop and/or Horizon View desktop. Is it possible to share one video card amongst multiple VMs on ESXi?

Failing that then I think I will need a low profile video card that can be "passed through" to a single VM for 1080p video playback.

Thanks!