Time for a new home VM Server

[NoOther]

So I retired my old VM Servers about 2 years ago and I am looking at building a new Virtual Server for home. I would appreciate some input and some examples of what people are using now.

Purpose: To quickly build and test various OS's, appliances, and combinations of systems. The main idea for this server is to have a few constant VMs as well as a number of test VMs. The constant VMs that I will use for home for media storage/media server, general data storage, a new home firewall, and general purpose OS (browsing, chat/voice programs, email, etc). The test VMs will consist of network appliances, myriad of OS options, some proprietary VM appliances for training, and a lot of experimental integration concepts.

General Notes:

Physical: I have a desk that supports anywhere up to 12 monitors, right now I have 3 for my current gaming rig, I will mount 3 more above it for the VM system. I do not have a rack system in place so the VM server will be a tower/ full tower desktop case (have not determined that yet, so suggestions are welcome). I plan to also buy an additional system to access the VMs so I don't have to use my gaming rig.

Setup: I plain to use Raid, ZFS, PFsense, and probably Linux home media server. Suggestions on how people currently have these setup and recommendations for them are appreciated. I have not used ZFS much before( I have used it sparingly on other systems in the past, but did not administer it), so it will pretty much be my first time working with it. I have experience using many other file and storage systems though.

Appliances/Applications: I have worked with quite a range of devices and systems in my line of work. The purpose of my home system is to continue to stay near the top of the curve. I have access to a number of various network appliances. I will be testing a number of network firewalls, packet inspection tools, logging and dashboard systems, privilege management systems, etc. I will also be setting up and testing various configurations for Linux and Windows systems.

Resources: I plan to continually run about 3-4 VMs. The main VMs I will keep up are my firewall, media server, media/file storage, and at least one test desktop VM. In addition to that I will need resources to run at least 2-3 other appliances. These appliances will mainly be network appliances, but I should have enough resources left over from my main 4 VMs to run at least 1 DB server and 1 Application server and 1 network appliance at the same time.

Optional: I would like to test out GPU passthrough, but this is not a deal breaker. I know it can be a bit of a pain sometimes, but I do have some interesting ideas if I can get it to work, like creating a gaming server to host games and stream them to my projection system. Currently I am looking at a single server, but in the future I may add an additional server for HA/FO. I also my buy a small rack system in the future, so the parts will be moved from the tower desktop to a rackmount case at some point.

Budget: My budget is fairly open, but I don't want to completely break the bank on the hardware. I would like reliable hardware with some versatility.

Brands: I generally don't care about brands, just want the best bang for the buck that is reliable and fairly high performing.

Extra: Any additional ideas people have for the general setup, appliances to test out, concepts or anything at all, please feel free to share. This initial server will be for a little more limited testbed. In a year I am looking to broaden it and will be testing and working on more distributed computing concepts. I plan to work with systems like hadoop and puppet to build a base platform and work from there. Anyone who already has some experience with playing around with that kind of stuff at home, I would appreciate some ideas there as well. I have worked on a large scale hadoop installation in the past, but mainly from an administrative/engineering perspective and not really from a development perspective.

 

[/dev/null]

Get a DL360 G6 from ebay for a couple hundred dollars.

For VM software, I reccomend KVM (either in your favorite linux distro) or in a baremetail type installer (proxmox ve), XenServer and esxi in that order.
http://www.ebay.com/itm/HP-Proliant...867749?pt=COMP_EN_Servers&hash=item48635fcba5
http://www.ebay.com/itm/HP-Proliant...309007?pt=COMP_EN_Servers&hash=item3f4018d58f
http://www.ebay.com/itm/HP-Proliant...250082?pt=COMP_EN_Servers&hash=item51c60f6aa2
http://www.ebay.com/itm/HP-ProLiant...085883?pt=COMP_EN_Servers&hash=item4d26a8e9bb

 

[/usr/home]

My list is the exact opposite of Robstar's. ESXi all the way. Run a real bare hypervisor, not KVM.

 

[lopoetve]

ESXi or XenServer all the way. I second /usr - run a real hypervisor.

 

[Netwerkz101]

So I retired my old VM Servers about 2 years ago and I am looking at building a new Virtual Server for home. I would appreciate some input and some examples of what people are using now.

Purpose: To quickly build and test various OS's, appliances, and combinations of systems. The main idea for this server is to have a few constant VMs as well as a number of test VMs. The constant VMs that I will use for home for media storage/media server, general data storage, a new home firewall, and general purpose OS (browsing, chat/voice programs, email, etc). The test VMs will consist of network appliances, myriad of OS options, some proprietary VM appliances for training, and a lot of experimental integration concepts.

General Notes:

Physical: I have a desk that supports anywhere up to 12 monitors, right now I have 3 for my current gaming rig, I will mount 3 more above it for the VM system. I do not have a rack system in place so the VM server will be a tower/ full tower desktop case (have not determined that yet, so suggestions are welcome). I plan to also buy an additional system to access the VMs so I don't have to use my gaming rig.

Setup: I plain to use Raid, ZFS, PFsense, and probably Linux home media server. Suggestions on how people currently have these setup and recommendations for them are appreciated. I have not used ZFS much before( I have used it sparingly on other systems in the past, but did not administer it), so it will pretty much be my first time working with it. I have experience using many other file and storage systems though.

Appliances/Applications: I have worked with quite a range of devices and systems in my line of work. The purpose of my home system is to continue to stay near the top of the curve. I have access to a number of various network appliances. I will be testing a number of network firewalls, packet inspection tools, logging and dashboard systems, privilege management systems, etc. I will also be setting up and testing various configurations for Linux and Windows systems.

Resources: I plan to continually run about 3-4 VMs. The main VMs I will keep up are my firewall, media server, media/file storage, and at least one test desktop VM. In addition to that I will need resources to run at least 2-3 other appliances. These appliances will mainly be network appliances, but I should have enough resources left over from my main 4 VMs to run at least 1 DB server and 1 Application server and 1 network appliance at the same time.

Optional: I would like to test out GPU passthrough, but this is not a deal breaker. I know it can be a bit of a pain sometimes, but I do have some interesting ideas if I can get it to work, like creating a gaming server to host games and stream them to my projection system. Currently I am looking at a single server, but in the future I may add an additional server for HA/FO. I also my buy a small rack system in the future, so the parts will be moved from the tower desktop to a rackmount case at some point.

Budget: My budget is fairly open, but I don't want to completely break the bank on the hardware. I would like reliable hardware with some versatility.

Brands: I generally don't care about brands, just want the best bang for the buck that is reliable and fairly high performing.

Extra: Any additional ideas people have for the general setup, appliances to test out, concepts or anything at all, please feel free to share. This initial server will be for a little more limited testbed. In a year I am looking to broaden it and will be testing and working on more distributed computing concepts. I plan to work with systems like hadoop and puppet to build a base platform and work from there. Anyone who already has some experience with playing around with that kind of stuff at home, I would appreciate some ideas there as well. I have worked on a large scale hadoop installation in the past, but mainly from an administrative/engineering perspective and not really from a development perspective.

So much was typed, yet nothing specific.

Purpose - okay you want to build a system (vs. buy). Got it!
- Don't know what "a lot of experimental integration concepts" means.
- Does not seem like there is anything special about your 24/7 VMs (except ZFS setup)

Physical - okay you don't have a rack.
- Do you want to buy a case twice?
- Are you sure you are going to convert to a rack later?
- A rack mount system can sit on a desk
- A tower system can sit in a rack

Setup - okay ....run ZFS.
- In a VM means passing controller to OS running ZFS
- So we are talking "all-in-one" setup now???
- ZFS = prefer ECC memory
- But now how to do HA/FO testing if storage is local??? You want versatility = get shared storage.

Appliances/Apps - I don't know enough about it ..but nothing seems special.
- VMware ESXi is known to have large amounts of virtual appliances available - makes for a good first choice in hypervisor.
- Citrix XenServer has some too ..not as many.

Resources - you would need to add up memory use for all your expected VMs running at same time:
- if > 32GB then 2011 server based motherboard (sticking with server grade)
- if <= 32GB then you have more options (like newer 1150 boards)

Optional - no experience 'cause I can't afford the cards (think XenServer HCL)
- both ESXi and XenServer have improved over time in this area (just from my reading - prefer XenServer for this)
- Versatility = you can multi-boot hypervisors.

Budget - fairly open but don't break the bank??
- You have millions to spend but you want to dedicate $100 to this build ????
- My guess is you can stay below $1000 for a really nice system ...but again ..depends on "specifics".

Brands - use known reputable brands
- I've used Supermicro for my server builds - no issues. Recommended
- I've used Intel for my Desktop builds - no issue. Recommended.

Extra - no experience in that stuff

 

[NoOther]

To give a bit more clarification on the purpose its three fold. Purpose one is to have a few generic VMs I use 24/7 for personal home use. Those will be a media server, file storage, firewall and general use VM for browsing/chatting/etc. The file storage may be relegated to a NAS as may the storage of the VMs themselves. I prefer to store and run some of the VMs locally for better performance. The final reason is I want something that is DIY, not store bought, not enterprise class, outside the mainstream. I will need more resources to test out the ideas I have, but the end result should be that the end integrated systems I configure should run on far less resources, or have higher scalability.

To explain the goal here, many of the appliances I have used over the past few years in my profession have started moving to Virtual appliances as opposed to just being sold as stand alone network appliances. Many of them are also coming with more API links to use with other programs. Right now I have about 5 different projects that all have some ways to interconnect, but haven't really been designed or configured for it. There are about 15 different network appliances in these programs, almost all of them offer a VM for their services. The intent is to see how well these VMs actually work compared to the standalone devices, and if I can get them all to work better together rather than being mostly separate. So the goal is to centralize all the appliances and then integrate them into one whole integrated system rather than 5 different systems. All of these systems consist of network defense/monitoring capabilities.

So much was typed, yet nothing specific.
Purpose - okay you want to build a system (vs. buy). Got it!
- Don't know what "a lot of experimental integration concepts" means.
- Does not seem like there is anything special about your 24/7 VMs (except ZFS setup)

There is nothing special about the 24/7 VMs, they are just going to be standard fare for my home use. The experimental integration concepts is purposely vague, as they are "experimental". I mentioned I would be using a combination of special use appliances such as

network firewalls, packet inspection tools, logging and dashboard systems, privilege management systems, etc.

Even you need more specifications, think Palo Alto, F5, Arbor, Splunk, Arcsight, Cisco, Juniper, etc. The purpose of such a venture is the intent to develop a system that would more successfully integrate all of these appliances into a single system. There are many options out there and I am looking to find which ones work best together or can be made to work better together. I don't have any preconceived notion on how I will go about it as that never seems to work in favor of a good design.

Physical - okay you don't have a rack.
- Do you want to buy a case twice?
- Are you sure you are going to convert to a rack later?
- A rack mount system can sit on a desk
- A tower system can sit in a rack

Yes, I want to buy a case twice.
No, but it is very likely.
I am well aware, but I have other reasons for buying the case now and a rack later.
That is debatable. Can it? Yes, but that is just silly.

Setup - okay ....run ZFS.
- In a VM means passing controller to OS running ZFS
- So we are talking "all-in-one" setup now???
- ZFS = prefer ECC memory
- But now how to do HA/FO testing if storage is local??? You want versatility = get shared storage.

Not necessarily. I may run ZFS within a VM that I tie to my 24/7 group. I may not necessarily run it for the ESXi server or whatever hypervisor I choose to use. In fact it is likely I will not use it for that. More than likely I will end up using a NAS.

Appliances/Apps - I don't know enough about it ..but nothing seems special.
- VMware ESXi is known to have large amounts of virtual appliances available - makes for a good first choice in hypervisor.
- Citrix XenServer has some too ..not as many.

From the 24/7 group nothing is special. But from the group of VMs I am testing, they are all special. Most of the ones I will be testing have appliances built for VMware, so that is likely what I would use.

Resources - you would need to add up memory use for all your expected VMs running at same time:
- if > 32GB then 2011 server based motherboard (sticking with server grade)
- if <= 32GB then you have more options (like newer 1150 boards)

Great than 32GB most likely. I am thinking dual xeon chips with 64GB of RAM.

Optional - no experience 'cause I can't afford the cards (think XenServer HCL)
- both ESXi and XenServer have improved over time in this area (just from my reading - prefer XenServer for this)
- Versatility = you can multi-boot hypervisors.

I would prefer not to multi-boot, if anything I would tear down the whole setup and rebuild with a different hypervisor if I didn't like the one I was using.

Budget - fairly open but don't break the bank??
- You have millions to spend but you want to dedicate $100 to this build ????
- My guess is you can stay below $1000 for a really nice system ...but again ..depends on "specifics".

Meaning I can afford over $5k, but I don't want to just be spending money to get some ultra expensive item if it only provides me a modicum of improvement over a much cheaper item. For instance, I really like Xeon chips, but if I can get similar VM server performance out of AMD chips and save a lot of money, then I will. The thing is there are a lot of things to think about as far as how all the hardware works together. The types of hard drives to use for storage and operation, the RAM, the RAID cards, best compatible GPU for passthrough, etc.

Brands - use known reputable brands
- I've used Supermicro for my server builds - no issues. Recommended
- I've used Intel for my Desktop builds - no issue. Recommended.

I have used and like Supermicro, but they sometimes have issues with certain configurations. I may want to use a combination of SSDs and HDDs.

Extra - no experience in that stuff

The additional intent of the "extras" is mainly for other integration ideas to test in the VM environment. I have worked on large distributed systems before, but I don't currently have any idea of what I want to do with that. And the cost will be much higher considering it will require more servers. I have thought about working with some people to create a cloud gaming system that could centrally serve a house, group of homes, or even an apartment complex or dorm. That is part of the reason for wanting to play with passthrough. So far passthrough hasn't come far enough for the concepts I had, but perhaps there are other ways to make it work. Obviously there have already been some companies out there working on this concept. The idea here is if I can find a way for the not so average Joe to configure their own. In a work environment I have already been able to do this with many applications at a much lower cost for the company, but not so much for the average consumer.

 

[Netwerkz101]

First .. I am starting to think I need to relax on the coffee a bit.
It looks like it's time for me to spend more time blogging
I hope my initial response didn't seem too crazy or offensive - not my intent.
My replies are simply my thoughts on how to help you ...at the end of the day
I'm just another guy on the Interweb with an opinion.

This single statement:
"But from the group of VMs I am testing, they are all special. Most of the ones I will be testing have appliances built for VMware"

^^ Pretty much sums up ESXi is your hypervisor.of choice then.

Good luck with your build, and please post up with what you end up with..
I am always curious as to what others build out there.

 

[NoOther]

First .. I am starting to think I need to relax on the coffee a bit.
It looks like it's time for me to spend more time blogging
I hope my initial response didn't seem too crazy or offensive - not my intent.
My replies are simply my thoughts on how to help you ...at the end of the day
I'm just another guy on the Interweb with an opinion.

No worries, I didn't really take it that way. It was a lot of stuff I had in there and some things I was intentially vague on. etc.

This single statement:
"But from the group of VMs I am testing, they are all special. Most of the ones I will be testing have appliances built for VMware"

^^ Pretty much sums up ESXi is your hypervisor.of choice then.

Good luck with your build, and please post up with what you end up with..
I am always curious as to what others build out there.

Hypervisor I am not so much worried about as hardware capability issues with certain things like ZFS, Hadoop (for future consideration), GPU passthrough,

 

[nicholasfarmer]

You may already have your answer above but I started out with large expensive servers for my home lab. The power bill started to get out of hand....
I now run a wad of VMs across some Intel i5 NUCs. My DMZ nuc uses 20 watts and is running two DB servers, two web servers and a small test vm I use to pass scripts and data between the network segments. A single NUC can run four cores, 16gb of ram, two wired Gbps adapters, and two drives (1TB msata and a 2TB spinner for space and IOPS)

Also to add... I run my 24/7 VMs (syslog, small file server, test domain, DNS, proxie, firewall, etc) on a single NUC. Quiet, cool, 20 watts.... Easy to buy a replacement if it breaks.....

Just some data to consider.
Nick

 

[bebble]

For the virtualization software part, there is a discussion in thie board about "ESXi vs hyper-v". I personally got a esxi whitebox, and it runs pretty stable.

 

[int]

You may already have your answer above but I started out with large expensive servers for my home lab.

Mind answering a couple questions about your setup? I just started down the same path as OP, but I'm realizing that a bare hypervisor (and all the specialized hardware that goes with it) is probably overkill for me (a .NET developer who just wants to learn more about Linux/DNS/pfSense/Docker/etc.). I'll have a few 24/7 VMs like you have for non-critical things, but otherwise just toys/learning VMs.

In lieu of a bare hypervisor, what do you use? Do you just throw VirtualBox (or similar) on the main disk and then remote in when you need to set up new VMs, or is there a fancier solution than that?

Which NUC are you using that has 2 NICs? It looks like the beefiest one Intel puts out is this feller: http://amzn.com/B00HZDLNWO but it's only got one NIC and ships with a dual core, not quad.

 

[NoOther]

Mind answering a couple questions about your setup? I just started down the same path as OP, but I'm realizing that a bare hypervisor (and all the specialized hardware that goes with it) is probably overkill for me (a .NET developer who just wants to learn more about Linux/DNS/pfSense/Docker/etc.). I'll have a few 24/7 VMs like you have for non-critical things, but otherwise just toys/learning VMs.

In lieu of a bare hypervisor, what do you use? Do you just throw VirtualBox (or similar) on the main disk and then remote in when you need to set up new VMs, or is there a fancier solution than that?

Which NUC are you using that has 2 NICs? It looks like the beefiest one Intel puts out is this feller: http://amzn.com/B00HZDLNWO but it's only got one NIC and ships with a dual core, not quad.

For just testing out VMs, I generally use VMware Player or VirtualBox. Personally I like VMware Player better, I have had a few issues with VirtualBox.

 

[bebble]

It might be off the topic. Is GPC passthrough a requirement for building game server?

 

[NoOther]

It might be off the topic. Is GPC passthrough a requirement for building game server?

GPU passthrough? It depends on what you want from the game server. In my particular project it is. If you are just looking for something that will host games, it isn't really a requirement. If you want to be able to play 3D accelerated games on a VM, then yes, it is a requirement.

 

[tycoonbob]

I used to be big on Hyper-V, but have moved to KVM and haven't looked back. Using management software on KVM (oVirt, Proxmox, etc) makes for a nice package, that's free with no limits like ESXi. I still like Hyper-V, just don't have a need to run it any longer.

I personally avoid XenServer like the plague. Bad experiences at a previous employer where we had 12 8-node XenServer cluster. Talk about PTSD..

Hardware wise, you just can't beat the cost of a HP DL360 Gen6 or Dell C1100 or R610 from eBay. Find one with dual Xeon L5520 CPU's, and 24GB of RAM and it will run you $400-500. You just can't beat it. Power draw will be somewhere between 110-140W, which is quite reasonable for a powerful home server. When it comes to virtualization, you have 3 main limitations that you may, or may not, run into. Storage I/O, amount of RAM, CPU Cycles. You don't need the latest and greatest CPUs.

Just my .02 from personal experiences.