Server for Virtualized pfSense / HTPC / Windows Servers / etc

[BfA]

I'm building a new server that will be running multiple VMs 24x7. It will also be used as the point of entry for Internet into the residence so it's important it runs very stable. The computing won't be very hardcore, however, I am building it so that it will hopefully last at least seven years and so I want to be able to do incremental upgrades during that time. For instance moving to NVME based SSD / higher capacity HDDs / a higher level video card. As such, the motherboard, cpu and power supply are over provisioned with that in mind.

2) What's your budget? Are tax and shipping included?
3k, yes

4) What exact parts do you need for that budget?
case, power supply, motherboard, cpu, heatsink/fan, memory, m.2 ssd, 2 x 8tb hdds, video card capable of being virtualized

6) Will you be overclocking?
No

7) What is the max resolution of your monitor?
N/A, HTPC will output 1080p to 60" tv

8) When do you plan on building/buying the PC?
Now

9) What features do you need in a motherboard?
e5 v4 support, m.2 (two would be great), ipmi or a way to access it as a headless server, preferably matx, mitx


VMs Running 24x7
- pfSense as Internet router (I will have a hardware switch as well)
- Windows Nano Server 2016 Domain Controller
- Windows Nano Server 2016 DNS
- Windows Nano Server 2016 DHCP
- Ubuntu
- HTPC (Ubuntu / Windows / ?) - will be used for Kodi, playing all media files, recording IPTV, light 1080p gaming
- Windows 10
- either Windows Nano Server 2016 to run the file share

I'm aware I can do most of the networking things through pfSense, however, I want to also be able to use this for education so I can spin up multiple server VMs at the same time.

Hardware Passthrough

- 2 x HDD to one of the VMs to be used in a ZFS or ReFS mirror, the file share will be used as data drives for the other VMs when needed. This data is very important and my main goal is keeping it safe and secure.

- video card will be passed through to the HTPC and the HDMI cable will be connected to a receiver for processing

Chosen Parts

Case: ? - I need something small that looks good in a media center

Power Supply: EVGA SuperNOVA P2 650W 80+ Platinum Certified Fully-Modular ATX Power Supply
Note: Motherboard says "Support 12V only power supply"

Motherboard: Supermicro MBD-X10SRM-F-O

CPU: Intel Xeon E5-2620 V4 2.1GHz 8-Core Processor

Heatsink / Fan: Noctua NH-L12 37.8 CFM CPU Cooler

Memory: ? - 32GB needed of ECC & unbuffered, will 2400 work? Dual channel should be fine.

Data Drives:
Supermicro SSD-DM016-PHI SATA DOM (SuperDOM) Solutions
What is a SATA DOM?
Note: The core VMware / Hyper-V installtion will be run from this drive. I could run from USB stick but I think this will be more stable long term and the SATA DOM would be higher quality than a USB drive.

Samsung 850 EVO 500GB M.2-2280 Solid State Drive
Note: VMs will be run from this drive

2 x Western Digital Red 8TB 3.5" 5400RPM Internal Hard Drive
Note: To be mirrored via software. These will be used for data only, some of it shareable between VMs

Video Card: ? (I'm thinking RX 460 as the GeForce cards aren't virtualizable)

So if anyone managed to read that thanks! If you have any ideas on how to improve it, even better!

 

[rhansen5_99]

I suppose the first question is what is your hypervisor going to be, Hyper-v, VMware ESXi, Xenserver, or something else?

A few suggestions would be to take advantage of the quad channel memory controller on the 2011-v3 platform, and I think you might saturate a standard m.2 sata type drive like the 850 evo, especially if you have a lot of activity rolling around. I would maybe look at a pcie or nvme drive as a replacement for that.

If you are not planning on actually using the 8 cores of the e5-2620, you might look at the socket 1151 Xeons like the e3-1245v5 Intel Xeon E3-1245 v5 SkyLake 3.5 GHz 8MB L3 Cache LGA 1151 80W BX80662E31245V5 Server Processor - Newegg.com, you will get a nice boost in IPC per core moving up to skylake, and you could use the igpu until you make a gpu decision (if you even need it). I would consider the rx460 if you need the h265 encode / decode block for media/ a hdmi 2.0b port for 4k60 to a TV.

You could even go itx (here) if you were okay with 32gb ddr4 limit with 1151 (note no m.2 on this one)

 

[Deleted member 88227]

If I were you I'd use a separate box for the HTPC. I too once tried to run everything off a system that was also an HTPC, but later found that my money went a lot further if I just separated them into two different boxes. Now I have a very capable gaming HTPC that runs Plex and a server that's doubled as a NAS and PMS. Just going with a 6 core HT processor would instead of the quad core in my setup would be easy. I think you could easily achieve that with your $3k budget.

My HTPC case is a Fractal Design Node 605 and the NAS/PMS/Server case is a Fractal Design Refine R5. Both of which have a nice, sleek black design that looks at home in a media center. Although you could move the server itself to a different room.

 

[BfA]

I suppose the first question is what is your hypervisor going to be, Hyper-v, VMware ESXi, Xenserver, or something else?

Most likely Hyper-V 2016.

A few suggestions would be to take advantage of the quad channel memory controller on the 2011-v3 platform, and I think you might saturate a standard m.2 sata type drive like the 850 evo, especially if you have a lot of activity rolling around. I would maybe look at a pcie or nvme drive as a replacement for that.

I would take advantage of the quad channel memory controller, however I've read that there's not much benefit going from dual to quad channel like there is from single to quad. I'm aware I could saturate the 850 Evo, however, I think to start off with it should be fine. The reason I'm starting with that SSD is that I think the 950 Pro's are over priced and 60-80% of the time only the pfsense VM and perhaps the HTPC VM will be doing anything. I also like to have at least one upgrade step on each box I put together so getting a newer video card, RAM and SSD in a few years would make a great upgrade at a much better value.

If you are not planning on actually using the 8 cores of the e5-2620, you might look at the socket 1151 Xeons like the e3-1245v5 Intel Xeon E3-1245 v5 SkyLake 3.5 GHz 8MB L3 Cache LGA 1151 80W BX80662E31245V5 Server Processor - Newegg.com, you will get a nice boost in IPC per core moving up to skylake, and you could use the igpu until you make a gpu decision (if you even need it). I would consider the rx460 if you need the h265 encode / decode block for media/ a hdmi 2.0b port for 4k60 to a TV.

I chose the E5 because I have a seperate gaming rig i5-6600 that I will be using for gaming and most likely connecting to all the VMs. I thought hard about the e3's but if I'm going to have this for 5-10 years more cores should allow more options I believe if I'm not doing anything that requires a high MHZ per core. I've also read that the E5's provide more compatibility with hypervisors like Xen etc at least previously. I'd be very interested in a reason to go E3 v5 though as I'm still open to it.

I think I've decided on the RX 460 for the GPU. It should do everything I need for my 1080p environment. If I decide to go to 4k in a few years I can get a new one. It would also probably give me a boost for light gaming in 1080p versus the 530 graphics on the Skylake E3.

You could even go itx (here) if you were okay with 32gb ddr4 limit with 1151 (note no m.2 on this one)

I'm open to any mobo, but I would like to have m.2 on it and IPMI as it will likely be setup as a headless server that I will remote into from my gaming PC when needed.

 

[BfA]

If I were you I'd use a separate box for the HTPC. I too once tried to run everything off a system that was also an HTPC, but later found that my money went a lot further if I just separated them into two different boxes. Now I have a very capable gaming HTPC that runs Plex and a server that's doubled as a NAS and PMS. Just going with a 6 core HT processor would instead of the quad core in my setup would be easy. I think you could easily achieve that with your $3k budget.

I have a separate gaming PC so really the only VM I would be running that would have output to a real world device would be the HTPC VM. The server will be right under my TV so that's the reason for me wanting to run an HTPC VM there. I understand what you mean about more bang for your buck doing two different boxes and I could but I don't think I would gain much since it's just going to be playing media and light gaming on the RX 460. Having the hdd's in the same box but given to a different VM should allow me to separate the two VMs logically so if needed I can easily switch to an HTPC later.

Did you run into any problems you can remember running them all off one system?

My HTPC case is a Fractal Design Node 605 and the NAS/PMS/Server case is a Fractal Design Refine R5. Both of which have a nice, sleek black design that looks at home in a media center. Although you could move the server itself to a different room.

Thanks for the ideas, the 605 looks pretty sweet, but I'm not sure I could spend that much on a case. These are a few of the ones I'm looking at: Apevia X-QPACK3, Fractal Design Node 804 and Thermaltake Core V21. Here's the horizontal ones: Silverstone ML04, nMedia 600B, Fractal Node 605. If you have other ideas that would be good and can fit a mATX board I'd love to hear them, I'm a bit behind on cases these days. I have my gaming PC in a Silverstone Grandia GD09.

 

[rhansen5_99]

Very cool, I really like the Fractal 804, but it is pretty huge in person. The Silverstone Granada G09 is a good one too (and you can fit an eatx/ssi-ceb board in there if you wanted to do a dual socket board).

If you are planning on Hyper-V 2016 you might look at remote FX, I have had some success with it on 2012 and cad applications. I would think that the RX460 should be able to pass through to the VM this way and give you a decent experience.

As for a board this Asrock ATX board looks really awesome , dual nic, dual m.2, Xeon support. I have not seen a matx board with dual m.2 slots.

I am not sure if 2011-3 with x99 works the same way as x79 2011-1, but I was able to run ecc registered ddr3 on an EVGA Dark x79 and a MSI matx x79 board, so you might be able to do that with this too.

Just my 2c

 

[BfA]

Very cool, I really like the Fractal 804, but it is pretty huge in person. The Silverstone Granada G09 is a good one too (and you can fit an eatx/ssi-ceb board in there if you wanted to do a dual socket board).

If you are planning on Hyper-V 2016 you might look at remote FX, I have had some success with it on 2012 and cad applications. I would think that the RX460 should be able to pass through to the VM this way and give you a decent experience.

As for a board this Asrock ATX board looks really awesome , dual nic, dual m.2, Xeon support. I have not seen a matx board with dual m.2 slots.

I am not sure if 2011-3 with x99 works the same way as x79 2011-1, but I was able to run ecc registered ddr3 on an EVGA Dark x79 and a MSI matx x79 board, so you might be able to do that with this too.

Just my 2c

Thanks for the insight. I already have my gaming rig in a Granada so I'm looking for a different type of rig for this one. I was thinking of dual socket but that's double the power and chances are I'd rather get a newer, lower power processor when I want to ugprade.

I'll check out Remote FX, although I believe that let's you run other VMs using the virtualized video card. I'm wondering if doing passthrough direct to one VM would be a lot faster. Althought Remote FX might work a lot easier.

That board does look really cool and I would consider it, but as it's going to be a headless server I was looking for something with IPMI. Plus, I don't have much experience with it so I wanted to learn more.

I'll definitely be getting 32 GB of ECC. The mobo I linked supports 2400 but I can't seem to find any ECC at that speed unfortunately. I can just leave it at 2100 though and if I need to upgrade in a few years. I'm thinking of going quad channel even though it doesn't give that much of a performance boost.