New Virtual Server Project Build

[NeoMatriX724]

Hiya Folks--

I've got a large amount of experience dealing with regular old PC's and the like but have been looking for a new challenge project...It's been almost 2 years since I've built a new rig and the current hardware out there hasn't really intrigued me enough to scratch the high-performance box itch.

That being said with a lot of the new stuff being done in virtualization, I think this can be a project to properly relieve the upgrade itch I've been feeling.

I have some time and am not in a particular rush. I'm aiming to build this around the June or July time frame with a budget of approximately $5000 (stretchable to $7000 if I absolutely want to do something crazy). I won't have space for a rack but should have space enough for a large tower (or mountain mods box).

So the premise as I have set now is a Hexacore rig with 32GB RAM (though I would entertain a dual socket machine despite my lack of experience).

I haven't figured out the OS yet, but am considering ESXi, but am 100% looking for suggestions.

I would be looking to run a number of VM's including the following:

1 SteamOS VM for both local gaming (with a direct connection to a TV) as well as streamed gaming from my Windows Desktop. Estimated resource allocation 8gb RAM.

1 General Linux VM (Probably CentOS, maybe Ubuntu). This will be used to run my "flavor of the month" Server, such as Minecraft or Terraria or persistent game for myself and friends. Estimated resource allocation of 8gb RAM.

1 PLEX Media Server VM

1 PLEX Client Image (for download via netboot to thin clients like an Aspire REVO)

1 FreeNAS VM -- Connected to built in storage array.

The storage array is planned for 8 2TB Drives in RAID 6. With 4 small SSD's to house the above mentioned OS's. I'm mainly looking to perform these functions because the HTPC box and NAS box in my signature were fried during Hurricane Sandy. My backups are intact, but I haven't really moved them into a "live" machine again.

So I'm looking for the following from you fine folks:

1) Is the above hardware spec seem reasonable for what I want to do?
2) Any suggestions for the OS / software setup?
3) Should i go with a gigabit built in NIC or separate?
4) Any pitfalls or such I should know about going into it?
5) Whatever else you'd like to add

 

[MysticRyuujin]

#1 thing you should know, no Type-1 hypervisor is going be that great for games on a VM (except for the host OS in Hyper-V on Windows 8.1 Pro.)

You cannot use ESXi or Xen locally for almost anything as it's all command line and Remote Management and just generally not supported. Hyper-V is a little different in that it virtualizes the host OS on top of it as well but still allows local interaction with the host OS. However, you can expect some performance hits.

Now if you drop the requirement to game on it locally there are a number of ways to improve VM gaming performance such as RemoteFX and GPU passthrough. However don't expect miracles...it'll be good but not great.

With Windows 8.1 Pro Hyper-V you will get almost all the bells and whistles except for RemoteFX and Live Migration stuff. RemoteFX wouldn't be needed if you gamed locally on Windows 8.1 Pro. However, taxing the Hyper-V Host OS will diminish your VM performance a lot as Hyper-V gives priority to your host OS.

The one thing I will say is that not a lot is CPU bound these days but for VMs you need memory. If it comes down to more memory or more HDD space, and you're serious about playing with VMs, get more memory. I have 2x servers with 16GB RAM each and I keep running out of memory playing with stuff, I'm upgrading them both to 32GB (already ordered).

So in short I'd say:
Windows 8.1 Hyper-V (or a Type-2 such as VMWare Workstation) if you want to game locally.
OR
ESXi or Xen or Server 2012 R2 Hyper-V if you want to game inside a VM but you have to make sure you meet the requirements for RemoteFX or the other GPU passthrough technologies (research it?)

Running the VMs OS on an SSD is highly recommended. I also really like the Seagate SSHDD (Hybrid Drives).

As for built-in or seperate NICs? Doesn't really matter. I'd say find yourself a mobo with dual gigabit NICs or more.

If you're looking for a good single socket motherboard I suggest:
http://ark.intel.com/products/66137/

Quad Gigabit NICs, Full ECC Registered RAM support, 8x SAS ports, 2x SATAIII Ports, 4x SATAII ports.

 

[NeoMatriX724]

#1 thing you should know, no Type-1 hypervisor is going be that great for games on a VM (except for the host OS in Hyper-V on Windows 8.1 Pro.)

You cannot use ESXi or Xen locally for almost anything as it's all command line and Remote Management and just generally not supported. Hyper-V is a little different in that it virtualizes the host OS on top of it as well but still allows local interaction with the host OS. However, you can expect some performance hits.

Now if you drop the requirement to game on it locally there are a number of ways to improve VM gaming performance such as RemoteFX and GPU passthrough. However don't expect miracles...it'll be good but not great.

So in short I'd say:
Windows 8.1 Hyper-V (or a Type-2 such as VMWare Workstation) if you want to game locally.
OR
ESXi or Xen or Server 2012 R2 Hyper-V if you want to game inside a VM but you have to make sure you meet the requirements for RemoteFX or the other GPU passthrough technologies (research it?)

Hrmm, thanks for the info, that may kind of kill the usability of the box as a SteamOS machine. Right now I have my main desktop, but don't really want to have a full fledged PC in the living room. I was hoping to run an HDMI cable and a USB hub into the living room to operate a remote / controller and then just have a TV which would display the SteamOS machine. I think there'd be a pretty substantial performance hit if I can't run locally on the SteamOS instance and connect to it via thin client instead.

The thin client wouldn't necessarily matter for something like PLEX, but I don't think that'd be very reliable for gaming... May need to take that back to the drawing board then =/. I wouldn't necessarily see the point of running Windows 8.1 Hyper-V on this server if my main desktop is already running windows 8.

 

[MysticRyuujin]

I really don't know that much about SteamOS but I know it's Linux and if it's linux you could run VirtualBox on top of it. Probably wont be "supported" but VirtualBox is pretty good and runs under Linux.

 

[MysticRyuujin]

Q: What software runs on SteamOS?
SteamOS is designed to run Steam and Steam games. It also provides a desktop mode which can run regular Linux applications. SteamOS makes use of the standard APT package manager for software updates; you can add third-party sources to your subscribed repositories to gain access to more applications. SteamOS currently provides a limited set of packages, but many Debian wheezy packages work fine on SteamOS. We plan to make a wider variety of packages vailable directly from the SteamOS repositories over time.

That tells me VirtualBox should work...

 

[koolaidkitten]

I think you should go the ESXi with passthrough capability route. It is what I'm currently using on the box in my sig and it works GREAT without any issues. But like MysticRyuujin says, you need to do research before making any purchases. There are little things you have to know as far as compatibility is concerned.

ESXi 5.5 will allow you to get your feet wet with virtualization and gives you a great testbed to learn server administration and networking principles. You don't have to compromise either because you can still do your main computing on it and not have any separate boxes or sacrifice any performance.

I haven't got around to trying SteamOS in VM but I'll give it a whirl when I can.

 

[NeoMatriX724]

I'm not too concerned about the Server Admin portion or the Network administration portion. More with the virtualization and performance optimization in that scenario. It would be a very bad situation to be in to have one of the primary use cases of the box not work so definitely will be throwing in a lot of research beforehand.

I won't be buying hardware for at least 6 months so that should be enough time to get a better grasp of what I'm aiming for.

I think I may go with running SteamOS locally with virtualbox on the side, and then setting up cronjobs and startup jobs to start up the underlying vm's in case the box goes down.

My one main concern is the stability of the SteamOS system since it would suck to have that go down and kill all the VM's (part of the reason I was looking into running each machine as it's own VM on the host.

 

[NeoMatriX724]

So at this point I've been pricing out the storage array:

Starting off with the Controller: LSI MegaRAID SAS 9260-8i RAID .

I'm still mulling around the drive types to get. Currently priced out with 8x 2TB Western Digital Reds, but am considering other alternatives with the batch QA issues I've been hearing about with the Reds and Greens. Suggestions for drives here are definitely open.

The storage profile is mostly media and bulk storage. This stuff won't be high throughput and won't be used regularly so raw speed isn't necessarily a huge concern.

In terms of housing this beast, I was originally going to go with a a Rack Mounted machine but ended up changing my mind for noise concerns and setting up the whole thing in an apartment. At this stage in the game, I'm looking at using a Mountain Mods Case with a number of 5.25 <> 3.5 conversion bays to handle the drives (as well as dual PSU).

I'm waiting a bit before I spec out the Mobo and Proc. I've been mulling the idea of water cooling to keep the noise down, but don't think I want to mess with that for a Server that I would like to keep running 24/7 for as long as possible.


Software wise, I'm more and more leaning towards Xen + Arch Linux as a base and launching instances off of that. Mainly the Xen Passthrough for video seems to be more mature and doesn't seem too terrible to pick up.

 

[MysticRyuujin]

I just built my own storage server at home using a "NORCO RPC-2212"

I must say I'm pretty happy with the noise level. The 4 fans that it comes with are pretty good. I'm sure I could replace them with even quieter fans but I'm happy with its current level. I'm also using a ZALMAN low profile quiet fan for the CPU. Standard 2U PSU from Seasonic.

 

[MysticRyuujin]

For drives consider the SSHDs from Seagate such as ST4000DX001 and similar models. I was worried about how SSHDs would perform under RAID but I found this slightly older article that suggests they do just fine: http://www.storagereview.com/seagate_momentus_xt_750gb_review

I am using Server 2012 R2 storage spaces instead of a true RAID but I'm getting over 600MB/s read out of 8x of the mentioned drives with 2-Way Mirror. Giving me about 15TB of usable space.

 

[NeoMatriX724]

I just built my own storage server at home using a "NORCO RPC-2212"

I must say I'm pretty happy with the noise level. The 4 fans that it comes with are pretty good. I'm sure I could replace them with even quieter fans but I'm happy with its current level. I'm also using a ZALMAN low profile quiet fan for the CPU. Standard 2U PSU from Seasonic.

As much as I want to go with rack mount gear (mostly for being able to quickly expand hard-disk space), the depth of the chassis makes it complicated. Storing this in an apartment setting makes it a difficult sell.

 

[metril]

@NeoMatriX724

Get a Xigmatek Elysium case. I have 12x3TB drives, 2xSSD, and 4x500TB laptop drives in the case. I almost forgot that I have 2x2TB and 1x1TB drives in the case as well.

I'm using an X9SRL-F with an E5-1650v2 and 64GB ECC RAM. Three LSI 9211-8i cards control all the drives, but the SSDs are connected to the onboard for ESXI data stores.

I live in an apartment too. The case fits right under my desk. The case has wheels!