• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Home virtualization server

E

eradicator006

n00b
Joined
Mar 7, 2013
Messages
20
Hi,

I'm looking for some advice on setting up a virtualization server for my home network. I basically need advice regarding software and hardware.

My home server which was a dual 3.4ghz p4 xeon with 8GM ecc ram decided to die on me. Power supply blew after upgrading ram. It's an old dell so I'll likely just garbage it. I was running Ubuntu 12.04 64bit server on it. The server acted as my firewall, dns, dhcp, netatalk for timemachine on my mac, samba, plex media server and general samba file server. Everything was running off of 1 1TB sata drive.

Here is my plan for the new server. I want to run a hypervisor and run pfsense for firewall, some sort of nas (freenas, openindiana, omnios, etc..) which will do zfs on 4 x 2TB hdds. Then I'll have another VM running ubuntu for plex. So, 3 vms; pfsense, freenas or something similar, and ubuntu. My plan is to use pci passthrough and pass the onboard sata to the nas vm. I'll get a cheap 2 port sata card for the hypervisor.

For new hardware I'm leaning more towards a Gigabyte GA-970A-DS3, AMD phenom ii x4 965 black, 16GB RAM. I am not sure how well that will work with esxi. I may also consider an i5 3330 with a gigabyte GA-B75M-D3H .

Now, for the software question. I am debating between using esxi or citrix xenserver. Esxi seems to be picky on hardware. I want to go cheap on the hardware so I don't know if esxi will even work. xenserver seems more forgiving on the hardware however it appears to not work with freebsd guests (which is what pfsense and freenas are).

So what should I do? I'm torn betwen amd, intel, esxi, xenserver, heck throw in KVM in there too.
 
I would honestly tell you to just use a low power desktop like an IB i3 or so with about 16GB running VMware Workstation, ignore the passthrough for the controller and call it a day. Save some coin and get plenty of horsepower to suite what you are wanting to do.
 
peanuthead said:
I would honestly tell you to just use a low power desktop like an IB i3 or so with about 16GB running VMware Workstation, ignore the passthrough for the controller and call it a day. Save some coin and get plenty of horsepower to suite what you are wanting to do.
Click to expand...

Hmm, you gave me another idea. I do have a first gen i5 system that I put together in 2009. It's an Asus p7p55d with i5-750. I work from home and that's my workstation. It runs mac os x mountain lion and I use vmware fusion with windows 7 pro for some of my apps. When my server died I put my i5 system in its place and I work off of my 2006 macbok pro core2duo.

What I could do is throw together an i3 system as my workstation and put esxi on my i5 with 16GB ram. How well will the i3 work for running windows 7 pro in vmware? On my i5 I had 2 cores allocated with 4GB RAM. Will I notice much difference if I only allocate 1 core from the i3?

I would still prefer to use a hypervisor of some sort. More so for the learning experience. It's something new to me.
 
That's what I setup; Same direction as you want to go.

Running ESXi 5.1 with pfSense, FreeBSD/ZFS and Ubuntu/Plex and other Ubuntus's.
The hardware is based on a MB from Supermicro supporting ECC ram and a XEON CPU. Reason is that ZFS will require ECC as it cache lots in memory; therefore higher quality is key.

The onboard SATA I keep for ESXi and VMs and bought an additional host adapter (LSI 9207-8i) for FreeBSD/XFS to connect my storage. That one is fully on passthrough. Works well.

One issue you will have is with network cards; ESXi is picky on hardware, as you said. So solid Intel-chip-based NICs are required to have less problems..

You will struggle with graphic cards passthrough; I got I once half running for Ubuntu but never for Windows. There are few success stories for graphic cards but I can't yet reproduce. So keep a windows box separate seems a good idea. I run Windows 8 virtual on my Mac via VMware Fusion.
 
That sounds like a better way to do things. I can pick up a supermicro 1155 matx board and a xeon e3 for close to the cost of an i5 system. ecc ram isn't even that expensive either. 16GB for $153 canadian.
I'm hoping the dual onboard nics will work in esxi. I would love to pick up an HBA of some sort however they are quite expensive. Even used M1015's are pricey.

I did boot esxi on my p7p55d i5-750 and the onboard realtek nic worked fine. I was quite surprised at that.
 
One intel NIC chip on my X9SCM-F is not supported (yet) by ESXi.
There are ways to patch ESXi; but I added one Intel card to get around the lazy way.
 
I don't disagree with any of the above. Since you mention you have an i5 why not just use that. I run anywhere from 4-8 VMs (including 2 ESXi hosts) on my laptop with an i3 and 16GB RAM and a single SSD and it runs fine. The cpu is not your limiting factor, it's RAM and maybe IOPs (drive I/O) if you hit the set VERY hard all at once.
 
I was kind of hoping I could get away with using my i5 for esxi (at least for now). It seams everything works except directed io. I have vt-d enabled in the bios.
 
Likely a pain requiring a special trip to some drop-off but please recycle that old Dell beast. And I don't think ~$80 is pricey for a used M1015 which is actually what I managed to score a never-used one with a full-height bracket for about six weeks ago. A couple Buy It Nows up there for $95 right now...
 
wrigleyvillain said:
Likely a pain requiring a special trip to some drop-off but please recycle that old Dell beast. And I don't think ~$80 is pricey for a used M1015 which is actually what I managed to score a never-used one with a full-height bracket for about six weeks ago. A couple Buy It Nows up there for $95 right now...
Click to expand...

I live 2 blocks from the recycling depot so that won't be a problem.

I just looked on ebay.ca (I am in canada) and the cheapest M1015 that I could find is $144 plus $20 shipping. I don't know, maybe I am using the wrong term to search. I just put in M1015. The search comes back with only 2 cards. 1 has no pci bracket, Oh, I bet ebay.ca only shows the ones that will ship to canada. It seems people are afraid to ship here for some reason.
 
peanuthead said:
Vt-d doesn't work because the i5-750 doesn't support it per Intel.
http://ark.intel.com/products/42915/Intel-Core-i5-750-Processor-8M-Cache-2_66-GHz
Click to expand...

Darn. I looked at that page last night and I was almost certain it supported vt-d. Oh well. I guess that settles that.

Down to 2 choices now. Intel S1200BTLR server board with xeon e3-1220 and ecc ram or phenom x4 ii 965 black edition with some sort of gigabyte board. The amd is half the price of the xeon set up however I will likely run in to compatibility issues with esxi. Hate to buy it and have problems.
 
eradicator006 said:
Darn. I looked at that page last night and I was almost certain it supported vt-d. Oh well. I guess that settles that.

Down to 2 choices now. Intel S1200BTLR server board with xeon e3-1220 and ecc ram or phenom x4 ii 965 black edition with some sort of gigabyte board. The amd is half the price of the xeon set up however I will likely run in to compatibility issues with esxi. Hate to buy it and have problems.
Click to expand...

What kind of compatibility issues are you thinking you will run into?
 
peanuthead said:
What kind of compatibility issues are you thinking you will run into?
Click to expand...

Issues with onboard nic. The motherboards I've been looking at use realtek nics. I thought the realtek 8168 on my p7p55d was working ok but it wasn't. After a short while it would stop working. Also, my biggest concern is that IOMMU won't work properly. That is the main thing I need. I'll be passing through either onboard sata or else a M1015.
 
Maybe I should just forget about pci passthrough and go with an i3 system. Would it be a smart idea to use RDM for zfs instead of passing through controller itself?
 
While the 990FX supports IOMMU, the Intel stuff has had way more testing put into it (I mean mainly in terms of software, e.g. devs mostly use Intel hardware, but probably also true on the hardware testing side as well). I wouldn't recommend AMD for this type of stuff. And I really do recommend the PCI passthru idea if possible. I actually do that in my setup, M1015 + C2D E8500 + Intel S3200SH + 8GB ECC DDR2 (DDR2 is limiting though, 2GB/stick max, ugh). Also using passthru for my "WAN" NIC passing through to pfSense, to lessen the chances of hypervisor exploits affecting me (not that it's impossible with my setup either but hypervisors aren't firewalls).
 
Last edited:
Guess I'll pass on the AMD system then. Ok, it's final. Intel e3 1220 with Intel Server Board S1200BTL. I can get those for a little bit cheaper than I paid for my i5 cpu and motherboard back in 2009.

I'll have to check ebay regularly for a cheap M1015.
 
eradicator006 said:
Guess I'll pass on the AMD system then. Ok, it's final. Intel e3 1220 with Intel Server Board S1200BTL. I can get those for a little bit cheaper than I paid for my i5 cpu and motherboard back in 2009.

I'll have to check ebay regularly for a cheap M1015.
Click to expand...

You could save $50 or so by getting a Supermicro X9SCL+-F board and putting $30 into an Intel E3 1230 V2 to get hyperthreading.

Here's a link to some price comparisons: (Canadian pricing though)
http://www.pricebat.ca/mbd-x9scl-f-b-supermicro-matx-lga1155-ddr3-ecc/
 
Bisu said:
You could save $50 or so by getting a Supermicro X9SCL+-F board and putting $30 into an Intel E3 1230 V2 to get hyperthreading.

Here's a link to some price comparisons: (Canadian pricing though)
http://www.pricebat.ca/mbd-x9scl-f-b-supermicro-matx-lga1155-ddr3-ecc/
Click to expand...

Well that looks like a good motherboard. From what I've read you need the +- version so that both NICs work. Does that still apply to the newest esxi? I find that you have to watch carefully because some sites list it as x9scl+-f and the part # is actually a x9scl-f. Also, with this board, can you pci passthrough each nic separately or are they both tied together?

Thanks.
 
dandragonrage said:
While the 990FX supports IOMMU, the Intel stuff has had way more testing put into it (I mean mainly in terms of software, e.g. devs mostly use Intel hardware, but probably also true on the hardware testing side as well). I wouldn't recommend AMD for this type of stuff. And I really do recommend the PCI passthru idea if possible. I actually do that in my setup, M1015 + C2D E8500 + Intel S3200SH + 8GB ECC DDR2 (DDR2 is limiting though, 2GB/stick max, ugh). Also using passthru for my "WAN" NIC passing through to pfSense, to lessen the chances of hypervisor exploits affecting me (not that it's impossible with my setup either but hypervisors aren't firewalls).
Click to expand...
The AMD systems seem to do really well with ESXI and AFAIK can use ECC memory.

I don't know where you read this but if you could post where the hardware doesn't work. So far I have read it works very well.

8 cores, pop in a cheap 2 port NIC and a raid card.
 
Hagrid said:
The AMD systems seem to do really well with ESXI and AFAIK can use ECC memory.

I don't know where you read this but if you could post where the hardware doesn't work. So far I have read it works very well.

8 cores, pop in a cheap 2 port NIC and a raid card.
Click to expand...

I'm stil considering this option too. I am considering the piledriver 6300 and the gigabyte 970a motherboard. I can at least get those locally where as the e3 setup would need to be ordered online

Yes, amd system can use ECC. I've read this in numerous places. Once you add in ecc ram there will be some ecc options that appear in the bios.
 
eradicator006 said:
I'm stil considering this option too. I am considering the piledriver 6300 and the gigabyte 970a motherboard. I can at least get those locally where as the e3 setup would need to be ordered online

Yes, amd system can use ECC. I've read this in numerous places. Once you add in ecc ram there will be some ecc options that appear in the bios.
Click to expand...
I have run the 970 just for trying pass through, and it worked fine.

I am in the middle of upgrading the kids comps again after the move, so I am going to need new esxi hosts....

Hate it when that happens. :D
 
Regarding the hypervisor, have you considered Xen? There are a few things where I believe Xen is ahead of the rest, incl. the commercial ESXi. Here is a shortlist on how I use Xen:

1. Linux Mint (Ubuntu) host OS (dom0) using multiple LVM drives (LVM = logical volume manager)
2. Windows 7 Pro guest OS (domU)
3. VGA passthrough of a dedicated graphics card to Windows 7 guest for native (bare metal) graphics performance
4. PCI passthrough of some USB and SATA controllers to Windows guest.

I wrote some backup scripts for both dom0 (host OS) and domU (Windows guest). Since I use LVM, using the snapshot feature allows me to backup a running system and, when necessary, restore it.

I wrote a how-to here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It should work likewise for Ubuntu.

On the hardware side, you need VT-d support, and for VGA passthrough its best to look at AMD cards.

By the way, since I use a router and since I regularly backup my Windows VM, all I use is Microsoft Security Essentials (MSE) under Windows. Linux and the router are setup to not allow any access from outside the home network. If you need access from the outside, it's pretty easy to setup too, for example using ssh.

As for the server part and disks, I use two 2TB HDDs with striped LVM so it gives me RAID0 performance, and Samba shares under Linux for shared data storage. Sometimes I run additional VMs, usually for testing purposes. Can't complain about performance.

Hope I have been able to answer some questions and perhaps offer a new option.
 
Powerhouse64 said:
Regarding the hypervisor, have you considered Xen? There are a few things where I believe Xen is ahead of the rest, incl. the commercial ESXi. Here is a shortlist on how I use Xen:

1. Linux Mint (Ubuntu) host OS (dom0) using multiple LVM drives (LVM = logical volume manager)
2. Windows 7 Pro guest OS (domU)
3. VGA passthrough of a dedicated graphics card to Windows 7 guest for native (bare metal) graphics performance
4. PCI passthrough of some USB and SATA controllers to Windows guest.

I wrote some backup scripts for both dom0 (host OS) and domU (Windows guest). Since I use LVM, using the snapshot feature allows me to backup a running system and, when necessary, restore it.

I wrote a how-to here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It should work likewise for Ubuntu.

On the hardware side, you need VT-d support, and for VGA passthrough its best to look at AMD cards.

By the way, since I use a router and since I regularly backup my Windows VM, all I use is Microsoft Security Essentials (MSE) under Windows. Linux and the router are setup to not allow any access from outside the home network. If you need access from the outside, it's pretty easy to setup too, for example using ssh.

As for the server part and disks, I use two 2TB HDDs with striped LVM so it gives me RAID0 performance, and Samba shares under Linux for shared data storage. Sometimes I run additional VMs, usually for testing purposes. Can't complain about performance.

Hope I have been able to answer some questions and perhaps offer a new option.
Click to expand...
What are the things that Xen is ahead of ESXI of?
 
Powerhouse64 said:
Regarding the hypervisor, have you considered Xen? There are a few things where I believe Xen is ahead of the rest, incl. the commercial ESXi. Here is a shortlist on how I use Xen:

1. Linux Mint (Ubuntu) host OS (dom0) using multiple LVM drives (LVM = logical volume manager)
2. Windows 7 Pro guest OS (domU)
3. VGA passthrough of a dedicated graphics card to Windows 7 guest for native (bare metal) graphics performance
4. PCI passthrough of some USB and SATA controllers to Windows guest.

I wrote some backup scripts for both dom0 (host OS) and domU (Windows guest). Since I use LVM, using the snapshot feature allows me to backup a running system and, when necessary, restore it.

I wrote a how-to here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It should work likewise for Ubuntu.

On the hardware side, you need VT-d support, and for VGA passthrough its best to look at AMD cards.

By the way, since I use a router and since I regularly backup my Windows VM, all I use is Microsoft Security Essentials (MSE) under Windows. Linux and the router are setup to not allow any access from outside the home network. If you need access from the outside, it's pretty easy to setup too, for example using ssh.

As for the server part and disks, I use two 2TB HDDs with striped LVM so it gives me RAID0 performance, and Samba shares under Linux for shared data storage. Sometimes I run additional VMs, usually for testing purposes. Can't complain about performance.

Hope I have been able to answer some questions and perhaps offer a new option.
Click to expand...

I've looked in to citrix xenserver. I think it would great for me if 2 of my virtual machines weren't freebsd based (freenas & pfsense).
 
Hagrid said:
What are the things that Xen is ahead of ESXI of?
Click to expand...

I'm talking about the free, open source Xen hypervisor, not the commercial Citric Xenserver (I haven't used that).

The main advantages I can see are:

1. VGA passthrough, and probably PCI passthrough - a.f.a.i.k. Xen was the first to offer that feature and still seems to be years ahead. Of course, it's a non-issue if you don't use it.
2. Virtualization options: Xen offers both paravirtualized guests as well as fully virtualized guests. Many Linux distributions are available as paravirtualized guests, having optimized drivers for performance boost. PV drivers are also available for fully virtualized Windows guests.
3. Performance: Paravirtualized guests under Xen typically offer better performance than fully virtualized guests. Unfortunately VMware doesn't allow the publication of performance figures, so everyone would need to test this on his/her own. In theory, paravirtual VMs should beat fully virtualized VMs anytime, as long as the PV driver is halfway decent. In practice I saw a significant improvement in disk I/O performance using PV drivers in a Windows VM on Xen.
4. No limitations to Xen - unlike the free ESXi version, Xen doesn't impose any limitations.
5. Open source: Xen is open source. This has led to the establishment of a developer community in addition to the main contributor Citrix who together improve and enlarge Xen features. As far as security threats are concerned, open source software has had a significantly better security record compared to closed source software. Of course, you never know what you get when buying closed source software.
6. Better compatibility: I gather from forums and the hardware compatibility lists that Xen has better hardware and software support. On the software side, Xen supports (almost?) anything from Linux via Windows (incl. Windows 8) through NetBSD etc., see here: http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines. Better software support also includes GPLPV (paravirtualization) drivers for Windows VMs.
7. It's free! If you need support and perhaps some better management GUI / features, you can choose to buy Citrix Xenserver. But still, the Xen hypervisor is the real thing, not some tuned down freebie version.

For example, you could use Xen to run some cloud computing data center with features such as life migration etc. etc. Another free version of Xen called XCP is actually geared toward that application. There are also a number of open source free as well as commercial management tools that support Xen.
 
eradicator006 said:
I've looked in to citrix xenserver. I think it would great for me if 2 of my virtual machines weren't freebsd based (freenas & pfsense).
Click to expand...

As far as I know, FreeBSD is supported by Xen. See here: http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines. Here is a somewhat old how-to on creating a FreeBSD guest under Xen: http://forums.freebsd.org/showthread.php?t=10268

I just noticed that the Xen compatibility list on this Wiki page is terribly outdated. I'm running Windows 7 Pro as a guest on Xen, and I know enough people running Windows 8. Windows 8 was already supported when the first Windows 8 Beta version came out.

Citrix Xenserver is the commercial product. If you run this for private use at home, I suggest to use the Xen hypervisor. The free Xen hypervisor might even come with extra features compared to the commercial Xenserver, except the Xenserver management. Creating virtual machines under Xen, even without GUI, is quite easy and well documented. If you need a GUI, you can try virt-manager. I still prefer editing the guest config file, since virt-manager has a very strange way of handling guest configurations.

To install Xen, just open Synaptic (in Ubuntu) and install the xen-system-amd64 meta-package. See my first post with the link to the Linux Mint how-to for details.
 
Last edited:
Powerhouse64 said:
As far as I know, FreeBSD is supported by Xen. See here: http://en.wikipedia.org/wiki/Comparison_of_platform_virtual_machines. Here is a somewhat old how-to on creating a FreeBSD guest under Xen: http://forums.freebsd.org/showthread.php?t=10268

I just noticed that the Xen compatibility list on this Wiki page is terribly outdated. I'm running Windows 7 Pro as a guest on Xen, and I know enough people running Windows 8. Windows 8 was already supported when the first Windows 8 Beta version came out.

Citrix Xenserver is the commercial product. If you run this for private use at home, I suggest to use the Xen hypervisor. The free Xen hypervisor might even come with extra features compared to the commercial Xenserver, except the Xenserver management. Creating virtual machines under Xen, even without GUI, is quite easy and well documented. If you need a GUI, you can try virt-manager. I still prefer editing the guest config file, since virt-manager has a very strange way of handling guest configurations.

To install Xen, just open Synaptic (in Ubuntu) and install the xen-system-amd64 meta-package. See my first post with the link to the Linux Mint how-to for details.
Click to expand...

From what I recall FreeBSD 32bit has the PVM options in the kernel where as 64 bit does not. My plan is to run 64 bit freenas so I can use ZFS.

There is a free version of citrix xenserver. I think it's also opensource. Supposedly the free xenserver does more than free esxi.

I am a fan of opensource. I love it. If the freebsd support was in xen (or xenserver) I would go that way for sure. I can run freebsd as HVM how ever I hear there is a big performance loss doing it that way.
 
eradicator006 said:
From what I recall FreeBSD 32bit has the PVM options in the kernel where as 64 bit does not. My plan is to run 64 bit freenas so I can use ZFS.

There is a free version of citrix xenserver. I think it's also opensource. Supposedly the free xenserver does more than free esxi.

I am a fan of opensource. I love it. If the freebsd support was in xen (or xenserver) I would go that way for sure. I can run freebsd as HVM how ever I hear there is a big performance loss doing it that way.
Click to expand...

I haven't used FreeBSD for a long time, and the Xen related info I posted is from Internet searches. Looking at the FreeNAS website they do offer a x86 version, but it is unclear to me whether or not it comes with PVM in the kernel (or perhaps amd64 does too?).

I found this thread that could be informative: http://forums.freenas.org/showthread.php?4482-FreeNAS-in-a-xen-domU. Check posts #4, #7 and #10 in particular.

Both ESXi and Xen allow you to pass through SATA controllers to the guest. This way you could install FreeNAS as HVM guest and pass through the disk controllers to use the native FreeNAS driver. For network performance, it may be advisable to add a NIC and pass that through too.

The big advantage of the Xen hypervisor (not the Xenserver, though it might work too) is when you use it with VGA passthrough and a Windows guest.

About ZFS and FreeNAS, would striped LVM disks with ext4 or other native Linux file system be an alternative? LVM works nicely under Xen, with good performance and easy backup options. Just a thought.

P.S.: You made me curious about FreeNAS. Hope to be able to give it a try, but currently I haven't got any spare disks.
 
OK I got my new motherboard yesterday. It's a Gigabyte 990FXA-UD3. Now I just need to get cpu, ram and a power supply.

Does anybody know if this motherboard can boot without a video card? My server will be headless so I am hoping I can get away with no video card.
 
You must log in or register to reply here.
Back
Top