• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Virtualized home file server

D

Devroush

n00b
Joined
Sep 22, 2011
Messages
10
Hi,

I'm looking to buy a new machine to run ESXi 5.0. This is what I'm currently looking at:

ASUS P8H67-M EVO (should work according to http://www.vm-help.com/forum/viewtopic.php?f=27&t=3377)
Intel Core i5-2500K
Corsair 16 GB DDR3-1333
3x Western Digital WD30EZRX
1x 1 TB disk

The plan is to run a variety of VMs, one of them acting as a file server. The 1 TB will be used to store the VMs, the 3 3 TB disks would be used by a VM using disk passthrough, mainly so that I can use SMART and power management of the disks.

I want to put them in a ZFS pool so OpenIndiana seems like a good choice as the file server VM. Maybe I'll also try out FreeNAS because it has a sweet web interface :). Anyway, any suggestions? Is this a bad idea? Anyone know how ZFS performs in a VM? I thought about buying a storage controller and giving the file server VM direct access using VT-d but I don't really see the point.

Thanks!
 
Solaris 11 Express and its free variants (OpenIndiana, Nextenta, etc) work wonderfully as ZFS hosts on ESXi VM with passthrough disk controllers.

Take a look at napp-it as the front-end for ZFS. Once you use it you'll never look back. Simple, powerful, well supported. Designed from the beginning to run under ESXi.

http://www.napp-it.org/ - include a how-to on building exactly what you describe in the OP.
http://hardforum.com/showthread.php?t=1573272
 
Last edited:
I agree with PigLover ... Gea's Nappit is a great tool. I would suggest if you "think" you might want to do VT-d at some point that you buy a CPU that supports it. If memory serves the K (unlocked) desktop CPUs don't ... most of the non-K (locked) quad core (1155s) do. Also I believe that only the Q67 versions of Intels desktop chipsets support VT-d. I have not run any benchmarks for VT-d based VMDP HBAs vs just passing disks through. My guess would be that there is a performance increase using VMDP for the HBA.
 
One more important note to the OP: ESXi passes through PCI/PCIe controllers. This means you have to pass through the disk controllers, not individual disks. It appears from your parts list that you plan to pass through 3 disks and have the fourth one running ESXi. You can't do this with just the on-board ICH10 controller on the MB - you either have to pass through the entire ICH10 or not.

You need to get a separate disk controller either for the boot/datastore disk or the ones you pass through. Since ESXi is picky about what disk controllers it will support you probably need to get something for the 3 NAS disks.
 
Ah I didn't know that, thank you for pointing that out. Guess I'll have to buy a seperate controller after all.
 
Last edited:
Best to get something like a IBM BR10i or M1015 for your HBA. They are both available as "pulls" on eBay for around $50 (BR10i) or $75 (M1015). Both are widely used in ESXi implementations. I have several BR10is. You would be best off flashing them to IT mode to turn them in to a basic HBA for passing through. This works best for Solaris/OI ZFS.
 
lopoetve said:
You REALLY want the real controller, especially with cache.
Click to expand...

For ZFS? Not really. You want a simple HBA with high performance. Might agree if the OP was talking about doing hardware raid or running something under ESXi, but for ZFS the caches on the controller just gets in the way. The controllers he mentions are actually the best ones for the application.

One caution on the BR10i: its LSI 1068 based which means no support for >2TB drives. It won't support the drives you've selected. Go with the M1015 (rebranded LSI 9240-8i w/out Raid-5) or any other LSI2008 based card.
 
+1 PigLover. It can be an adventure reflashing the m1015 if you are the wrong motherboard, but if it works, the HBA is rock-solid.
 
What about the Perc 6i? Not a lot of M1015s for sale in Europe and they cost +100 euros.
 
Another thought: you could get a controller supported by ESXi for your boot/datastore drive and passthrough the MB based controller.

If you do _Gea's suggestion for an all-in-one then you don't need any kind of performance for the boot/datastore drive You could go REALLY cheap for this one. Really old SiL-3112/3114 PCI based SATA-I controllers are supported (like all software raid cards, only in non-raid mode). Unfortunately, none of the Silicon Image SATA-II or PCIe controllers have standard ESXi drivers, but the 3112/3114 is included in the base software of ESXi 4.x and 5.0. You can get these really cheap ($10-20 or less).
 
Good idea: depends on how many ports his mobo has and whether that is good enough, but that is in fact what I was doing before getting an m1015.
 
PigLover said:
For ZFS? Not really. You want a simple HBA with high performance. Might agree if the OP was talking about doing hardware raid or running something under ESXi, but for ZFS the caches on the controller just gets in the way. The controllers he mentions are actually the best ones for the application.

One caution on the BR10i: its LSI 1068 based which means no support for >2TB drives. It won't support the drives you've selected. Go with the M1015 (rebranded LSI 9240-8i w/out Raid-5) or any other LSI2008 based card.
Click to expand...

I meant way more over the ICH10 stuff. I've just seen really bad luck with those for any kind of real performance.
 
Mobo has 6 SATA ports, which is enough for me.I've found a cheap SiL 3114 on eBay so that's good. Unfortunately I was going to use the single PCI slot for a second network card. Hmm, though decision :).
 
I'm just not a fan of local controllers, but I also don't believe in the all-in-one stuff either (too much of an enterprise level guy). :) Lemme know how it works.
 
Devroush said:
Mobo has 6 SATA ports, which is enough for me.I've found a cheap SiL 3114 on eBay so that's good. Unfortunately I was going to use the single PCI slot for a second network card. Hmm, though decision :).
Click to expand...

PCIe x1 slot for the LAN card and your golden...
 
One point of note, not sure about with ESXi 5, but with 4.1 I ran into this problem .... the rtl network drivers are crap for linux, and if your doing to much traffic it'll kernel panic ... its happened to me with asus P8P67 + slackware, gigabyte ga-h67m + esxi 4.1, and gigabye ga-P55 w esxi 4... in all 3 cases I disabled the onboard and shoved a cheap intel gigabit card in and the problems went away.
 
All you need for ZFS is a decent HBA, it was designed to replace hardware controllers. Supermicro USAS is LSI 1068 and works perfectly, the others will be just fine too. Mine is passed through to an Open Indiana VM and runs a 2 drive mirror for my datastores and a 6 drive raid-z2 for storage.
 
Last edited:
Time to bump this topic.

I bought a m1015 after all and messed around yesterday with trying to get everything to run.

Disks and the m1015 controller where recognized without problems... until I tried to boot ESXi from a usb stick. For some reason, it would hang somewhere during post (didn't even see the m1015 screen) when a usb stick was connected to the pc and the 3 TB drives were connected to the m1015. Shit.

I updated the firmware (which was quite a pain in the ass but that's just because I don't know anything about storage controllers :)). At last I could update the firmware and ESXi now booted with the disks conencted. Hooray!

Next problem: when trying to add the 3 TB disks to a vm guest, raw device mappings was greyed out. Apparently, you can't use that option for local drives? I knew my MB didn't support vt-d but I didn't know I couldn't use RDM then. Well. Shit.

Fortunately, I found a guide on how you can manually make a RDM. http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/ This worked like a charm.

Next problem: the Nexenta guest didn't receive an ip (yes I was using the e1000 adapter). Apparently this is some kind of bug? I didn't find a lot of information about this but I found a solution here: http://www.nexenta.org/boards/1/topics/1118
Like the last guy says: you have to enable the nwam service by typing "svcadm enable nwam". After rebooting this problem was fixed as well.

I then installed Nappit without problems (great software!). I was very curious to see what performance I would get out of it:

write:
root@nexenta:/data/familie# dd if=/dev/zero of=testfile bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 96.1384 seconds, 218 MB/s

read:
root@nexenta:/data/familie# dd if=testfile of=/dev/null bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 86.3706 seconds, 243 MB/s

My jaw hit the floor. I think I won't have any problems with maxing my gbit connection :).

Sorry for the ramblings but maybe somebody will find this information useful!
 
Last edited:
DD isn't a good test. Use iometer to a raw device.

In actual shared usage, you'll never be limited by interconnect.
 
You must log in or register to reply here.
Back
Top