ESXi All-In-One build implementation and optimization help wanted

[EZStuff]

#### EDIT ###
##ADDITION##
So far this thread contains the following

Post #1 to 3 : Backgrounder, Specs, wanted end state...
Post # 12 + : VT-d vs Local RDM passthrough
and NFS vs SMB (inc ZFS Sync enabled/disabled, random and sync read/write)
Coming soon - VMCI enabled vs VMCI disabled for SMB and NFS (when transfeering to a none VM machine)

### END###

Here's the usual, This is my first post but have been reading the forum for awhile blurb... which I have, I swear !

*****Backgrounder*****
With that out of the way, after many years of wishing I had a home lab to play around with, I've finally pulled the trigger, waiting on a few loose ends, but the bulk of it is here.
Go big... or...

My specs:
Supermicro 7047R-3RF4+ which included
Motherboard X9DR3-LN4F+ (Quad i350 Intel NIC + Dedicated IPMI)
Chassis 745TQ-R920B

2x Xeon E5-2620 (Total "24" Cores with HT)
12x 4GB ECC RDIMM (Total 48 GB)
5x Mushkin Deluxe 240GB SSD
4x 1TB WD Black HDD (FAEX)
2x 1TB WD Black HDD (FALS)

All protected by a Liebert PowerSure PS1500RT3-120 1500VA

Still waiting on my Cisco SG 300-20 Port Gigabit Managed Switch (SRW2016-K9-NA)
and a 3X5.25 to 5X3.5IN SATA Drive Bay Converter which I'm going to hack into the case (ie: remove the fan)

 

[EZStuff]

The final setup should/will allow me to have
- An isolated VLAN for rooms I rent out, which is protected/throttled by pfSense VM
- A community VLAN for my home network, with a DC for auth, a NAS serving Home folders and used to stream to the media center. This VLAN would also go through PfSense before going out to the internet.
- A VLAN for a future IP Cam w/ DVR and/or Asterix VOIP setup. Every room in my house has 2 network jacks, which helps alot!

Current Setup
- ESXi booting from USB stick
- Local SSD datastore
- IPMI port directly connected to the second NIC of my main desktop
- PfSense VM installed but unused, still waiting on my Cisco switch to arrive
- Windows XP VM with vClient installed, I RDP to this auto-starting VM to manage my VMs as my main desktop runs Linux.
- OI+napp-it currently running (VT-d) 4x WD HDD in a ZFS mirror-strip (RAID 10) config, not sure what Ill do with the 4x SDDs and 2x HDDs I have left
- 2x Nested ESXi - Currently unused, but will be used to test different virt tech that require more then 1 ESXi host.
- Test VMs, including NAS4Free, Illumian, XBMC, Ubuntu, Win SBS, Win2008R2, etc.

I'm looking for advice on different ways of setting things up, to achieve a fast "production" setup and a flexible testing/lab environment.

 

[EZStuff]

Any tips, ideas, improvements I can bring to this setup, bring it on... as for me... here is the first specific backgrounder + question I'm currently facing.

- In my current setup I was unable to VT-d the 8 Hotswap bays, which are in fact running off a Patsburg Dual 4-Port SATA/SAS controller, after researching the issue, it seems the link/switch between these 2 4-port controllers is not VT-d compatible. So I VT-d the other onboard controller (a Patsburg 6 port SATA controller), passed it to the OI+napp-it VM and setup ZFS.

This causes 2 issues
- My 2x SATA III ports are now VT-d to OI, instead of serving my local ESXi Datastore.
- Loss of the ability to enable VMCI, which can be a big deal if I consider the fact that I could be serving my Nested ESXi host and other VMs files "over the network" much faster with this enabled.

With these two issues, would you suggest I disable this VT-d and instead do RDMs to the OI VM ? Is it feasible to even do this ? Drawbacks ? Would their be a noticable performance increase/hit ?

 

[_Gea]

With these two issues, would you suggest I disable this VT-d and instead do RDMs to the OI VM ? Is it feasible to even do this ? Drawbacks ? Would their be a noticable performance increase/hit ?

If you like to have a comparable performance and stability like a NAS build on hardware
you should not use RDM. Your other problem may be the unsupported onboard SAS
(I have never tried)

Your options:
- passthrough SATA to OI
- use Onboard SAS for your ESXi local datastore needed for OI

If ESXi does not support your SAS (I suppose OI does not), your main option
is to buy a cheap IBM 1015 controller.

Either use the 1015 for ESXi (Raid -1) and passthrough the SATA ports

or use Sata for ESXi and passthrough the 1015 (best flashed to IT mode in this case) to OI
I would prefer this due to better performance, hotplug, 6G, disks > 2TB etc



In any case, you need one ESXi supported controller for your local datastorw with OI (you can also put ESXi on it)
and one OI supported controller for pass-through via vt-d and your ZFS pools

 

[EZStuff]

In any case, you need one ESXi supported controller for your local datastorw with OI (you can also put ESXi on it)
and one OI supported controller for pass-through via vt-d and your ZFS pools

This is currently how I have it setup, however as the controller (SAS) I initially wanted to passthrough is not support as a passthrough device by ESXi, I had to reverse my setup and VT-d the other controller (SATA).

For now the setup works, but may upgrade in the future to a SATA/SAS controller card that supports SATA3/SAS2 and VT-d.

So the loss of VMCI support, due to a VT-d'd device, is not a major issue when considering "network" bandwidth between VMs for my NAS ? I guess there is no otherway as the NAS requires direct access to the HW.

 

[dandragonrage]

I doubt VMCI will be necessary for a home server. I'd take VT-d over VMCI for sure. I also agree that if you do buy a SAS card, the M1015 is a good choice. I bought one myself and use it in ESXi with VT-d and Illumian.

 

[EZStuff]

I doubt VMCI will be necessary for a home server. I'd take VT-d over VMCI for sure. I also agree that if you do buy a SAS card, the M1015 is a good choice. I bought one myself and use it in ESXi with VT-d and Illumian.

Gea - Your napp-it site recommends OI, but you use Illumian... is there a reason for that ?

I'm currently doing some testing, VT-d (without VMCI) vs RDM (with and without VMCI) ... local benchmarks and network SMB/NFS benchmarks on OI+napp-it ZFS mirroed-strip... I'll post my results in a day or two...

I have the hardware so figured why not test it before I getting everything settled and dont feel like switching/testing anymore...

 

[dandragonrage]

Illumian uses Debian's package system, which I will take over any package system for Unix (not sure if people still love FreeBSD's Ports, but I think it, like all Unix package managers, is crap).

Illumian thus trended towards usability. OpenIndiana did not. OpenIndiana can thus shove off as far as I'm concerned (nothing personal to those who disagree). If you don't care (much) about the package system, though, OI is fine. Really it's not a big deal but when the option for it is there, I'll take it, thus Illumian.

 

[_Gea]

Gea - Your napp-it site recommends OI, but you use Illumian... is there a reason for that ?

.

I support Illumian as an option like Solaris 11 but currently all of my machines are OpenIndiana.
While Illumian is declared as a 1.0 release it is a beta under construction and nobody
knows if Nexenta is willing to offer it for more than a community beta to improve the
quality of NexentaStor4. Currently it lacks a lot of the OI packages and options.

Thats similar like SmartOS that is not the suggested solution for anything but a KVM base.

Up to now, OpenIndiana and Solaris 11 are the only solutions usable for any server (or even dektop) use.
I would add OmniOS (think of it like a minimalistic OpenIndiana) as a possible third server option in future.

 

[dandragonrage]

Up to now, OpenIndiana and Solaris 11 are the only solutions usable for any server (or even dektop) use.
I would add OmniOS (think of it like a minimalistic OpenIndiana) as a possible third server option in future.

While I have never recommended Illumian to anyone for enterprise purposes, you are certainly wrong about it being usable. It's been 100% stable for me, and has everything I could need of it for what I do with it. I haven't really gone looking for much as far as packages, but I don't see the need to, either.

I would not use OpenIndiana for enterprise purposes, either. I see it as a development distro only.

 

[_Gea]

Illumian uses Debian's package system, which I will take over any package system for Unix (not sure if people still love FreeBSD's Ports, but I think it, like all Unix package managers, is crap).

Illumian thus trended towards usability. OpenIndiana did not. OpenIndiana can thus shove off as far as I'm concerned (nothing personal to those who disagree). If you don't care (much) about the package system, though, OI is fine. Really it's not a big deal but when the option for it is there, I'll take it, thus Illumian.

From my view it does not matter if you have to use pkg install something instead of apt-get install something- The main problem is, that IPS is the original Solaris packaging system. If you move to apt, you must provide anything and you cannot use whats available - a problem of NexentaCore and a problem of current Illumian.

The answer what is better depends on whom you ask.

 

[EZStuff]

So I've completed and compared the benchmarks of Local RDM vs VT-d
The results to me seem clear... performance is equal... as close as it gets without nit picking.

I was also testing SMB vs NFS with ZFS sync enabled and sync disabled at the same time, and figured I'd compare these on RDM and VT-d for the hell of it.
SMB/NFS speed are for random read/write of many 1.5 MB or small files. Large file transfer results will be posted later (1GB+)
Results are rounded as this was just to evaluate if there would be a real world difference


Setup
OI+napp-it VM, mirrored-strip ZFS Pool (4x HDD RAID10)

VT-d passthrough
Sync Enabled SMB write/read 34/30 (Very stable speed, little fluctuation)
Sync Disabled SMB write/read 28/25 (Very stable speed, little fluctuation)
Sync Enabled NFS write/read 6 to 12/38 (Read speed increased over time)
Sync Disabled NFS write/read 28/25 (Very stable speed, little fluctuation)
OI dd read bench ran twice 202MB/s and 189MB/s
bonnie++ 191MB/s Seq-Write / 288MB/s Seq-read

Local RDM passthrough
Sync Enabled SMB write/read 28/25 (Very stable speed, little fluctuation)
Sync Disabled SMB write/read 28/24 (Very stable speed, little fluctuation)
Sync Enabled NFS write/read 8 to 16/38 (Read speed increased over time)
Sync Disabled NFS write/read 48/38 (Very stable speed, little fluctuation)
OI dd read bench ran twice 197MB/s and 189MB/s
bonnie++ 183MB/s Seq-write / 278MB/s Seq-read

NFS performed better or equal on the RDM device and SMB performed better or equal on the VT-d device overall... strange... so most likely due to margin of error. I keep my statement Local RDM passthrough and VT-d passthrough are equal.

P.S. The RDM method I use is a RDM hack which allows you to RDM a local drive, no LUN required, I also used the vmkfstool -z switch vs the -r switch... which by it's description fit better into what I was looking to accomplish (bypasses ESXi iSCSI commands, creating a RDM passthrough device)

 

[EZStuff]

So now that we know performance is near equal... what are the advantages/disadvantages of using VT-d and those of Local RDM passthrough...

VT-d
- The whole controlelr must be passthrough (this can be taken as an advantage or disadvantage), in my case a disadvantage due to the issue with my onboard controller as prev mentioned.
- Unable to perform hot snaps (disadvantage)
- Memory allocation is locked/reserve in full by the VM (disadvantage)
- VMCI disabled/greyed out

Local RDM passthrough
- Pass exactly the device you want and don't want, in my case on either onboard controller.
- Able to hot snap
- Memory assigned to VM not locked/reserved
- VMCI can now be enabled !

Conclusion, for me anyways, is now I'm going to move my harddrives back into their hotswap bays, back onto the other onboard controller, move my local SSD datastore back to the SATA III port and Local RDM passthrough the hard drives I wish to use on my NAS.

Thoughts ? Have other advantages/disadvantages ?!

 

[EZStuff]

Later tonight I will post more SMB/NFS speeds as the current posted speeds are for multiple small file transfer (1.5MB or smaller). I will test when transfering large files (1GB or larger). I expect close to double the speeds.

I also previously mentioned I would test RDM with VMCI enabled and disable. For me most of my network traffic from my NAS will be to a none-VM machines, so I will be testing to see if VMCI actually increases NFS/SMB speed to an external none-VM machine first, I'm guessing little to no increase as this is not the purpose of VMCI, once my final NAS setup is completed in 2-3 days I will retest intenal VM network speeds.

I will also add the VT-d bonnie++ bench which was previously missing.

Once these benches are completed, I will be dismounting my local DataStore, moving it back to my main onboard SATA III port, rebuilding my OI+napp-it, rebuilding my ZFS pool using the same HDDs as RDM devices on the other controller and doing a final set of benches that will compare the Supermicro X9DR3-LN4F+ Onboard Patsburg 6 port SATA controller (directly connected) and the Onboard Dual 4-Port SATA/SAS Patsburg controller (connected via backplane). This might end up happening tomorrow.

If these benches are as expected (same speeds) I will be keeping this setup for my Local datastore and NAS ZFS pool.

 

[EZStuff]

So here are the step to recreate my "Local RDM Passthrough" device as previously mentioned in this thread.

On the ESXi host console
- What disks does ESXi see
# ls -l /dev/disk | more

- Find the disk you want to passthrough. I did this by the serial number and the numbering it uses (port number on controller)

- I went into my Local data store folder
# cd /vmfs/volumes/"Localstorename"

- Created a RDM folder
# mkdir RDMs
# cd RDMs

- Then ran vkmfstools to try and create the RDM map file
The same disks showed up a few times with somewhat similar names when doing ls =l /dev/disks, so I tried running vkmfstools using one of them and got an Access Denied error, so I tried again using its different name, until on the 3rd try it work.
# vmkfstools -z /vmfs/devices/disks/"name of disk" RDM1.vmdk -a lsilogic

Dont forget the -z instead of -r switch !!! This allowed me to go from RDM to VT-d by reimporting the ZFS pool, no data loss! No that is RAW data !

example disk names
naa.50014dd25c8e5304
vml.01000000002020202020202020202020203951454358423630535433453030
t10.ATA____WDC_WD10EAXS2D00Z5B1__________________________WD2DCAWU044559

Once these vmdk files are created, go back to your VM, add a Disk and choose "Already Exists" Point it to the file in your RDM folder.
Voila!

 

[EZStuff]

Posst #12 had small file transfer benches (1.5MB or smaller) The following are NFS and SMB large file speeds. All speeds are MB/s

VT-d passthrough
Sync Enabled SMB write/read 35/32 (Very stable speed, little fluctuation)
Sync Disabled SMB write/read 34/31 (Very stable speed, little fluctuation)
Sync Enabled NFS write/read 70 to 60 (Read speed decreased over time)
Sync Disabled NFS write/read 110/90 (Read speed decreased over time)

Local RDM passthrough
Sync Enabled SMB write/read 36/30 (Very stable speed, little fluctuation)
Sync Disabled SMB write/read 35/31 (Very stable speed, little fluctuation)
Sync Enabled NFS write/read 80 to 60/38 (Read speed decreased over time)
Sync Disabled NFS write/read 99/50 (Very stable speed, little fluctuation)

I also did the VMCI enable test, transfering externally outside the host and as expected no difference. Ill post VMCI speed once I have another VM up and running properly.

P.S. I also added the bonnie++ bench results for VT-d on post # 12 as promised

 

[danswartz]

Why do you think vmci will help network transfers? Using vmxnet3 NIC?

 

[EZStuff]

From my understanding... which may very well be wrong...
VMCI removes the network layer when communicating between VMs and VM to Host, dramatically increasing communications speeds.

My assumption is communication speeds includes network traffic, excluding the use of VMXNET3

VMXNet3 would help speed up communication with machines outside the host. or with other VMs that also had a VMXNET3 adapter...


My VMCI bench to an external machine, was a quick test to see if maybe the better VM-Host communication actually yielded noticeable network speeds. As I previously stated, I doubted it would.

Unrelated Side Note:
By the way, I switched my RDM ZFS RAID10 to the Patsburg 6 Port SATA/SAS Onboard controller thats on my Supermicro X9DR3-3RF4+ and re-ran the Bonnie++ bench here are the results
196 MB/s Seq-Write / 317 MB/s Seq-Read
compared to when it was on the Patsburg Dual 4-Port SATA Onboard controller, as previously posted
183MB/s Seq-write / 278MB/s Seq-read