Upgrading Dual CPU ESXi for ZFS

[MySongRanHills]

My current system is as follows:

Supermicro X8DTi
Corsair CX500 500W PSU
(2) Intel Xeon W5590 @3.3GHZ
24GB G.SKILL DDR3 1333 (6 x 4GB)
(1) 2TB WD HDD connected to onboard controller (Datastore to hold VMs)
(5) 3TB Seagate ST3000DM001s connected to LSI 9220-8i cross flashed to a LSI9211-8i in IT mode
(1)Intel EXPI9402PT Dual Port Gigabit PCI-E (both ports assigned to WAN/LAN on PfSense VM)

Running ESXi 5.5 with the following VMs:
Windows Server 2012 R2 - Runs 24/7 with Plex, FlexRAID, Sickbeard (accesses the Seagate drives by physical RDM)
PfSense 2.1.5 - Runs 24/7 as my router/firewall and hosts a OpenVPN server for tunneling into my home network
Windows Server 2012 R2 - test DC runs occasionally
Windows 10 TP - runs occasionally
Ubuntu 14 - runs occasionally when I need to use Linux for something

For various reasons FlexRAID just isn't cutting it anymore as a solution for managing my data. So here is my new plan, I'd like to move to a Supermicro 24 bay HDD chassis(SC846E1-R900B) either keep my existing motherboard or switch to a X8DAH+-F-O (more on this later). Change out my non ECC RAM for 48 GB of ECC RAM - 6x8GB HMT31GR7AFR4C-H9 (listed on mobo specs as compatiable). Then change CPUs to (2) X5687s. The main reason I'm thinking of switching CPUs out is because my W5590s don't support AES-NI and I'd like to encrypt my pool.

Under my new setup I'd like to run the following VMs
FreeNAS--------------- 2 vCPUs -16GB
Linux w/ Plex/SB--- 2 vCPUs -4GB
PfSense---------------- 2 vCPUs -1GB
Linux XBMC distro--1 vCPUs -4GB
Win8.1-------------------4 vCPUs -16GB

For the Linux XBMC distro and the Windows 8.1 VM I'd like to passthrough AMD 6450 GPUs and run HDMI to a TV for a XBMC HTPC and run HDMI to monitors and use the win8.1 VM as my desktop
However my current board only has 1 16x pci-e slot so if I want to Virtualize my desktop I'd need to change motherboards to the aforementioned X8DAH+-F-O(has more x16 slots). I haven't decided quite yet if I'd like to take that dive or not.

If anyone can offer any suggestions or point out things I'm overlooking I'd really appreciate it. Also got some noob questions if anyone would care to take a stab

1.)For hard drives I want to get new ones and am taking all recommendations. I've seen most people prefer 5400RPM drives , do they actually make a noticeable difference in heat? Performance wise I don't really care about individual drive performance as long as I can read/write to the pool @ Gbps speeds (usually get 112MB/s with cifs)which shouldn't be an issue I'd think.

2.)I was thinking of doing RAIDz2 with 6 drives, but after reading this (site down now google web cache link)it sounds like URE rates are greatly exaggerated and that RAIDz is likely still safe. How does ZFS handle handle running into a URE during resilvering?

3.)In an ideal world I'd get a new dual socket 2011 v3 mobo and (2) E5-2623 v3s but that would be at minimum $1,000 more , does anyone know how older faster (56XX) CPUs stack against new but lower freq. CPUs (26xx V2 & v3)? I'm pretty sure it isn't worth the cost, but maybe.

4.)Does any one know what kind of ballpark performance one can expect on non AES-NI CPUs with encrypted pools? Does this performance scale better with higher frequency or higher core count?

5.)I know for most ESXi builds everyone uses passthrough to pass their HBA to the FreeNAS VM, but is there any documented reason not to use physical RDM as I did previously with FlexRAID? This article documents the advantages of using phyiscal RDM over vt-d passthrough.

6.)Is my understanding correct , that the SC846E1 back plane just accepts 1 of the connections from my m1015 for all 24 drive bays??

7.) If I use a SSD for storing the VMs .vmdks and not the pool itself do I need a separate ZIl/SLOG or anything else for the type of performance I'm looking for (around 115 MB/s read/write to pool)

Any help or guidance is greatly appreciated!

 

[ser_rhaegar]

1.) There is a noticeable difference in power draw, so I would assume temps as well. 5400RPM will do you just fine, depending on your raid type. I use 12 drives in mirror mode in FreeNAS and get awesome results. All 5400RPM, mix of brands and models.

2.) URE rates are mathematical facts. If you're going over something like 12TB, raid5/raidz1 will almost certainly experience a failure during a rebuild. If you use raidz1 and have a URE during a rebuild, you lose your pool. You're down a drive so you have no parity to recover with. That is why it is highly recommended to use raidz2/raid6.

If you use enterprise drives, that have a lower chance of a URE to the point where raidz1 and raid5 are OK. But those drives are far more expensive and no consumer drive that I've see has a URE rate of this level.

3.) I used X5500, L5600 and E5-2600v1 and v2 chips in my home lab. I've only experienced a difference in high computation situations like handbrake VMs. The newer chips out performed for me, but you would need to check your specific models in benchmarks and see which has the best computation results. My chips were closer in Ghz.

Also if you're not doing high computation all the time, it isn't likely to be beneficial for you. All my other VMs perform identically from before I upgraded. And for me, I don't mind if handbrake runs for 1 day or 1-2 days converting a batch of files. The only reason I upgraded was to cut power costs and I got a steal of a deal on some 2011 socket motherboards.

4.) My pool is encrypted and shows 99.69% idle avg except when replicating. Then SSH makes it jump. Before I switch to AES-NI, it was 50-60% idle avg, meaning a huge difference in power draw.

I don't virtualize FreeNAS anymore though, it is running on a dedicated box. This is because a change in a motherboard setting for the onboard SATA ports cause the FreeNAS VM which did not use the onboard SATA to corrupt any files it wrote. It was instead using an M1015 in iT mode passed through by ESXi. Luckily I had backups. After that I switched it because it showed how unreliable virtualizing it was.

When I virtualized it, I can tell you if there is core contention, it will run like shit. I would start out with 1 vCPU per VM and only bump up VMs that are CPU starved. This is recommended in general by VMWare as well.

5.) I never used RDM. Can't say but I will say virtualizing is a big risk. Host changes unrelated to the VM can corrupt your pool. I've experienced this personally.

6.) Can't say on that model. For the 3U chassis models, some backplanes require 1 connector per drive (i.e. SAS breakout cable) and some require 1 SAS cable for the entire backplane.

7.) Never tried, can't answer. I don't use a SLOG/ZIL for my pool. I don't experience speed issues on my VMs and they are on spinning disks. Though I use mirror instead of raidz1/2/3 for higher IOPS.

 

[MySongRanHills]

2.) URE rates are mathematical facts. If you're going over something like 12TB, raid5/raidz1 will almost certainly experience a failure during a rebuild. If you use raidz1 and have a URE during a rebuild, you lose your pool. You're down a drive so you have no parity to recover with. That is why it is highly recommended to use raidz2/raid6.

I think some thing was wrong with my previous link, here is the correct one:http://www.high-rely.com/hr_66/blog/why-raid-5-stops-working-in-2009-not/

Basically that bigger than 12TB RAID 5 stuff is crap b/c UREs are way lower then manf. reporting.That said I will use RAIDz2 anyway cuz I'm just not that brave

3.) I used X5500, L5600 and E5-2600v1 and v2 chips in my home lab. I've only experienced a difference in high computation situations like handbrake VMs. The newer chips out performed for me, but you would need to check your specific models in benchmarks and see which has the best computation results. My chips were closer in Ghz.

Also if you're not doing high computation all the time, it isn't likely to be beneficial for you. All my other VMs perform identically from before I upgraded. And for me, I don't mind if handbrake runs for 1 day or 1-2 days converting a batch of files. The only reason I upgraded was to cut power costs and I got a steal of a deal on some 2011 socket motherboards.

Yeah I searched everywhere trying to find a benchmark comparison between the 2 to no avail, but from as best as I can tell from processors from the same family ,the E5-2623 v3 is about 20-30% more powerful than the X5687.

I will be doing some encoding as well, and I'm curious how many vCPUs do you assign your handbrake VM? I read a white paper somewhere about vCPU assignment with HT CPUs. The just of it was if you have 8 physical cores /16 threads you shouldn't assign any single vm more than 8 vCPUs b/c it cant use the HT cores if all real cores are pegged. Would having an 8 vCPU encoding VM make everything unstable I'm wondering. I could leave just pfsense and freenas running (2-4 vcpus combined) during encodes. I'll have to test this at some point.

I don't virtualize FreeNAS anymore though, it is running on a dedicated box. This is because a change in a motherboard setting for the onboard SATA ports cause the FreeNAS VM which did not use the onboard SATA to corrupt any files it wrote. It was instead using an M1015 in iT mode passed through by ESXi. Luckily I had backups. After that I switched it because it showed how unreliable virtualizing it was.

THIS ^ gives me serious pause. After a ton of research all the virtualization problems I have come across involved people using RDM. A m1015 in passthrough is exactly what I was going to use...What happened in your case? It would be no big deal if the freenas vm itself got corrupted by a change in the host BIOS ..I'd just re-install and import the pool. I'm bewildered how the pool itself could become corrupt.

Oh and when you were virtualizing FreeNAS what did you use as a datastore to for the FreeNAS VM?

 

[Netwerkz101]

<snipped for brevity>
THIS ^ gives me serious pause. After a ton of research all the virtualization problems I have come across involved people using RDM. A m1015 in passthrough is exactly what I was going to use...What happened in your case? It would be no big deal if the freenas vm itself got corrupted by a change in the host BIOS ..I'd just re-install and import the pool. I'm bewildered how the pool itself could become corrupt.

Oh and when you were virtualizing FreeNAS what did you use as a datastore to for the FreeNAS VM?

You should visit the FreeNAS forums and post up the issues you have with your virtualized FreeNAS build
Just kidding ...don't! Gaskets will blow .....

They recommend (highly) that FreeNAS should _not_ be virtualized.

Search and view some of member "_Gea"'s posts on Nappit if you want to run a virtualized solution.

 

[drescherjm]

2.) URE rates are mathematical facts. If you're going over something like 12TB, raid5/raidz1 will almost certainly experience a failure during a rebuild. If you use raidz1 and have a URE during a rebuild, you lose your pool. You're down a drive so you have no parity to recover with. That is why it is highly recommended to use raidz2/raid6.

I say in practice as a person who has 70TB spinning at work this is a very false statement. URE rates are not uniform working drives have much lower URE rates than published with drives that are on their way out have much higher rates.

 

[ser_rhaegar]

I will be doing some encoding as well, and I'm curious how many vCPUs do you assign your handbrake VM? I read a white paper somewhere about vCPU assignment with HT CPUs. The just of it was if you have 8 physical cores /16 threads you shouldn't assign any single vm more than 8 vCPUs b/c it cant use the HT cores if all real cores are pegged. Would having an 8 vCPU encoding VM make everything unstable I'm wondering. I could leave just pfsense and freenas running (2-4 vcpus combined) during encodes. I'll have to test this at some point.

I have an 8 core host and a 6 core host, both use HT but I don't count HT for anything, only physical resources. I have two handbrake VMs, each has 4 vCPUs. While I could assign the one on the 8 core system more and then give it a host affinity so DRS doesn't move it, I try to keep things uniform where possible. With 4 vCPUs on the 6 core machine, other VMs can run without causing CPU contention. I think the most cores another VM might have is 2, 90% of my VMs run with 1 vCPU. Processing is about 15fps for 1080 video with this setup.

THIS ^ gives me serious pause. After a ton of research all the virtualization problems I have come across involved people using RDM. A m1015 in passthrough is exactly what I was going to use...What happened in your case? It would be no big deal if the freenas vm itself got corrupted by a change in the host BIOS ..I'd just re-install and import the pool. I'm bewildered how the pool itself could become corrupt.

Oh and when you were virtualizing FreeNAS what did you use as a datastore to for the FreeNAS VM?

When I changed the motherboard raid controller to SATA mode the FreeNAS VM would write corrupted data to the ZFS pool. The M1015 which was in passthrough was getting corrupted data due to the change. As unrelated as it is (motherboard BIOS setting for motherboard SATA controller vs PCIe SAS/SATA controller), this is what happened for me. This was in an HP ML310e Gen8 v2 server.

I think it is unlikely to happen again with a different motherboard (I've moved to all Supermicro) but it isn't worth the risk to me anymore. I didn't notice the issue for a few days and by the time I did, major corruption had occurred and it took me a long time to narrow down the culprit because a lot of things changed in those few days. I had backups that were solid so no real harm but it was a lot of effort to clean up everything and restore the backups.

When I virtualized it, the FreeNAS VM was on an SSD in the host ESXi system. Originally both of my ESXi hosts had 240GB SSDs. Now neither has any local storage. I use FreeNAS NFS shares for all my VMs today.

 

[MySongRanHills]

You should visit the FreeNAS forums and post up the issues you have with your virtualized FreeNAS build
Just kidding ...don't! Gaskets will blow .....

They recommend (highly) that FreeNAS should _not_ be virtualized.

Search and view some of member "_Gea"'s posts on Nappit if you want to run a virtualized solution.

Napp-It = No Encryption

Read all the FreeNAS Forum stickies, even the one on how to run it as a VM even though it isn't recommended/supported.

When I virtualized it, the FreeNAS VM was on an SSD in the host ESXi system.

On a separate m1015 or pci-e HBA? Cause if the FreeNAS SSD was on the on board ports it would be more understandable(the SATA mode change causing corruption)

I was hoping to run FreeNAS from a usb (it loads into memory and doesn't run from the drive I believe) , but that would require on board usb passthrough (something that hasn't worked for me before on supermicro motherboard) So I may have to spare a small SSD I guess.