Howto: Freenas 8/9S VM on ESX5i with SAS2008 HBA and multiple CPUs and Multiple Nics

O

ozra

n00b
Joined
Mar 11, 2012
Messages
18
My First post, constant lurker.......
Hopefully this helps someone else and my reasons for wanting to do the above are clarified below.

I have had multiple problems with the SAS2008 HBA and Freenas 8.0x (FreeBSD 8.2) running under ESX5i as a VM. I have tried Freenas 9S (FreeBSD 9 based) and encountered the exact same problems. It seems the same driver is used in 8 and 9.

Being a FreeBSD noob did not help!

All credit for solving it to the Web and Google! EVENTUALLY!!!!


The cards (LSI SAS2008: IBM M1015 / Dell H200 + SAS Expander) work fine under Freenas 8 on a standalone PC.
I could not get them to work under Freenas 8.0.x under ESX5 (Passed Through), ALTHOUGH, ESX5i could work with the card without any problems whatsoever.
I did not want to add a VMFS layer in-between ESX5 and Freenas ZFS.
OpenIndiana can also access the card fine under ESX5 (passed though) but I have already gone down the Freenas road with about 30TB of storage (Norco 24 Drive with Chenbro CK23601 6GB/s 36 Port SAS Expander)

So I figured that it’s the Freebsd SAS2008 Driver.

Please note that the Chenbro CK23601 card played no role in my problems experienced. My problems started before I even attached it to the SAS HBA.

Running an AMD FX 8150 with 16GB of RAM and a 4 port Intel Network card (LAGG on a managed switch) just for Freenas seemed like a bit of a waste.

My ESX server with all my other Virtual Machines was underutilized.
As I was running ESX anyway for SABNzb/Couchpotato/Sickbeard/Win Domain/ Asterix/ Exchange and various other SIP servers with multiple snapshot capability and configs I wanted to utilize it better.

The solution below works fro Freenas 9S as well, juts put your changes into loader.conf

Installing Freenas 8.0.x under ESX5i with the SAS2008 card passed through (VMDirectpath) produced the following errors during Freenas boot:

run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config mps_startup
run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config mps_startup

and it never gets past that stage.

To solve that problem:
Shut down the VM
Edit the Freenas VM Settings.
Detach the SAS Card.
Boot Freenas.

Edit the loader.conf
Add the following to it:

hw.pci.enable_msi="0" # Driver Interrupts problem SAS2008
hw.pci.enable_msix="0" # Driver Interrupts problem SAS2008

Shut down the VM
Edit the Freenas VM Settings.
Add the PassThrough SAS Card. (Your reserved memory should be the same value as the memory allocated to the VM (VMDirectpath requirement) otherwise it will not boot.
Boot Freenas.

Problem Solved!

If you use Multiple NICS for lagg with MTU 9000 you may have to add the following (I have 4 NICS) to loader.conf (or loaders in 8.0.3 p1)
kern.ipc.nmbclusters="262144" # Network Buffers Problem with MTU9000 and Multiple NICS

Otherwise not all my NICS are able to be utilized due to buffer constraints. (And a lot of my Jumbo frames get thrown away)


Under ESX5i, if you add more than 1 CPU to the Freenas VM, I encounter “Interrupt Storm on IRQx” (IRQ18 in my case).
To solve that, boot into Freenas VM Bios: Disable Floppy drive, Com ports, Printer ports and anything else that you are not going to use.
Save changes.
Reboot
Solved!




STATS:

ESX5i

Freenas 8.0.3 p1

Virtual Machine: Freebsd 64 Bit: 16GB RAM: 2 vCPU of 1 socket each: E1000 VMWare adapter

Passthru PCIe Card: Dell H200 Flashed to LSI 9211_8i (P12) Firmware (Chipset: SAS2008)


For Testing I used some old drives:

4 x 750GB WD Green Drives in Striped UFS config:

[root@freenas8-test] /mnt# dd if=/dev/zero of=/mnt/raid0/testfile bs=8192k count=1000
1000+0 records in
1000+0 records out
8388608000 bytes transferred in 25.843491 secs (324592679 bytes/sec)
[root@freenas8-test] /mnt# dd if=/dev/zero of=/mnt/raid0/testfile bs=8192k count=10000
10000+0 records in
10000+0 records out
83886080000 bytes transferred in 271.635798 secs (308818207 bytes/sec)


3 x 750GB WD Green Drives in Raidz1 ZFS config:

[root@freenas8-test] ~# dd if=/dev/zero of=/mnt/test1-raidz1/testfile bs=8192k count=1000
1000+0 records in
1000+0 records out
8388608000 bytes transferred in 48.232895 secs (173918817 bytes/sec)
[root@freenas8-test] ~# dd if=/dev/zero of=/mnt/test1-raidz1/testfile bs=8192k count=10000
10000+0 records in
10000+0 records out
83886080000 bytes transferred in 536.209856 secs (156442630 bytes/sec)



4 x 750GB WD Green Drives in Raidz1 ZFS config:

[root@freenas8-test] /mnt/test1-raidz1# dd if=/dev/zero of=/mnt/test1-raidz1/testfile bs=8192k count=1000
1000+0 records in
1000+0 records out
8388608000 bytes transferred in 29.539966 secs (283974871 bytes/sec)
[root@freenas8-test] /mnt/test1-raidz1# dd if=/dev/zero of=/mnt/test1-raidz1/testfile bs=8192k count=10000
10000+0 records in
10000+0 records out
83886080000 bytes transferred in 378.921389 secs (221381222 bytes/sec)

Samba Transfer rate: 60MB/s Write and 75MB/s Read with AIO on: Read size: 8192 / Write size: 8192
 
how does one edit the loader.conf?

I tried looking online and haven't had any success. Total newbie here.
 
From Google: freenas edit loader.conf

To answer your question, I was able to go to the shell and do the following

mount -rw /
vi /boot/loader.conf

hit 'i' to enter interactive or insert mode

make any desired changes, hit 'esc' to get out of interactive mode and then :wq to write and quit vi

then reboot

Your nick seems appropriate.:D
 
I did what you suggested but still couldn't get Freenas working with 2 cores. Alot of Interrupt Storm on IRQx
 
mwroobel said:
The reason you are seeing interrupt storms is because you disabled the message signaled interrupts which then fall back to less advantageous interrupt methods.
Click to expand...

You mean these two lines:

hw.pci.enable_msi="0" # Driver Interrupts problem SAS2008
hw.pci.enable_msix="0" # Driver Interrupts problem SAS2008

If I don't, I couldn't get pass boot with:

run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config mps_startup
run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config mps_startup
 
arryo said:
You mean these two lines:

hw.pci.enable_msi="0" # Driver Interrupts problem SAS2008
hw.pci.enable_msix="0" # Driver Interrupts problem SAS2008

If I don't, I couldn't get pass boot with:

run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config mps_startup
run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config mps_startup
Click to expand...


There are a lot of issues with FreeNAS with pass-through devices in ESXi that don't occur with the same hardware on bare metal. I have seen it happen with HBAs and with COMM cards. One thing you can try and do is disable EVERY device you aren't using. LPT ports, COM ports, USB legacy, onboard devices, floppy ports, IDE ports, Option ROMS. Try and free up more interrupts. I gave up on freenas a long time ago with hypervisors.
 
I'm running it with 1 core, so it's good. problem happens only when i try adding 2 or more cores
 
arryo said:
I'm running it with 1 core, so it's good. problem happens only when i try adding 2 or more cores
Click to expand...

what kidn of performance do you get? Which CPU do you have? I have a hex-core 1045t I was going to dedicate 2 or 4 cores to it, but sounds that might be an issue.
 
It needs alot of ram, so if you put plenty of memory in, i think it shouldn't bother you too much
 
ozra said:
From Google: freenas edit loader.conf

To answer your question, I was able to go to the shell and do the following

mount -rw /
vi /boot/loader.conf

hit 'i' to enter interactive or insert mode

make any desired changes, hit 'esc' to get out of interactive mode and then :wq to write and quit vi

then reboot

Your nick seems appropriate.:D
Click to expand...

BTW, thank you for your help, I spent a good 45min-1hr trying to figure out how to edit loader, I've never used FreeBSD or FreeNAS, I'll give this a shot. Thanks. Oh I've been having trouble with signing into FreeNAS to manage it, several things to try to figure out.
 
ozra said:
From Google: freenas edit loader.conf

To answer your question, I was able to go to the shell and do the following

mount -rw /
vi /boot/loader.conf

hit 'i' to enter interactive or insert mode

make any desired changes, hit 'esc' to get out of interactive mode and then :wq to write and quit vi

then reboot

Your nick seems appropriate.:D
Click to expand...

Thank you for your help! I have the virtualized FreeNAS up and running. As I was in the FreeNAS virtual machine's BIOS to disable unused devices, I noticed that there was an option for ECC. I have ECC ram running in my server, should or I enable ecc on the virtual machine for it to work at the FreeNAS level? what do you guys think?

Options in Bios are:

-Disabled
-EC (Error Checking)
-ECC (Error Checking & Correction)
-ECC Scrub. (Error Checking & Correction with scrubbing

BTW I disabled Floppy, Serial ports, parallel, and am running 3 virtual sockets x 1 core per socket just fine. I have a 6-core 1045t. 2 sockets x 2 core resulted in IRQ storm.
 
Last edited:
I am having so much trouble with FreeNAS on ESXI. I have it running, everything is set up. I have 7 x 2TB drives I am running on raidz2 and when I try to do large transfers, it always fails. FreeNAS freezes entirely in ESXI, and I have no clue why. Any help or guidance appreciated. I tried reducing number of cpu cores down to 2 without any success.

When I am in the hypervisor console, the virtual machine is completely frozen. I have to restart the virtual machine to get it up and running, can't get into the webgui.

My transfer rate when it works is around 68-72mb/s
 
UPDATE: it looks like it was the CPU issue. I lowered it down to one core and it seems to be working fine now. Transferred 400GBs and still going...speeds only at 58-60MB/s though.

Even though you only assign 1 CPU, there is no limit to use one cpu, resources consumed show 3.5ghz, even though mine's only 2.8ghz, so i guess it does use other cores even though one is assigned(reserved?).

I tried OI, setting up ACL was a bit challenging and I gave up on it. I will perhaps try it again in the future, what are you running now?
 
I'm running OI+Napp-it. The acls and such don't bother me, cuz I don't use them. I'm using a simple unix user == windows user model, so the share is chown'ed to each user and windows just connects transparently with windows user's credentials.
 
So I followed the directions in building my ESX5i all-in-one FreeNAS 8 (using a Supermicro X9SCM and an IBM M1015 flashed to IT) and got the same interrupt storm errors when I tried to have more than 1 cpu / core.

If I want to be able to use more than 1 CPU, do I need to look at OI / Napp-IT instead?
 
Unless you can prove you have enough CPU need on the SAN VM to mandate more than 1 processor, I wouldn't bother.
 
If I'm starting with 6x2TB drives in Raid Z2 with room to grow to 24 drives (norco 4224), wouldn't the extra cores help? I'm using an Xeon E3-1230 v2.
 
Help with what? A non-existent CPU load? Like I said, unless you know you have a CPU load issue, an extra core is likely not worth it. How busy does your box get? My moderate/light use SAN serving up NFS for 6 VMs with the 'w' command:

2:01pm up 5 day(s), 19:46, 1 user, load average: 0.07, 0.06, 0.05

Virtually no CPU load. Granted this is on a dual-core, but this is bare metal. Hard to imagine one core would be any worse...
 
I've read in a couple forum posts via google that ZFS does multi-thread well and takes advantage of multiple cores which is why I thought it would be helpful. I went ahead and installed OI with Napp-IT and I have to say that so far I prefer FreeNas' interface much more. But I'll have to spend a little more time getting to know OI/Napp-IT before making any final decisions.

danswartz - I see from your earlier posts that you stopped using FreeNAS and went with OI/Napp-IT. Is that still the route you would suggest?
 
Yes. I had instability issues with freenas under heavy load. Maybe it was the mptsas driver - dunno.
 
You must log in or register to reply here.
Back
Top