• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Troubleshooting ZFS AiO Box Severe Performance Issues

C

coolrunnings

Limp Gawd
Joined
Nov 21, 2004
Messages
511
I have a Napp-IT All in One with the following specs:
SuperMicro X9SCM-IIF-O
E3-1240V2
32gb Samsung DDR3 1600 ECC RAM
Supermicro 846-series 24 bay hotswap chassis
Seasonic X750 PSU
3x IT-flashed M1015/PERC H310 cards
1x Intel dual-port Gigabit adapter + 1x onboard gigabit adapter hooked up to switch
18x 2tb Samsung 204UI Drives
6x Western Digital Black Edition 1tb drives

Running OpenIndiana 151A8 - configured with 10Gb VMXNET3 Adapter
Latest version of Napp-IT free
ESXi 5.5 Free License

Pool Config:
6x WD 1tb = 3x mirrored VDEV's. Used for VM storage
18x 2tb Samsung 204UI drives = 3x 6-drive RAID-Z2. Used for mass-storage, no VM use.

7x VM's - all very low resource usage. 16gb of RAM allocated to OpenIndiana VM and all three HBA's passed through to it.

Here is the problem I'm having. I am streaming backups using Drive Snapshot, copying files over SMB from Windows 7 and XP boxes, and running Windows Easy Transfers directly to the mass storage. If I get more than one datastream going, the machine severely slows down but it does so gradually. It's like someone is slowly turning off the data faucet till I get delay-write-fail errors on all the machines copying to the server. If I power-cycle the server, it will work for a little while then I run into the same issues.

Here is what I have done:
1. Run a scrub on both pools. Turns up no errors.
2. Checked for SMART errors - didn't see any.
3. Checked for any errors in the events section in ESXi - nothing.

I am at a loss on how to troubleshoot what might be the culprit. I wouldn't think 2x streams of Windows Easy Transfer which max out around 3% of the gigabit throughput would tax the system enough to bring it to its knees! I don't recall having any issues like this prior to switching to ESXi 5.5 but I wasn't doing an awful lot of testing either. The machine has been running flawlessly for nearly 1.25 years. I did recently (last 3 months) create the large data pool, however. Before that was only 4x drives in a RAID Z1. I destroyed that pool though and created the 18-drive pool. Any suggestions for troubleshooting this would be greatly appreciated! Also let me know if there's any additional information about my configuration that you may need.
 
Try e1000 vnic instead. I have seen very odd performance anomalies with vmxnet3 and solaris type guests...
 
I will try that. Is there any way to set up multiple e1000 vnic's and pool them so that the traffic isn't bottle-necked? I only have 1G connections coming in but usually several of them and the VM's rely on the same link...
 
Okay I've got the E1000 set up and TCP Offload disabled as per Gea's instructions. I can't get it stable for my VM's now. They fail to fully start up and ESXi gives an error about the disk not being ready. Going back to VMXNET3 for now. At least the VM's worked well in that setup. Anyone got additional suggestions?
 
Last edited:
I just passed through a physical NIC and WHOA! What a difference! My data transfers are no longer severely hampered. There are definitely some performance issues with running the virtual nics here. Hopefully the illumos team will come up with some workarounds or VMWARE will get this fixed. Now I probably better look for a quad-port gigabit NIC. Anyone have a good guide on LACP in OpenIndiana?
 
danswartz said:
Dunno what was going on, but e1000 should be money for any VM...
Click to expand...


One would think! However, it was unstable as hell for me. I have jumbo frames turned off everywhere. Wonder if somehow it's turning them on in OpenIndiana. Whatever the case, it was an unstable piece of crap for reliability so for now I'm using the passed-through NIC and having quite good results! Hoping it stays that way!
 
Try this blog post... this helped tremendously with the VMXNET3 drivers on OpenIndiana for me:

http://blog.cyberexplorer.me/2013/03/improving-vm-to-vm-network-throughput.html

Seems the VMware tools installer for Solaris uses the incorrect driver, and you have to modify the installer a bit to get the installer to use the correct one. And I had to also disable LSO on the NIC.

The instructions are mostly very well laid out... but if you have any questions let me know.
 
I had issues with E1000's i ran them for ages on 5.1 with no issues then 5.5 hit and GEA's fix didnt work for me the only thing that got me stable with VMXNet3's I am using OmniOS with VMXNet3's with Napp-it running solid with an LSI-1068E card PCI Passthrough to the VM.

are your backups are they running over CIFS as well or NFS? Also how many vCPUs are allocated to the VM?
 
You must log in or register to reply here.
Back
Top