Solaris Express 11 - Comstar iSCSI Performance Issue

[DarkDiamond]

This is my first (of hopefully many) posts here, so please be kind

I have Solaris Express 11 installed on a box providing iSCSI storage to two ESXi 5 hosts and I'm seeing what appears to me a significant performance issue. I have Napp-It installed on the SE11 box (great software for Solaris noobs like me - thanks Gea!) and got benchmark results > 200mbs for sequential reads and writes.

What I'm about to say isn't an exact measurement, but when I'm copying files from my desktop (Core i5 with two 500GB drives in RAID 0) to my virtualized file server on ESX1 or ESX2, I'm only seeing transfers around 30 MBs. If I copy a file from one drive to another within the VM itself (which eliminates my PC's connection as a bottleneck), I see less than half of that (around 10 MBs). I've tried swapping cables (CAT6), changing switches (I have both 1400-8G and 1800-8G HP switches), direct connection between machines but unfortunately still haven't seen decent performance. After configuring jumbo frames between the SE11 box and ESXi, I saw a slight improvement but nothing earth shattering. Jumbo frames are configured end to end (including switch). Side note: When copying the files, I'm monitoring CPU usage on the VMs and Host and I'm not seeing either of them even close to being pegged.

Shouldn't I see something a lot better given the performance of the storage pool local to the machine?

Any ideas/tips that I can use to boost performance? Any ideas if I've missed something on the network or iSCSI configuration either on Solaris or ESXi? I realize both of those questions are extremely broad, but hopefully you all can bear with my noobish questions

Here's the specs for my setup...

Storage Server

​

Motherboard: Supermicro X8SIF-O

​

CPU: Xeon X3440

​

RAM: 16GB ECC Registered

​

ISCSI NIC: Intel 82574L Chipset

​

OS: Solaris Express 11

​

Storage Pool: 8-1TB WD RE3 in a Raid 10 Config


ESX1 and ESX2

​

Motherboard: Biostar TH76B

​

CPU: Intel i3-2100T

​

Ram: 8GB DDR3

​

ISCSI NIC: Intel 82574L Chipset


Thanks
DarkDiamond

 

[danswartz]

Could be sync mode. Try setting sync=disabled on the pool and re-test.

 

[ukaudiophile]

Hi,

Am I correct in thinking that you've carried out an ftp transfer from a desktop PC to the iSCSI server and see >200Mb/s, but then copying to a file server which is actually a hosted VM using SOL11E and COMSTAR iSCSI as the SAN you're only seeing around 30 Mb/s?

Have you tried FTP'ing to the file server VM? That would eliminate the SMB / SMB2 protocol used by Windows as the bottleneck and would help us figure out exactly what is going on.

Was the 10 Mb/s when copying between datastores hosted on the same SAN using the VMWare client datastore browser? If it was then this doesn't surprise me. The datastore browser and file handling is awful, incredibly slow and gives a fraction of the performance of FTP.

Regards,

Dave

 

[DarkDiamond]

Could be sync mode. Try setting sync=disabled on the pool and re-test.

I thought sync mode was an nfs related setting? I'll try that tonite.

 

[DarkDiamond]

Hi,

Am I correct in thinking that you've carried out an ftp transfer from a desktop PC to the iSCSI server and see >200Mb/s, but then copying to a file server which is actually a hosted VM using SOL11E and COMSTAR iSCSI as the SAN you're only seeing around 30 Mb/s?

Have you tried FTP'ing to the file server VM? That would eliminate the SMB / SMB2 protocol used by Windows as the bottleneck and would help us figure out exactly what is going on.

Was the 10 Mb/s when copying between datastores hosted on the same SAN using the VMWare client datastore browser? If it was then this doesn't surprise me. The datastore browser and file handling is awful, incredibly slow and gives a fraction of the performance of FTP.

Regards,

Dave

The 200 Mbps benchmark was done within napp-it. I believe it uses Bonnie+ to do the benchmarks. The 30 Mbps performance was either copying to or from my pc to a share on the vm. The time that I saw around 10 mbps was from one virtual disk to another on the same datastore. In this last case both the reading and writing traverse the same iscsi link. Maybe a networking issue?

I will try ftping something to the iscsi box tonite and separately to the file server vm.

 

[danswartz]

I thought sync mode was an nfs related setting? I'll try that tonite.

it is an attribute of any dataset written to. nfs from esxi is a common culprit, since they insist on using sync mode.

 

[DarkDiamond]

I tried disabling sync mode and didn't see any performance improvement. I haven't tried FTP'ing a file yet.

I have proved that the ESXi Initiator is the cause of the performance problems. I configured the Microsoft iSCSI Initiator inside the VM to point to a datastore I created on the Solaris box, dedicated a NIC and vSwitch to it and copied many large files to it. I was able to almost saturate the gigabit connection. When I would try again using a virtual disk mounted from an ESXi Initiator-based datastore, performance was < 30Mbps. I've tried it on both ESXi 5 and 4.

I'm going to (hopefully) try to prove that Comstar isn't conflicting with ESX somehow by trying a different target. When I installed Starwind on my PC and configured my ESXI 4 host initiator to point to it, it would crash ESXi when scanning for new datastores. I'm going to try ESXi 5 tonite.

Anyone have any other ideas?

 

[danswartz]

Ah, okay, sorry no ideas here. I see you posted this to the virt forum - good idea there. Some sharp vmware savvy folks (including an employee) who might have an idea.