OI ZFS move to ESXi slow transfers

P

PointandClick

Limp Gawd
Joined
Dec 6, 2008
Messages
383
I upgraded the OpenIndiana server in my sig to a Supermicro X10SLL-F-O with a Xeon e3 1220 and 20gb ram. Dell SAS 6/ir as hba. In the process I'm trying to move it over to ESXi, but so far it's been a royal pain in the ass. I got it all set up, but my file transfer speeds are practically unusable.

DD on the local raidz array I get 300+MB/s reads and writes.
Over AFP to a OS X 10.9 client I get ~ 300KB/s reads,
Over SMB to the same client I can manage between 30-50MB/s reads and writes,
SMB to Win8 laptop over 10/100 gets ~900KB/s reads.

My old setup with Realtek nics on both ends managed 100MB/s+ both directions.
The new setup bare metal seems to perform fine.

The most obvious thought is that I have the ESXi networking screwed up since I've never really used it before, but I didn't really change anything besides adding the second nic in to see if that helped anything. VM is using e1000 nic.

ESXi 5.5 with i210 and i217-lm drivers added in.
OpenIndiana VM has 2 cores and 14gb ram, SAS6 passed through.
Only other VM is Ubuntu Server with 1 core and 512mb ram, sitting doing nothing.
 
I seem to recall seeing complaints about e1000 on 5.5. Try changing nics to vmxnet3?
 
Just did that. Seems to be back where it should. Now I just can't get my interface to stay persistent. I set it up as dhcp according to every guide I've seen:

Code: 
# dladm show-phys
# ipadm create-if eth0
# ipadm create-addr -T dhcp eth0/v4

But then every boot I have to issue a:
Code: 
ipadm enable-if eth0
to start it back up.

At this point I won't at all be surprised if it's something stupidly simple I've missed. This project should have taken an hour and it seems like I've been working on it for a week.
 
Last edited:
weird. are you wedded to OI? if this is just for a NAS, you might want to try omnios...
 
None of that makes any sense.

eth0? you think your using linux?

e1000g0/v4? I thought you said you switched it to vmxnet3s?

I'm not sure about dhcp setup, as I would never use dhcp for a static server.
I was having horrible issues with the vmware vmxnet3 driver, and had to switch to the open-vm-tool one, to get jumboframes to work correctly.
 
danswartz said:
weird. are you wedded to OI? if this is just for a NAS, you might want to try omnios...
Click to expand...

No, I'm not. I had actually looked at moving to OmniOS and had this exact issue, which was one of the reasons I stuck with OI for the time being until I had more time to play with it.

None of that makes any sense.
eth0? you think your using linux?
Click to expand...
No, but eth0 is a lot simpler to type repeatedly than vmxnet3s. I renamed the physical interface.

e1000g0/v4? I thought you said you switched it to vmxnet3s?
Click to expand...
I did. I forgot to edit that in my copy paste... one of those days.

I'm not sure about dhcp setup, as I would never use dhcp for a static server.
I was having horrible issues with the vmware vmxnet3 driver, and had to switch to the open-vm-tool one, to get jumboframes to work correctly.
Click to expand...
I leave everything set to dhcp and give them static mappings in pfsense. Easier to do it all in one place if something needs to be changed.
 
Hmm, I've never seen this, but I always do full static IP - it is fairly easy to do, and seems to persist...
 
That may be what I have to do.

On a side note, does virtualized networking really have that much overhead? I'm seeing over 30% host cpu usage on sustained transfers. Or do the Solaris drivers just suck.
 
Last edited:
I would switch back to e1000 with static mode under omnios. Works fine for me. On their wiki, they show a 3-step process (a handful of trivial commands) to set up static IP and it 'just works'.
 
did you create the files in /etc for the new NIC e.g. /etc/hostname.eth0 ? The commands you ran aren't persistent. You need to create the config files.
 
I've definitely had to create files much more recently than 2010, though I don't use OpenSolaris(-based-things) anymore as of about a year ago. It could be a different file that I created at that time, I guess. I had the same issues with the commands not persisting. I was not doing it via ifconfig either. In any case, I was able to get it working and it took more than just ipadm, route commands.
 
Last edited:
If you look at the omnios wiki, you do 4 things (usually):

1. ipadm to create the static IP.
2. create a persistent default route (route -p).
3. put 'nameserver xxx' in /etc/resolv.conf.
4. cp /etc/nsswitch.dns /etc/nsswitch.conf
 
You must log in or register to reply here.
Back
Top