• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

VMs on SAN datastores

D

danswartz

2[H]4U
Joined
Feb 25, 2011
Messages
3,715
Here's a question for y'all: you have a datastore on a SAN - does one generally put the whole VM folder there, or just the VHDs for it? I have a iSCSI SAN with a datastore, and when ESXi booted, the SAN was apparently not yet up. In this case, the SAN is an opensolaris VM on the same box. I had ESXi set up to wait for 120 seconds before firing up the rest of the VMs, and had checked "continue immediately when vmware tools starts". After the host was up, I logged in with vsphere, and saw that not only were the other VMs not up, in the inventory pane, they all showed as 'unknown (inaccessible)', and the iSCSI LUN was not active. I'm not sure why this happened - either openindiana wasn't up before the 120 seconds was up, or something in the concept of having the virtual machine file on a SAN datastore that is not available is the issue. If the former, I can increase the timeout; it really doesn't matter *what* it is, as the others will start as soon as the OI vmware tools are up. If the latter, I suppose I can move the virtual machines to the local datastore, but leave the VHDs on the SAN? Another question is whether this is something that having the SAN on NFS would have prevented? I'm not really invested emotionally in whether the SAN uses iSCSI or NFS. Thanks!
 
Last edited:
From my knowledge, there are a few factors that would determine where the VM should sit

- VM should go on the fastest drives you have
- VM should be on SAN if you want HA

I'm sure others will point out more things, these are the two major ones that come to mind
 
danswartz said:
Here's a question for y'all: you have a datastore on a SAN - does one generally put the whole VM folder there, or just the VHDs for it? I have a iSCSI SAN with a datastore, and when ESXi booted, the SAN was apparently not yet up. In this case, the SAN is an opensolaris VM on the same box. I had ESXi set up to wait for 120 seconds before firing up the rest of the VMs, and had checked "continue immediately when vmware tools starts". After the host was up, I logged in with vsphere, and saw that not only were the other VMs not up, in the inventory pane, they all showed as 'unknown (inaccessible)', and the iSCSI LUN was not active. I'm not sure why this happened - either openindiana wasn't up before the 120 seconds was up, or something in the concept of having the virtual machine file on a SAN datastore that is not available is the issue. If the former, I can increase the timeout; it really doesn't matter *what* it is, as the others will start as soon as the OI vmware tools are up. If the latter, I suppose I can move the virtual machines to the local datastore, but leave the VHDs on the SAN? Another question is whether this is something that having the SAN on NFS would have prevented? I'm not really invested emotionally in whether the SAN uses iSCSI or NFS. Thanks!
Click to expand...

Sort of awkward the way you've done this. If you have an ESXi host, and a VM running on it that is exporting its virtual disk(s) as SAN storage, then a datastore created there and presented to the ESXi server that very VM is running on... lol. I mean what are you trying to do in the matrix, exactly :). Yeah the other VMs that are registered in that datastore aren't going to work right until the storage is working again.... 120 seconds isn't nearly enough time. It probably needs to be 300 seconds or so (5 minutes). I mean does that VM really get started up and everything working in 2 minutes?
 
The idea is the "all in one zfs" that gea over on the data storage systems board likes. the advantage is only one big box - Any files I really care about are regularly backed up off site. Yeah, I figure 300 seconds is probably more inline with reality. I guess what I was hoping for was to find out why the iSCSI LUN didn't come online when the OI SAN VM finally did. Some sort of fatal timeout? I was hoping for a clue as to whether NFS would be vulnerable to the same issue, or whether I should just increase the timeout for the SAN appliance to 300 seconds.
 
Let me take a step back and present this a little differently. Ignoring the way I have this set up, it seems to me the root cause of this is that when ESXi starts up, if it tries to access a SAN datastore, and that times out, it never recovers without manual intervention. Is that a known issue/factor? If so, is NFS subject to the same problem?
 
Because it doesn't automatically rescan for additional datastores within the time period allotted.

It will eventually connect (vmkiscsid retries every 60 seconds), and then the path eval/refresh at 15 min should catch it.
 
Great. thanks! I'm pretty sure I didn't wait 15 minutes when this happened. Does nfs work the same way? And since this is impacting the startup VM sequence, will the path eval/refresh cause the other VMs to start up, or has that code path timed out and failed?
 
1. Uncheck "continue immediately when VMware tools start" on your other VMs. This effectively over-rides the timer.

2. Shutdown your VMware host, start it back up, and then go back to Vsphere and look at the logs for your SAN client. You should be able to see the timestamps for when it started. Then go into openindiana for your SAN and check when it actually finished booting up. That should give you a good idea about what delay you should set for the other VMs.
 
Okay, well, this turned out to be better and worse than I thought. It doesn't seem to matter when the OI SAN VM starts up - the fact that there are VMs using a datastore not available causes them to be inaccessible. That is the bad news. The good news is that when I re-poll the HBA, the OI datastore becomes visible, the affected virtual machines become visible, and they are then started as expected. If I could reduce the time from 15 minutes to say 5, that would help. I'm also going to look and see if NFS has the same issue...
 
danswartz said:
Great. thanks! I'm pretty sure I didn't wait 15 minutes when this happened. Does nfs work the same way? And since this is impacting the startup VM sequence, will the path eval/refresh cause the other VMs to start up, or has that code path timed out and failed?
Click to expand...

Good question. I know the NFS refresh is faster than 60 seconds, the question is how hostd will manage that interaction. I would hazard a guess that it might work better.

The code path has already timed out at that point. That's why production-supported versions of this setup (lefthand vsa/etc) cluster the storage nodes - that way there is one up at boot time already :)

edit: apparently hostd/autostart hasn't given up (from your above). You could script a startup for that VM and a rescan/refresh pretty easily, I'd think... You on classic, or ESXi?
 
danswartz said:
Okay, well, this turned out to be better and worse than I thought. It doesn't seem to matter when the OI SAN VM starts up - the fact that there are VMs using a datastore not available causes them to be inaccessible. That is the bad news. The good news is that when I re-poll the HBA, the OI datastore becomes visible, the affected virtual machines become visible, and they are then started as expected. If I could reduce the time from 15 minutes to say 5, that would help. I'm also going to look and see if NFS has the same issue...
Click to expand...

Lemme check to see if that parameter has any way to be tuned.
 
I'm on ESXi. Thanks much! I'm trying a test VM on a NFS datastore also virtualized to see if it plays any better...
 
Well, spoke too soon. Created the NFS datastore and put a test VM on it. Rebooted. NFS datastore came back much more quickly (did not have to intervene manually). I rescanned HBAs and the SAN datastore came back, but the VMs did not autostart, so your initial concern was well-founded. I'm going to try copying them all to the NFS datastore and see what happens.
 
Wow, that was odd. I tried moving an 8GB or so VM from iSCSI datastore to NFS datastore. So the data will be read from the OI VM to esxi and written back to the OI VM. After a couple of minutes, my ssh session to the OI VM froze. My vsphere connection to esxi disconnected. Several minutes later I was able to reconnect, but the move had aborted and there were a bunch of messages in the log about connectivity being lost and restored (several of these.) I have vmxnet3 adapter on the OI VM, so obviously I can crank a ton of data through; I wonder if there is a bug there?
 
oih, that one I don't know.
try:
for arg in `ls /var/log/messages*.gz`
do
gunzip $arg
done
egrep -i vmkiscsid /var/log/messages*

and post what it returns.
 
I think there may be some wonkiness with the opensolaris vmxnet3 driver. I switched back to e1000 and did a copy of a much larger ubuntu VM from iSCSI => NFS with no glitch at all. I then shutdown and rebooted the hypervisor, and within a minute, the NFS datastore was visible and the 2 VMs I'd migrated to it fired right up. I think this is the way to go :) Thanks for the help!
 
danswartz said:
I think there may be some wonkiness with the opensolaris vmxnet3 driver. I switched back to e1000 and did a copy of a much larger ubuntu VM from iSCSI => NFS with no glitch at all. I then shutdown and rebooted the hypervisor, and within a minute, the NFS datastore was visible and the 2 VMs I'd migrated to it fired right up. I think this is the way to go :) Thanks for the help!
Click to expand...

Could be. I'll see if I can find anything. You had up-to-date tools installed, yes?
 
Oh nuts. My most recent test was a shutdown&reboot after migrating all of the VMs off iSCSI :(
 
No big deal to me, since even with e1000, I can top 3gb/sec between VMs on the same ESXi host.
 
danswartz said:
No big deal to me, since even with e1000, I can top 3gb/sec between VMs on the same ESXi host.
Click to expand...

Same here, it's pretty snappy.

I had a Sol10 guest on vmxnet sending backups to an NFS server elsewhere (M5000) and it would take days to send a 12gb tarball. Load avg also went through the roof, nfsstat reported tons of errors. DBAs complained about VNC sessions being unresponsive all the time.

Switched to E1000 and it was done in minutes. DBA VNC sessions work great now.
 
You must log in or register to reply here.
Back
Top