Install ESXi on USB or local storage

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I have a specific issue to address in regards to this question. We have several ESXi servers running in a HA cluster. The only local storage on the servers is a 60GB SSD drive which ESXi 5.1 is installed on. Last week one of the SSD drives starting failing. All the VMs continued to run without any issues, but we were not able to migrate the VMs to another host. This was a serious issue as we really can't have any downtime since they are customer VMs. Eventually the hard drive was working enough to allow us to migrate the VMs without any downtime. But this brings up a question about the best way to have ESXi running.

My ideal way would be two drives in RAID 1 with a third hot spare. Another engineer recommended a USB thumb drive so we don't have to worry about hard drive failures. But if ESXi is running off a USB drive, and that USB drive fails, wouldn't we be stuck in the same situation?
 
Interesting question, in for the answer.

It seems like one would run into the same issue with a USB drive and that using RAID1 with a hot spare would be the best way to avoid this like you said.
 
What kind of servers do you have? My R720s have redundant SD cards in them. Same concept.
 
You'd want redundant local storage such as RAID 1 hard drives/SSDs or mirrored flash. Boot from SAN is another option but adds complexity.
 
KapsZ28 said:
What about PXE boot?
Click to expand...

If you're on Enterprise Plus licensing you could try Auto Deploy. Then there's no hard drive to fail at all since the entire hypervisor gets put in RAM. However, Auto Deploy can be difficult to set up and even more difficult to manage. Worked with a client where it made perfect sense, they had 100+ hosts, a dedicated management cluster, etc, etc, but after a few months of having to continuously update host profiles and troubleshoot problems they put everything back on local RAID 1 storage.

My recommendation would be to get familiar with host profiles and distributed switches then put everything on local RAID 1 storage. You can quickly provision new hosts and replace failed ones.

Even better is to use UCS blades. If an entire blade fails you can just move its hard drives and identity (UID, WWNs, MAC addresses, etc.) to another blade and boot it up. No OS reinstall or config required! :D
 
We already have host profiles and distributed switches. Quickly deploying another host really isn't a concern since the other hosts in the cluster can handle the load. Only real concern is avoiding what happened last week where a failing host would not let use migrate the VMs. RAID 1 isn't overly expensive and it sounds like PXE is too much of a hassle.
 
How are you going to mirror with drives without a RAID controller? ESXi won't use the Intel RAID (if the system supports it).
 
KapsZ28 said:
I have a specific issue to address in regards to this question. We have several ESXi servers running in a HA cluster. The only local storage on the servers is a 60GB SSD drive which ESXi 5.1 is installed on. Last week one of the SSD drives starting failing. All the VMs continued to run without any issues, but we were not able to migrate the VMs to another host. This was a serious issue as we really can't have any downtime since they are customer VMs. Eventually the hard drive was working enough to allow us to migrate the VMs without any downtime. But this brings up a question about the best way to have ESXi running.

My ideal way would be two drives in RAID 1 with a third hot spare. Another engineer recommended a USB thumb drive so we don't have to worry about hard drive failures. But if ESXi is running off a USB drive, and that USB drive fails, wouldn't we be stuck in the same situation?
Click to expand...

Wait, so why in the world would a local drive failure cause a problem? ESXi - you can yank the drive ~out~ and it won't even notice, unless you have VMFS on there. If you have shared storage, why are you leaving local storage around - just yank the VMFS volume, and that boot device becomes meaningless once you're up and running.
 
lopoetve said:
Wait, so why in the world would a local drive failure cause a problem? ESXi - you can yank the drive ~out~ and it won't even notice, unless you have VMFS on there. If you have shared storage, why are you leaving local storage around - just yank the VMFS volume, and that boot device becomes meaningless once you're up and running.
Click to expand...

That is what VMware support said. I didn't work with them, but a colleague of mine did. They SSH'd into the host and a couple of services were not running and could not be started. Apparently the syslog was also being stored on the local hard drive which they were not able to access. Eventually after all the VMs were moved off of it, he tried to reboot the server and it never came back up. Once the hard drive was replaced and ESXi reinstalled, it is now working again.

Are you saying that once ESXi is running you can disconnect the internal hard drive and it will continue to run? I guess the same goes for a USB drive. You can pull the USB drive our and it will continue to run?
 
QHalo said:
It runs in memory after load.
Click to expand...

If that is the case, then what would have caused the problem with our host? I just checked and see that vShield App is installed on the local storage. Is that normal?
 
KapsZ28 said:
That is what VMware support said. I didn't work with them, but a colleague of mine did. They SSH'd into the host and a couple of services were not running and could not be started. Apparently the syslog was also being stored on the local hard drive which they were not able to access. Eventually after all the VMs were moved off of it, he tried to reboot the server and it never came back up. Once the hard drive was replaced and ESXi reinstalled, it is now working again.

Are you saying that once ESXi is running you can disconnect the internal hard drive and it will continue to run? I guess the same goes for a USB drive. You can pull the USB drive our and it will continue to run?
Click to expand...

Yes, once you are fully booted on the host the local storage is not doing anything unless you have your scratch area set to write to it (where the logging is stored) or you are using the local storage as a VMFS storage point.

I don't know the specifics of the above situation to fully comment but did the host have a PSOD at some point is it's life?
 
Last edited:
KapsZ28 said:
That is what VMware support said. I didn't work with them, but a colleague of mine did. They SSH'd into the host and a couple of services were not running and could not be started. Apparently the syslog was also being stored on the local hard drive which they were not able to access. Eventually after all the VMs were moved off of it, he tried to reboot the server and it never came back up. Once the hard drive was replaced and ESXi reinstalled, it is now working again.

Are you saying that once ESXi is running you can disconnect the internal hard drive and it will continue to run? I guess the same goes for a USB drive. You can pull the USB drive our and it will continue to run?
Click to expand...

Syslog will drool on itself if you lose the store for it, but everything will (should) keep running.

You betcha - we've had USB keys/etc drives die many times, or arrays go down for BFS luns, and it'll keep on trucking.
 
KapsZ28 said:
If that is the case, then what would have caused the problem with our host? I just checked and see that vShield App is installed on the local storage. Is that normal?
Click to expand...

Not that I've seen, but I don't work with vShield all that much. If it was running on local, then you could have absolutely had a problem - that's a VMFS volume going away, but not "dying", so it'll make the host unhappy. there's your answer.
 
For cases like this I wonder if there is a way to clone USB drives so you can drop one in to replace a dead stick
 
You must log in or register to reply here.
Back
Top