UPS for ESXi - Shutdown Script

[g0dM@n]

I did a ton of research and found a way to run a shutdown script that shuts down all of my VMs in the proper order, than shuts down the ESXi host.

The problem is I have no way of automatically launching the script.
I've done a USB passthrough to a Windows VM, so that I could then have the UPS software detect a power outage and launch an executable/script.

Here's the problem:

• So I had an old APC 1500, but the personal software doesn't allow shutdown scripts. I installed the business edition which allows scripts, but that software doesn't detect my UPS
• I stupidly purchased a Cyberpower 1500 because I was told they allow scripts. Their personal and business edition software detect the UPS, but neither offer an option for a path to an executable/script!!

Rookie mistake of taking someone's advice without checking up on it. I don't plan on shelling out hundreds for a business-class, commercial-grade UPS. I don't need it. I just need something that'll help with brown outs, short power outages, and safe shutdowns. My ESXi server is for home use.

I came across this (could be old Cyberpower software), but I can't find a download:
CyberPower PowerPanel Shutdown Service
http://www.cyberpowersystems.com/user-manuals/RMCARD100-Shutdown_Service_UM.pdf

Any ideas? I now have an APC 1500 and CyberPower 1500.

My only other option is to look into event log triggering a scheduled task, but now I feel like a dunce for wasting $$ on this Cyberpower 1500 (I have a ton of UPSs in my house, so I have no use for this extra one).

 

[farscapesg1]

What about running a linux NUT server (Network UPS Tools)? I've been meaning to set up an old Raspberry Pi to do this at home, mainly because I have two hosts and having to connect a UPS to the host defeats the purpose of being able to vmotion between the hosts.

 

[danswartz]

I was going to ask about this myself. You could then run vanilla client software on each VM, and have them query the NUT server (which can even be a VM...)

 

[mrwizardno2]

If you set startup / shutdown order, what do you need a separate script for? I use CyberPower PowerPanel business VM (kind of dumb for what it is, I completely agree). I have it tell my system to shut down, but I've defined the order within ESXi itself that seems to take care of things. Am I missing something?

 

[michalrz]

I guess the 'script' you're looking for would indeed need to pass a shutdown command through a SSH shell to the VM host. Let it handle graceful shutdowns of guests.
Now, a dedicated NUT server to catch the power event and send the SSH command is a good idea as long as the UPS is supported.

 

[g0dM@n]

If you set startup / shutdown order, what do you need a separate script for? I use CyberPower PowerPanel business VM (kind of dumb for what it is, I completely agree). I have it tell my system to shut down, but I've defined the order within ESXi itself that seems to take care of things. Am I missing something?

I guess the 'script' you're looking for would indeed need to pass a shutdown command through a SSH shell to the VM host. Let it handle graceful shutdowns of guests.
Now, a dedicated NUT server to catch the power event and send the SSH command is a good idea as long as the UPS is supported.

Guys, forgive me for not being clear.
The script runs an SSH command directly to the ESXi host to tell it to shutdown. The ESXi host has the correct shutdown order already set.

All I need is for something to kick off the script when the UPS is ready for shutdown.
Having an agent on each VM doesn't help entirely because then the ESXi host won't be shutdown.

I really didn't want to have a dedicated VM just for UPS shutdowns... just a waste of RAM on my host. I have plenty of disk and could care less, but my host is stuck at 32GB RAM and that's the only reason I keep my VM count as low as I can.

*Edit*
I'm most upset about wasting $$ on a new UPS. I can't believe I believed this other person that they had a script path option. Something so basic, right... they must not give out the option because it causes too many support calls to them from people who don't know how to use it correctly. I just thought Cyberpower would have had it since APC business ed does.

 

[michalrz]

Oh, alright then.

Well, in defence of the VM idea - a barebones GUI-less Debian 'netinstall' (minimal package set) would only cost you like 500 megabytes of RAM and almost no CPU time. You'd forward the UPS only to this one and install no other software on other guests. If you set the delays right, shutting down the host from the guest is doable.

Another really ugly but simple method would be to have the scrpt run on the host but detect outages another way, like periodically trying to ping an off-ups network device and rely on that as the sensor

 

[michalrz]

Maybe there's an option to buy a network interface for the new UPS? so it can broadcast autonomously?

 

[g0dM@n]

Oh, alright then.

Well, in defence of the VM idea - a barebones GUI-less Debian 'netinstall' (minimal package set) would only cost you like 500 megabytes of RAM and almost no CPU time. You'd forward the UPS only to this one and install no other software on other guests. If you set the delays right, shutting down the host from the guest is doable.

Another really ugly but simple method would be to have the scrpt run on the host but detect outages another way, like periodically trying to ping an off-ups network device and rely on that as the sensor

Maybe there's an option to buy a network interface for the new UPS? so it can broadcast autonomously?

All great ideas, but I think I'm beating myself up for wasting $$ now. I bought Cyberpower because I thought they had an option to run a script upon battery detection. APC has this in their business edition, so I believed that CP did also.

Having a dedicated VM also means I'd have to waste a PCI slot to do a USB passthrough to it. I was doing the USB passthrough on a windows VM that I'm also using a keyboard/mouse on.

I guess at this point, I have solutions, just none that work to my original plans. You've all been very helpful, though!

I'm hoping I can leave things as is, and see if I can use a task schedule event log trigger for say the UPS notifications on battery.

 

[cyclone3d]

for the best solution to this, I would hook into the Cyberpower monitoring program.. maybe through a .dll if there is one and program a custom monitor.

That way you can tell it to do whatever when the UPS reports it has lost AC power or when the battery only has a certain percentage left, etc.

Edit:
Another idea would be to monitor the log file of the Cyberpower software if it has one... I would think it would.

That would probably be a lot easier than trying to figure out how to grab data from the program itself.

 

[michalrz]

Having a dedicated VM also means I'd have to waste a PCI slot to do a USB passthrough to it.

An USB->RS232 converter might work (probably not)

Another idea would be to monitor the log file of the Cyberpower software if it has one

Good stuff, I'd add the system log as a potential source of events. you can export logs to csv files via MyEventViewer

 

[HammerSandwich]

I run a Cyberpower CP1350PFC with the PPBE agent VM. It seems to work, but I haven't retested it lately...

IMO, the overhead's insignificant in the context of ESXi. Give it 100MHz & 256MB, and it'll be fine. Isn't shutting down safely is far more important than ~1% of 32GB?

I just attach USB devices to VMs, instead of using VT-d on the root hub, so there's no need for a separate USB card. Performance seems fine, but I understand if your needs are different.

Ultimately, it seems that you're complaining about compromises that cost little but saved you $100s on the UPS.

 

[mrwizardno2]

Errrr. The CyberPower PowerPanel VM does this. Pass the UPS through via USB to the VM. The VM needs the password for the host and issues the shutdown command. If you want to see how mine's configured, I'll share some screen shots. You don't have to pass through the whole PCI slot just to get a single USB port through. Each device can be assigned to a VM on its own.

 

[g0dM@n]

HammerSandwich, and mrwizard...
How do I get a single USB port to the VM? I thought I had to do a full pass-through...

 

[g0dM@n]

I still have the UPS attached to a Win7 VM through USB pass-through. I have the Cyberpower agent installed, but couldn't get the Cyberpower CLIENT VM to see it.

I re-installed the agent on the Win7 VM, and then tried to connect the Cyberpower client VM to it again and it's finally successfully connected. I've already put in the root password for the ESXi host for a shutdown command.

I'm going to test this way and if I don't like it, I'll try what mrwizard said. I never passed a host USB through to a VM... stupidly didn't know I could do that!!

 

[g0dM@n]

*EDIT*
OK, I was able to add the USB device. I didn't know you had to add the USB controller and then go back and ADD HARDWARE again. I was deleting the USB controller and then going back and trying to re-add the "controller" every time. Ha!

Let's hope this does the trick. I lowered the Cyberpower VM to 256MB. I'll test and ensure it functions correctly.
I have to now inject the AGENT OVF instead of the client, since it's a direct USB connetion now.

Learned something new today!! I don't need a separate PCI USB card for each VM...

 

[HammerSandwich]

Yes, you're on the right track now. Agent VM is the only software you need; it'll talk straight to ESXi. Be sure ESXi configuration has the correct VM shutdown details.

BTW, my Agent currently shows <100MB guest RAM. You could probably cut it down to 128MB without a problem.

 

[HammerSandwich]

Also, if a VM has a VT-d connection, hardware cannot be added or removed while the VM runs. That's strictly a passthrough limitation. IOW, if you replace your Windows VM's PCI USB with the virtual controller, you will be able to attach & remove USB devices without rebooting the VM. Works nicely for flash drives, etc.

 

[g0dM@n]

Yes, you're on the right track now. Agent VM is the only software you need; it'll talk straight to ESXi. Be sure ESXi configuration has the correct VM shutdown details.

BTW, my Agent currently shows <100MB guest RAM. You could probably cut it down to 128MB without a problem.

Also, if a VM has a VT-d connection, hardware cannot be added or removed while the VM runs. That's strictly a passthrough limitation. IOW, if you replace your Windows VM's PCI USB with the virtual controller, you will be able to attach & remove USB devices without rebooting the VM. Works nicely for flash drives, etc.

I'm going to stick it out with 256MB and see how it rolls. 128 or 256 is SO MUCH more acceptable than the 2GB they configured the OVF with.

I think the most valuable thing I learned is not needing VT-d for USB. I totally overlooked that. I'll probably run out of onboard USB ports at some point so I'll still need PCI, but if ESXi can use that PCI for more USB, then I'll just resort to the host sharing USB, rather than PCI passthrough.

 

[g0dM@n]

So it's working like a champ @ 256MB ram. I'm not going to lower it because one time I powered on the VM the memory spiked to roughly 130MB. Also, I have the timezone set up properly, I have it set to shutdown the ESX host (the host is configured to shutdown VMs in a specific order), and I get emails when the power goes out and back on, etc.

The first thing is I cannot set up is the battery thresholds: low batt, voltage range, etc.
It's all greyed out.

Second issue is the the UPS diagnostics doesn't seem to work. Nothing happens when I hit INITIATE. The same thing goes for when I hit START on the calibration.

 

[g0dM@n]

Yes, you're on the right track now. Agent VM is the only software you need; it'll talk straight to ESXi. Be sure ESXi configuration has the correct VM shutdown details.

BTW, my Agent currently shows <100MB guest RAM. You could probably cut it down to 128MB without a problem.

Were you able to alter the battery threshold? It's grey for me, as well as the voltage bounds.
Were you also able to run the diagnostics tests? If I click Initiate, the button clocks, but nothing else happens. If I click on START for the runtime calibration, same thing.

I'm doing all of this through accessing the VM agent remotely via IE and/or FF (port 3052).

 

[michalrz]

I'm glad you got it working. Gotta say that add hub and then go back business sounds backwards.
Just bumped in to ask if this root hub sharing business is applicable when you're running the host off a thumbdrive?

 

[HammerSandwich]

Were you able to alter the battery threshold? It's grey for me, as well as the voltage bounds.
Were you also able to run the diagnostics tests? If I click Initiate, the button clocks, but nothing else happens. If I click on START for the runtime calibration, same thing.

Voltage limits are adjustable. Low-battery percentage is not.

Battery test works fine but doesn't update the date.

Last time I tried it, runtime calibration began OK but aborted after a while. Just kicked it off again, so we'll see.

 

[g0dM@n]

I'm glad you got it working. Gotta say that add hub and then go back business sounds backwards.
Just bumped in to ask if this root hub sharing business is applicable when you're running the host off a thumbdrive?

Add hub and go back business? Not sure I got you there... sorry

But as far as the root usb hub, I'm not sharing the root usb hub. Yes, I'm running ESXi off of a flash drive. The ESXi host sees the flash drive and ESXi does the passthrough of it AS a USB device, not as VT-d. I stupidly thought I had to use VT-d with an entire USB hub in order to get a USB recognized on a VM! The cool thing is the host itself can passthrough just the USB device, assuming the ESXi host can see it.

Voltage limits are adjustable. Low-battery percentage is not.

Battery test works fine but doesn't update the date.

Last time I tried it, runtime calibration began OK but aborted after a while. Just kicked it off again, so we'll see.

That's lame that we can't adjust the low batt percentage. I'll look at the UPS itself physically next chance I get... it's in the basement by my patch panel, so I'm being lazy

I can't adjust voltages either, though, and the test buttons and calibration buttons only click... but do nothing at all. I've tried with IE and FF.

 

[HammerSandwich]

So, my tests were interrupted by a 2-1/2 hour outage. Will report if I try again.

 

[michalrz]

Add hub and go back business? Not sure I got you there... sorry

My bad. I was thinking about what you said about adding hw in the correct order:

I didn't know you had to add the USB controller and then go back and ADD HARDWARE again

By hub I meant controller. I'm not as familiar with ESXi as I'm with KVM

 

[g0dM@n]

I did have this darn thing emailing me when the plug was pulled (or power out), but it hasn't been working. My test email works, but if I pull the plug out of the outlet, it never emails me. I also still can't change the battery threshold (and now it's grey at 0%), so I'm going to rebuild the VM. =/

*EDIT*
Imported OVF, still same issue with battery threshold. I checked EVENTS and I see 10 options as opposed to 9 which I had before. I'll test unplugging from the outlet and hope it'll email me. I have test emails working on the new setup.
I also powered the old VM back up, and even though it didn't email when battery was engaged, it did email me that it lost connectivity to the UPS (since I moved the USB over to the new VM). Weird...

So... I don't trust these Agent VMs, but I guess it's the only option I have.

 

[michalrz]

Can't help you with the threshold problem, but this is a red flag:

but if I pull the plug out of the outlet, it never emails me

But networking works fine aside from that, right? No ethernet disconnects?
How did you 'test' the e-mail? Who's the SMTP server - the hypervisor? a guest VM? ISP?

 

[g0dM@n]

Can't help you with the threshold problem, but this is a red flag:



But networking works fine aside from that, right? No ethernet disconnects?
How did you 'test' the e-mail? Who's the SMTP server - the hypervisor? a guest VM? ISP?

Networking is fine because I stay logged in and always able to log into the web interface. I don't see any disconnects at all. The test was dona via SMTP with outgoing.verizon.net (FIOS) with authentication over default port 25. The test works EVERY time.

Hypervisor is ESXi 5.5.0 1331820

I could try port 587, but what difference does it make if the TEST goes... I'd think if the test works that the alerts should work too. Maybe I can compare the events I create to the logs to see if they're even logged.

But... again, what's funny is that when I turned on the old VM, it quickly sent an email about not having connection to the UPS (since I moved it to the new VM).

 

[g0dM@n]

Finally got alerts to work by restarting the ESXi host. I still can't change the battery threshold value, so I guess I'll have to test leaving the host running on battery and hope it doesn't ungracefully shutdown.

It's not that it couldn't send an email, the agent VM was never notified of events (when I check its event logs). Now I see entries in there that match up with 3 emails I'll get in a normal, short power failure event:
1. power failure, battery power supplied
2. power restored
3. battery full

I see them all now. I'm definitely a little skeptical as to how reliable this setup is....

 

[michalrz]

You were using an outside SMTP server, and while I am not a specialist in e-mail, I do know that port 25 is often dropped, and 587 accepts only SMTP Auth - authorization.
These mail issues are a potential point of failure.

I know you won't like this, but it would probably a viable option if you installed a free simple SMTP server on the Agent VM and configure it to operate on the LAN and only accept the "Agent's" credentials.

Thus, the UPS software will authenticate to the 'localhost' and store mail there on itself, which should be faster and not prone to networking failures.

You will be then able to normally add this account to your e-mail clients which you have on your LAN machines in order to see the notifications.

 

[g0dM@n]

You were using an outside SMTP server, and while I am not a specialist in e-mail, I do know that port 25 is often dropped, and 587 accepts only SMTP Auth - authorization.
These mail issues are a potential point of failure.

I know you won't like this, but it would probably a viable option if you installed a free simple SMTP server on the Agent VM and configure it to operate on the LAN and only accept the "Agent's" credentials.

Thus, the UPS software will authenticate to the 'localhost' and store mail there on itself, which should be faster and not prone to networking failures.

You will be then able to normally add this account to your e-mail clients which you have on your LAN machines in order to see the notifications.

It's actually working now, ever since my ESXi reboot. I do need the emails going to gmail. This is for home use, so I don't have an exchange server here on my LAN.

The problem wasn't email... the problem was communication between the UPS and the Agent VM... I know this because events wouldn't even populate in the event logs.