Best UPS Hack for ESXi Free Edition?

[Zarathustra[H]]

Hey all,

I just picked up an APC SUA1500 for my server.

I have all the guests set up with VM Tools to auto shutdown when the system shuts down.

My SUA1500 has an AP9617 Network Management Card, Serial interface and USB interface.

I've read about having a small Linux guest with APCUPSd send SSH shutdown commands to ESXi, but that requires having SSH permanently enabled on the host.

Are there any other good methods?

Which method, using which interface would you use?

Appreciate the help!

--Matt

 

[danswartz]

How about just have each guest monitor the UPS and shutdown? No-one really cares about the host - nothing to get corrupted, AFAIK.

 

[bds1904]

Linux guest running nut and then use the ESXi nut client.

 

[Zarathustra[H]]

How about just have each guest monitor the UPS and shutdown? No-one really cares about the host - nothing to get corrupted, AFAIK.

I wound up with a corrupt local data store that way once.

And the vmfs doesn't have any good tools for repair, unfortunately.

 

[Zarathustra[H]]

Linux guest running nut and then use the ESXi nut client.

I read about Nut once. Why do you prefer it to APCUPSd?

The article I read seemed to suggest I needed a GUI for Nut, is that accurate?

Hoping to keep the UPS control guest console only to keep RAM use low.

Thanks for the suggestions guys!

 

[danswartz]

I wound up with a corrupt local data store that way once.

And the vmfs doesn't have any good tools for repair, unfortunately.

That's a bummer. It isn't supposed to work that way, I didn't think

 

[Zarathustra[H]]

That's a bummer. It isn't supposed to work that way, I didn't think

Well, to be clear, it wasn't because of a UPS running out of power and a hard shutoff, it was because of powerloss with no attached UPS.

Either way, I'd rather have the host shut down cleanly.

 

[dave99]

I use the apcupsd agent on guest that ssh's into the host and runs the shutdown script.

 

[danswartz]

"Well, to be clear, it wasn't because of a UPS running out of power and a hard shutoff, it was because of powerloss with no attached UPS."

Sorry, that's what I meant. I thought vmfs was more resilient than that...

 

[Zarathustra[H]]

I use the apcupsd agent on guest that ssh's into the host and runs the shutdown script.

Something like this?

I used this method last night, except I set it up with serial instead of USB.

The network management card was just a mess to get working properly (still nice to have to check in on the UPS remotely, just a pita for apcupsd)

I have had enough troubles with USB forwards to guests in the past, that I wasn't comfortable with it's reliability, so serial it was.

SHOULD be working, but haven't tested the scripts yet, as the server has been doing stuff I don't want to interrupt.

Looking forward to testing it though!

 

[dave99]

yeah, except from a windows vm. And I shut down early, like at 50-60% battery, as I had a power outage run out of juice right before the last vm (a windows sbs server, which can take a while to power off) was shutting down. Corrupted the vm (was able to fix luckily), and corrupted the esxi install itself which required a reinstall.

 

[Zarathustra[H]]

yeah, except from a windows vm. And I shut down early, like at 50-60% battery, as I had a power outage run out of juice right before the last vm (a windows sbs server, which can take a while to power off) was shutting down. Corrupted the vm (was able to fix luckily), and corrupted the esxi install itself which required a reinstall.

Yeah, I was concerned about that, but I did a few timed shutdowns, and shutting down all my guests and ESXi itself takes about 2min 35 sec, so I figure if I tell it to shut down with 5 min remaining that's almost a 100% buffer. Ought to be enough. (I hope)

 

[dave99]

Just remember that the runtime isn't always perfect, I don't trust the number at all, especially as the battery voltage starts dropping, that last few minutes can go quicker. I wouldn't cut it that close by a long way, but YMMV. Too much potential pain if it doesn't shut down quick enough. For biz clients, that extra runtime probably doesn't matter, because it will outlast the workstation UPS's, and for my home system, I'd like to have that extra power to run a LED light if it's an extended outage. Besides, my home UPS outlasts the dsl connection battery wherever it is upstream. I can go about 100 minutes, the dsl stops after about 20.

 

[Zarathustra[H]]

Just remember that the runtime isn't always perfect, I don't trust the number at all, especially as the battery voltage starts dropping, that last few minutes can go quicker. I wouldn't cut it that close by a long way, but YMMV. Too much potential pain if it doesn't shut down quick enough. For biz clients, that extra runtime probably doesn't matter, because it will outlast the workstation UPS's, and for my home system, I'd like to have that extra power to run a LED light if it's an extended outage. Besides, my home UPS outlasts the dsl connection battery wherever it is upstream. I can go about 100 minutes, the dsl stops after about 20.

Interesting, thank you.

This is my first UPS, so I did not know the remaining time was inaccurate.

I figured I'd be safe by doubling the time I know it takes my server to shut down.

I know that compared to an idle state, it would probably use more power during shutdown, shortening the time available, but my thought process was that it would be nowhere near double, so doubling the time would give me more than ample shutdown time.

I have changed it to 10 minutes. Hopefully that will be enough.

 

[feffrey]

I need to configure this for my Hyper-V cluster at home.
At work it is nice having a UPS backed by a auto on gen. As long as we got diesel and nothing fails we aren't going down due to power loss.

 

[Zarathustra[H]]

Alright, so I just got around to testing it.

Unfortunately no auto-shutdown. Any suggestions greatly appreciated!

I set everything up as per this example.

The relevant section of my /etc/apcupsd/apccontrol looks like this:

# WARNING: the apccontrol file will be overwritten every time you update your
# apcupsd, doing `make install'. Your own customized scripts will _not_ be
# overwritten. If you wish to make changes to this file (discouraged), you
# should change apccontrol.sh.in and then rerun the configure process.
#
if [ -f ${SCRIPTDIR}/${1} -a -x ${SCRIPTDIR}/${1} ]
then
    ${SCRIPTDIR}/${1} ${2} ${3} ${4}
    # exit code 99 means he does not want us to do default action
    if [ $? = 99 ] ; then
        exit 0
    fi
fi

case "$1" in
    killpower)
        echo "Apccontrol doing: ${APCUPSD} --killpower on UPS ${2}" | ${WALL}
        sleep 300
        ${APCUPSD} --killpower
        echo "Apccontrol has done: ${APCUPSD} --killpower on UPS ${2}" | ${WALL}
    ;;
    commfailure)
        echo "Warning communications lost with UPS ${2}" | ${WALL}
    ;;
    commok)
        echo "Communications restored with UPS ${2}" | ${WALL}
    ;;
#
# powerout, onbattery, offbattery, mainsback events occur
#   in that order.
#
    powerout)
    ;;
    onbattery)
        echo "Power failure on UPS ${2}. Running on batteries." | ${WALL}
    ;;
    offbattery)
        echo "Power has returned on UPS ${2}..." | ${WALL}
    ;;
    mainsback)
        if [ -f /etc/apcupsd/powerfail ] ; then
           printf "Continuing with shutdown."  | ${WALL}
        fi
    ;;
    failing)
        echo "Battery power exhaused on UPS ${2}. Doing shutdown." | ${WALL}
    ;;
    timeout)
        echo "Battery time limit exceeded on UPS ${2}. Doing shutdown." | ${WALL}
    ;;
    loadlimit)
        echo "Remaining battery charge below limit on UPS ${2}. Doing shutdown." | ${WALL}
    ;;
    runlimit)
        echo "Remaining battery runtime below limit on UPS ${2}. Doing shutdown." | ${WALL}
    ;;
    doreboot)
        echo "UPS ${2} initiating Reboot Sequence" | ${WALL}
        ${SHUTDOWN} -r now "apcupsd UPS ${2} initiated reboot"
    ;;
    doshutdown)
        echo "UPS ${2} initiated Shutdown Sequence" | ${WALL}
        ${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"
        echo "****** Executing ESXi Shutdown Command ******" | ${WALL}
        plink -ssh -2 -pw PASSWORD root@xxx.xxx.xxx.xxx "/sbin/shutdown.sh && /sbin/poweroff"
    ;;
    annoyme)
        echo "Power problems with UPS ${2}. Please logoff." | ${WALL}
    ;;
    emergency)
        echo "Emergency Shutdown. Possible battery failure on UPS ${2}." | ${WALL}
    ;;
    changeme)
        echo "Emergency! Batteries have failed on UPS ${2}. Change them NOW" | ${WALL}
    ;;
    remotedown)
        echo "Remote Shutdown. Beginning Shutdown Sequence." | ${WALL}
    ;;
    startselftest)
    ;;
    endselftest)
    ;;
    battdetach)
    ;;
    battattach)
    ;;
    *)  echo "Usage: ${0##*/} command"
        echo "       warning: this script is intended to be launched by"
        echo "       apcupsd and should never be launched by users."
        exit 1
    ;;
esac

When the time remaining goes below 10 min, I get the following message as expected, then the guest powers down:

****** Executing ESXi Shutdown Command ******

And then nothing....

ESXi stays running as if nothing has happened.

EXCEPT, maybe 5-10 minutes after the shutdown script has triggered, the ESXi does a random reboot without properly shutting down the guests. (exactly what I am trying to avoid GRRRR)

The funny thing is, if I copy and paste the "plink" command verbatim from the script, and paste it into the console on my APC guest, everything works beautifully... ESXi shuts down cleanly as expected.

I can't figure out why it is behaving like this? Maybe the script is causing the guest to shut down before the plink command to the ESXi host is issued completely, causing ESXi to get all confused and eventually reboot?

As I see it, I shouldn't even need a command to shutdown the guest in the apcupsd script, as following a successful plink to the host, the guest should auto shutdown cleanly anyway...

One thing I thought of trying was disabling the portion of the script that shuts down the guest, and seeing if that does the trick, but I am terrible at bash scripting, and can't seem to figure out how to do that part without breaking the rest of the script.

Any ideas?

Thanks,
Matt

 

[dave99]

did you ssh from the account that apcupsd is running under once, so you could confirm you wanted to add to known hosts?

 

[Zarathustra[H]]

did you ssh from the account that apcupsd is running under once, so you could confirm you wanted to add to known hosts?

Yep, as mentioned, if I copy and paste the entire plink command from the script into the console, it works exactly as it should. (I tested this, before doing an actual "disconnect power test".)

It shuts down all the guests in an orderly fashion, before turning itself off, exactly like I want.

When I let the script do it automatically, however, something doesn't work.

SOMETHING is happening, as the host gets confused and reboots a few minutes later, but it never properly shuts down the guests, and then comes right back up rather than shutting down.

Very odd behavior.

 

[dave99]

is apcupsd running under root or some other service type account?

 

[Zarathustra[H]]

is apcupsd running under root or some other service type account?

It is running as root.

 

[Zarathustra[H]]

Maybe I will just try the NUT method above instead, and see if that works any better...

 

[dave99]

I'd probably try moving the commands to a separate shell file and executing the sh file from the apcupsd shutdown script.

 

[Zarathustra[H]]

I'd probably try moving the commands to a separate shell file and executing the sh file from the apcupsd shutdown script.

I might try that. Do you know how I can disable the apcupsd poweroff portion of the script? I think I'd rather just have ESXi shut down the apcupsd guest, that way I know the command to ESXi completes.

 

[dave99]

probably just rem out the:
sleep 300
${APCUPSD} --killpower

part.

 

[Zarathustra[H]]

probably just rem out the:
sleep 300
${APCUPSD} --killpower

part.

I think --killpower is the part responsible for shutting off the UPS (not sure why you'd even want to do that, rather than just let it continue running after system shutdown, but..)

 

[dave99]

oh, I thought that is what you were wanting. Reason for shutting down the UPS is it can trigger a restart when power returns on server systems. If the UPS remains running, but with no load, it will have a long runtime. The power supplies will never see the power actually turn off, so they won't know when it's returned. Versus shutting down the UPS, when the power returns the UPS turns back on, the system power supplies see power return, and after whatever delay you have programmed in the BIOS, the server powers on and everything recovers automatically.

 

[Zarathustra[H]]

oh, I thought that is what you were wanting. Reason for shutting down the UPS is it can trigger a restart when power returns on server systems. If the UPS remains running, but with no load, it will have a long runtime. The power supplies will never see the power actually turn off, so they won't know when it's returned. Versus shutting down the UPS, when the power returns the UPS turns back on, the system power supplies see power return, and after whatever delay you have programmed in the BIOS, the server powers on and everything recovers automatically.

Ahh, that makes sense...


No, what I'd like to disable is the system shutdown, (that in this configuration issues a shutdown -P now (or something) command to the linux guest, and instead ONLY have it send the commands to ESXi for shutdown.

 

[dave99]

got it, remove this piece then:
${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"

 

[Zarathustra[H]]

got it, remove this piece then:
${SHUTDOWN} -h now "apcupsd UPS ${2} initiated shutdown"

Going to have to try this next.

In the interim, I restarted but using NUT instead of apcupsd, as NUT actually has a client VIB for ESXi shutdowns.

I learned that NUT may be great for some UPS:es, but APC compatibility is absolute garbage, and shouldn't even be considered beta.

My SUA1500 is an "unrecognized device" via serial connection, and thus doesn't work, via USB and SNMP (using the AP9617 card) I just get segmentation faults and other driver crashes.

Anyway, so back to APCUPSd...