Advice on Running Scripts Remotely and Seeing STDOUT In Real Time

[svet-am]

At work, I handle some scripts that we use to spread work amongst our server farm. We have some pre-existing scripts that constantly monitor load on the farm. I did not write these scripts and since they work well, we'd like to leave them alone.

What my script does is call the loading script to figure out which machines have available resources and then dispatch your job over to that machine.

Here's where the problem comes in. Currently, my script uses rsh to fire-and-forget the job. Rsh spools STDOUT and then reports the whole contents of STDOUT for that job in one shot once the process finishes.

Well, some of our runs are taking long and longer to complete and we'd like to be able to monitor STDOUT from the remote machine for that process in real time.

Is there a way to do this?

 

[nameless_centurian]

<edit> warning, i have never used RSH, only ssh and telent, so i may be talking out of my ass. take my advice with a grain of salt.</edit>

yes.. print the output out as it is generated???

rather than spooling it, print it. print it to a file, print it to a database. print it through a pipe to another process that does the spooling and will allow you to query the output.

in all the scenarios, the result is the same.. you have to edit the original scripts.

 

[svet-am]

<edit> warning, i have never used RSH, only ssh and telent, so i may be talking out of my ass. take my advice with a grain of salt.</edit>

yes.. print the output out as it is generated???

rather than spooling it, print it. print it to a file, print it to a database. print it through a pipe to another process that does the spooling and will allow you to query the output.

in all the scenarios, the result is the same.. you have to edit the original scripts.

thanks for the suggestion, but what are you talking about?!?

my script simply calls a different command on a remote machine. I can't make specific since it would divulge some of what we do here, but let me give an example using the simple ls command.

currently, you call my script with:

load_balance <command>

load_balance is my script and <command> is the command that you wish to execute remotely.

So, when you call it with:

load_balance ls

then, the load_balance script first checks for an "available" machine and then uses rsh to dispatch the command to that machine in a 100% fire and forget manner. In this case, we'd be executing ls. To be fair, generally ls will complete in a fraction of a second and so you get the illusion of instantaneous results. However, the reality is that rsh spools STDOUT and only reports STDOUT back to the host that called it when the process completes.

Some of the commands that we call take a couple of days to complete. During that time, you're seeing NOTHING reported back from rsh.

So, what I want is an option where you can call:

load_balance <command>

and still see the results coming back in realtime.

I've looked at ssh for this, but the only way to get it to be fire-and-forget compatible is to disable the ssh login functionality, which we cannot and will not do for security reasons.

I need my users to be able to call one command and one command only and get realtime STDOUT output from a different machine.

How does your idea accomplish this?

 

[nameless_centurian]

let me make myself more clear --if whatever script/command your are calling on the remote end doesn't produce any output until it is completed you HAVE to modify that script/command.

now, if the issue is RSH itself, then you should be able to "fire and forget" that remote script/command but still tell it to redirect output to a file or another process that will spool the output. if you write another small client that just dumps the last x number of messages or x number of minutes of output, you should be able to look at the new process to view the spool in real-time.

edit: again though, this may require changing (ie, re-coding) the remote script/command

 

[svet-am]

let me make myself more clear --if whatever script/command your are calling on the remote end doesn't produce any output until it is completed you HAVE to modify that script/command.

now, if the issue is RSH itself, then you should be able to "fire and forget" that remote script/command but still tell it to redirect output to a file or another process that will spool the output. if you write another small client that just dumps the last x number of messages or x number of minutes of output, you should be able to look at the new process to view the spool in real-time.

edit: again though, this may require changing (ie, re-coding) the remote script/command

the remote commands that we're calling (internal implementation tools for FPGA's) print A LOT to STDOUT. In fact, our tools are command line tools by nature so their default mode is to print to STDOUT all of the data that we need, so that's not a problem.

I just need a way to report back STDOUT in real time to the machine that's calling it remotely.

 

[CEpeep]

Can't you just run them over SSH? Then you can see it in the SSH process's window and watch what's going on. If you set it to run in Screen, it will continue to run if you accidentally close the window or get logged out.

Also, rsh isn't a good idea anymore. It uses completely unencrypted transmission, so it's a good idea to move your workflow over to SSH anyway.

 

[svet-am]

Can't you just run them over SSH? Then you can see it in the SSH process's window and watch what's going on. If you set it to run in Screen, it will continue to run if you accidentally close the window or get logged out.

Also, rsh isn't a good idea anymore. It uses completely unencrypted transmission, so it's a good idea to move your workflow over to SSH anyway.

We absolutely must have the fire and forget functionality that RSH provides. SSH requires you to supply a password (unless you disable SSH authentication altogether) and that entirely breaks our use case.

 

[mrgstiffler]

We absolutely must have the fire and forget functionality that RSH provides. SSH requires you to supply a password (unless you disable SSH authentication altogether) and that entirely breaks our use case.

I believe you can use a certificate with an empty passphrase to connect.

 

[CEpeep]

We absolutely must have the fire and forget functionality that RSH provides. SSH requires you to supply a password (unless you disable SSH authentication altogether) and that entirely breaks our use case.

You can either use an authentication certificate or specify the password right in the run line for the command. You can also authenticate as a user that has no password.

If you must use rsh, why do you need the output on STDOUT? Can't you just redirect it to another terminal or to a file?

 

[svet-am]

You can either use an authentication certificate or specify the password right in the run line for the command. You can also authenticate as a user that has no password.

If you must use rsh, why do you need the output on STDOUT? Can't you just redirect it to another terminal or to a file?

We cannot supply the password because it is MY script that is doing the call to the remote box and I have no way of anticipating the user's password.

No, I cannot redirect to a different terminal or file. We need STDOUT to be sent back to the calling machine in real time so that we can make judgement calls about the results we're seeing and end the process early rather than waiting for it to spend it's entire runtime and then return. As I mentioned previously, some of our runs would take a few days to complete if we let them run to completion and that's a lot of wasted time for a run whose results we disregard anyway.

 

[CEpeep]

We cannot supply the password because it is MY script that is doing the call to the remote box and I have no way of anticipating the user's password.

No, I cannot redirect to a different terminal or file. We need STDOUT to be sent back to the calling machine in real time so that we can make judgement calls about the results we're seeing and end the process early rather than waiting for it to spend it's entire runtime and then return. As I mentioned previously, some of our runs would take a few days to complete if we let them run to completion and that's a lot of wasted time for a run whose results we disregard anyway.

Are you currently running something like this?

rsh hostname yourcommand.sh >> /dev/stdout

If so, that's the best you're going to get. If you set it to run in Screen, then all you need to do is connect to that Screen and you'll get a realtime display of whatever the command is outputting.

 

[svet-am]

Are you currently running something like this?

Nope, what we're currently doing is:

rsh <command>

Piping the output of rsh doesn't change how fast the data is returned to STDOUT on the local machine because RSH spools the contents of STDOUT and _only_ returns them once the process has either finished or died. Thus, even if you pipe the output, it still won't get piped until the target process completes.

 

[CEpeep]

Thus, even if you pipe the output, it still won't get piped until the target process completes.

You have just answered your own question. There are several ways to do what you want, but not without modifying the way you are running them. You can either switch to a different way of running the commands or you can switch to a different method of outputting the data. If you must continue using rsh and require the output on /dev/stdout, there is no way to do what you want.

 

[svet-am]

You have just answered your own question. There are several ways to do what you want, but not without modifying the way you are running them. You can either switch to a different way of running the commands or you can switch to a different method of outputting the data. If you must continue using rsh and require the output on /dev/stdout, there is no way to do what you want.

I know we cannot use RSH and we dont WANT to use RSH anymore. What I need is the fire-and-forget functionality.

 

[CEpeep]

I know we cannot use RSH and we dont WANT to use RSH anymore. What I need is the fire-and-forget functionality.

You can use SSH with keyfile authentication. As long as the machine that starts the SSH connection has the keyfile on it, you will be able to log in without any authentication prompts. You could also just use telnet, if you don't want to deal with any sort of authentication. It's not secure at all, but neither is rsh.

 

[svet-am]

You can use SSH with keyfile authentication. As long as the machine that starts the SSH connection has the keyfile on it, you will be able to log in without any authentication prompts. You could also just use telnet, if you don't want to deal with any sort of authentication. It's not secure at all, but neither is rsh.

I'll bring this up at our meeting today, but I don't think it's going to fly since our farm has about 500 machines in it and the keyfile method will require setting up the keyfile on every single remote machine.

thanks for the input, everyone.

 

[Sgarissta]

You're asking for asynchronous command running, but synchronous output? Those things are by definition mutually exclusive. I don't know much about RSH and how it runs commands, but here's what I'd do.

The 'command' I ran on the remote machine, would pass through a wrapper on the target machine. This wrapper would pipe STDOUT to something that either wrote the data to a database of some sort, sent it to a log on the original host, etc. This way you can still monitor the output, without waiting on the command to actually finish. Obviously you'd need to have the wrapper script installed on every machine, but I don't know a way around that.

 

[svet-am]

for the record, I got this working by having my script use ssh instead of rsh. it requires the user to enter a password, thus destroying our fire-and-forget methodology, but after speaking with the stakeholders on this, it was worth it to get us the realtime functionality.

for those intersted, here's the code in my bash script:

$o = ssh $remote_host "$commands";
print $o;

 

[nameless_centurian]

glad it worked out for you, though I do not follow how being forced to leave a terminal running and come to the same terminal every time you want to check on it is extremely beneficial when you could achieve exactly what you want with some of the methods listed earlier.

 

[svet-am]

glad it worked out for you, though I do not follow how being forced to leave a terminal running and come to the same terminal every time you want to check on it is extremely beneficial when you could achieve exactly what you want with some of the methods listed earlier.

it's beneficial because it replicates exactly what our tools do already (ie, output to STDOUT), but allow the job to be distributed out amongst our cloud of computers in our farm.

plus, whether it's "good" or "bad", it's what our users want and expect and we're maintaining (as closely as possible) their current experience.

 

[nameless_centurian]

plus, whether it's "good" or "bad", it's what our users want and expect and we're maintaining (as closely as possible) their current experience.

QFT.

I'm pretty new to the SW engineering and integration world, but i have 'discovered' that if your stakeholders are happy with a simpler solution and know the disadvantages, it's hard to argue with them.