screen for cluster management

After my parallel ssh roundup and hacking my own solution...

I just realized screen can do this too using "at" and "stuff".  Here's how:

1. open a bunch of screens and ssh to various hosts
2. get to a screen command prompt (control key followed by colon, ^C-a : )
3. at the prompt, enter the command "at \#", this tells screen to run command that follows on all windows (yes, include the backslash and don't press enter yet)
4. continue with the command "stuff", which tells screen to quite literally stuff whatever follows it into the input of a screen terminal (don't press enter quite yet)
5. follow stuff with the command you want to run, you'll need to put it in quotes if the command has arguments (not yet)
6. lastly, you need to end the line with a return so the shell(s) will run the command, so end the line with "\012" which is octal for a newline character

Now, press enter and what your command get run across all of your screen sessions.  This is particular cool for long running commands where you can switch back and forth between screen windows and check on the progress of many machines or also useful if you have a couple of quick maintenance task to run on a bunch of hosts.

So, putting it all together, the whole sequence looks like this:

^C-a : at \# stuff ls \012
^C-a : at \# stuff 'ls -alR' \012