Managing *Nix Clusters

[hokatichenci]

So for those of you who manage multiple environment clusters with different tasks, how do you do it? Currently I use a script that just ssh's to every server and runs a command, ie:
ssh_all.sh "rpm -U /tmp/newpackage-2.0.rpm"

This is less than ideal since it isn't threaded and can't really do anything special. It also becomes a pain if say, I don't want to run the command on every server, I want to run it on say, just the database servers. That requires adding a script called say, ssh_database.sh, and then of course, what if you're running 20 services, how do you manage all that? I looked into other alternatives, and there seems to be a really old perl script that isn't available anymore called clsh, which does what I want. I also found a project called clusterssh, but that works with xterms and I really don't want 500 xterm windows open on my system. Plus, it isn't portable - it needs to be able to run from the command line.

So ideally I'm looking for something that can -
* manage server attributes and be able to execute on those attributes (services offered, properties, operating system (solaris9 vs solaris10 vs rhel5 vs gentoo)
* quickly load these settings for minimal overhead
* Execute these things in a variable parallel form, 1 at a time, 5 at a time, 50 at a time, etc
* Delay commands, ie, I want to reboot all the servers, but I want it staggered by 30seconds

Other things that would rock -
* be able to diff output (ie, servera has known good output, serverb/c/d/e should compare their output to servera and print differences)
* resilient scripting - issue a bunch of a commands and if one fails do a rollback to a known good configuration
* robust search/replace
* command replacement / easy file pushing, ie: "connect all_servers push file.rpm" -> servers do a wget/scp/whatever to a known fileserver that'll have it

So far most of the stuff that are really necessary are built into a python script that I wrote, however the overhead is a bit much.