High Availability Clustering Across the Board

[hokatichenci]

Ok so in our LUG we have a lot of cheap hardware. As we all know, cheap hardware tends to fail a lot. So for the past while I've been tinkering a lot with clustering, trying to make a very high availability solution. So far, I've come to discover a lot of software that can be glued together to form a really nice solution, but I'm not quite there yet. What I'd like is to kind of share my knowledge and find out what other people know, so perhaps some missing blocks can be filled in on any side.

If people are really interested in this stuff, I can try and write a tutorial tying in everything to build a web cluster. If you find anything interesting, please say so. This is a electrical taped set of information so I might not have hit everything I needed to as well.

The basic concept is that every (system) point of failure should have some sort of backup, or redundancy. Currently I've been focusing on our web server, since that is where most of the failures tend to happen when we get hit hard by crawler bots or linkfests or whatever. So far my weapon of choice has been the Linux Virtual Server (LVS) project, not to be confused with the other "Virtual Server" project that deals with system virtualization (like VMWare or whatever). Since LVS on its own only load balances/forwards and doesn't do any heartbeating, I paired it with keepalived because of its simplicity and VRRP integration (I'll get into that later).

So basically, we have the director which holds the virtual ip address (where www.blah.com resolves) and the actual web servers (SRIP1/2/3 etc for Server Real IP Address) and the database servers (DB1/2/3 or whatever). The first problem I encountered was that I had no backup mechanism for the director, so whenever the head node went down nothing was accessible because there was no system to point people anywhere. I'm just now beginning to utilize the VRRP framework which is very easy to configure so that multiple directors will automatically negotiate over the virtual ip address specified so that if the master director goes down, another will pick up the slack.

Our primary software run on our web server is mediawiki, which thanks to Wikipedia has immense support for clustering. The first thing to do was enable and install memcached which is a caching mechanism where the web servers can centralize login information and queries and whatnot. This made things a lot more responsive and allowed for login sessions to spread across multiple servers (so you can truly round robin schedule). I then discovered a fancy application called Sharedance that I've yet to try, but is supposed to transparently centralize PHP sessions over TCP, so it should basically instant-cluster-aware any PHP application without any code changing so long as it doesn't rely on off-shot directories that can't be shared (localized /tmp for example).

Now here is where I am hitting a brick wall: the database. We run postgres and mysql, and both seem to have meager clustering support at best. Postgres doesn't seem to have any support, and MySQL has Master/Slave that has issues with syncing or NBD which doesn't have support for any features. The best option so far seems to be a network distributed file system with strong posix locks (read: correct and fast), so the database servers would basically be using the same directory just as if multiple threads or processes were hitting the same locks/files. This of course adds in a whole new layer to worry about, and the availability of that layering comes into question. NFS is pretty much out, GFS seems to be out, and the only remaining solution that appears to work is PeerFS (I don't know if Lustre supports this kind of system), which unfortunately costs ~$1k USD per license. I'd hate to have to rely on NBD or Master/Slave operation, so a distributed already high availability filesystem really seems to be the way to go.

Theres a ton of other services I have yet to hit with the HA stick such as the fileserver (NFS), LDAP, and mail. Has anyone done

Cliff notes:
Operating System: Linux 2.6 with LVS Support Enabled in the Kernel (OpenBSD with pf/carp experience anyone?)
Heartbeat software for services: Keepalived using VRRP for director failover (heartbeatd, ucarp, other experiences?)
Home directories: NFS shared (NFS server is still a SPOF)
Mediawiki solutions: memcached (kind of a SPOF, minimal HA ability)
PHP Sessions: Sharedance?
MySQL: ?
Postgres: ?

Distributed filesystem: Lustre?
Mail: ?
LDAP: LVS + Master/Slave operation?
NFS Shares: ???