• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Citrix Xen Server FT

D

DESmack

Weaksauce
Joined
Sep 1, 2004
Messages
95
Hi,

I am trying to find out if anyone in Team [H] has used the Remus technology in Citrix Xen server to implement Fault Tolerance for a VM.

http://dsg.cs.ubc.ca/remus/index.html#about

I want to know if it supports a virtual machine the has more than one core allocated, thus in the event of a node failure, the vm continues to run on the backup/remote physical node...Does this also work with Redhat Linux.

I also want to know what type of bandwidth the system needs to facilitate the asynchronous replication?

Kind regards,

Desmack.
 
I spoke to the product manger of Citrix Xen Server in the UK, via senior Pre sales engineer. They have no plans to put this technology into Citrix Xen Server. :(

These are some other notes I found out from a very long slashdot.org thread....


--------------------


Overview

http://www.usenix.org/events/nsdi08/tech/full_papers/cully/cully_html/

Remus runs paired servers in an active-passive configuration. We employ three major techniques in order to overcome the difficulties traditionally associated with this approach. First, we base our system on a virtualized infrastructure to facilitate whole-system replication. Second, we increase system performance through speculative execution, which decouples external output from synchronization points. This allows the primary server to remain productive, while synchronization with the replicated server is performed asynchronously. The basic stages of operation in Remus are given in Figure


Remus is a novel system for retrofitting high availability onto existing software running on commodity hardware. The system uses virtualization to encapsulate a protected VM, and performs frequent whole-system checkpoints to asynchronously replicate the state of a single speculatively executing virtual machine.
Providing high availability is a challenging task and one that has traditionally required considerable cost and engineering effort. Remus commodifies high availability by presenting it as a service at the virtualization platform layer: HA may simply be “switched on” for specific virtual machines. As with any HA system, protection does not come without a cost: The network buffering required to ensure consistent replication imposes a performance overhead on applications that require very low latency. Administrators must also deploy additional hardware, which may be used in N-to-1 configurations with a single backup protecting a number of active hosts. In exchange for this overhead, Remus completely eliminates the task of modifying individual applications in order to provide HA facilities, and it does so without requiring special-purpose hardware.
Remus represents a previously unexplored point in the design space of HA for modern servers. The system allows protection to be simply and dynamically provided to running VMs at the push of a button. We feel that this model is particularly attractive for hosting providers, who desire to offer differentiated services to customers.




Bandwidth needed:

It depends pretty heavily on your workload. Basically, the amount of bandwidth you need is proportional to the number of different memory addresses your application wrote to since the last checkpoint. Reads are free -- only changed memory needs to be copied. Also, if you keep writing to the same address over and over, you only have to send the last write before a checkpoint, so you can actually write to memory at a rate which is much higher than the amount of bandwidth required. We have some nice graphs in the paper, but for example, IIRC, a kernel compilation checkpointed every 100ms burned somewhere between 50 and 100 megabits. By the way, there's plenty of room to shrink this through compression and other fairly straightforward techniques, which we're prototyping.


Split Brain Scenario.
Surely there is a strong possibility of a failure where both VMs run at once- the original image thinking it has lost touch with a dead backup, and the backup thinking the master is dead, and so starting to execute independently? If they're connected to the same storage / network segment, it could cause data loss, bring down the network service and so on. I've not investigated these types of lockstep VMs, but it seems you have to make some pretty strong assumptions about failure modes, which always break eventually commodity hardware (I've seen bad backplanes, network chips, CPU caches, RAM of course, switches...). How can you possibly handle these cases to avoid having to mop up after your VM is accidentally cloned?
Re:Wrong place to put a failsafe? (Score:4, Informative)
by bcully (1676724) writes: on Wednesday November 11 2009, @09:26PM (#30068270)
Split brain is a possibility, if the link between the primary and backup dies. Remus replicates the disks rather than requiring shared storage, which provides some protection over the data. But there are already a number of protocols for managing which replica is active (e.g., "shoot-the-other-node-in-the-head") -- we're worried about maintaining the replica, but happy to use something like linux-HA to control the actual failover.


e:Wrong place to put a failsafe? (Score:5, Interesting)
by dido (9125) writes: <`dido' `at' `imperium.ph'> on Wednesday November 11 2009, @09:51PM (#30068462) Homepage
This is something that the much simpler Linux-HA environment deals with by using something they call STONITH, which basically means to Shoot The Other Node In The Head. STONITH peripherals are devices that can completely shut down a server physically, e.g. a power strip that can be controlled via a serial port. If you wind up with a partitioned cluster, which they more colorfully call a 'split brain' condition, where each node thinks the other one is dead, each of them uses the STONITH device to make sure, if it is able. One of them will activate the STONITH device before the other, and the one which wins keeps on running, while the one that loses really kicks the bucket if it isn't fully dead. I imagine that Remus must have similar mechanisms to guard against split brain conditions as well. I've had several Linux-HA clusters go split brain on me, and I tell you it's never pretty. The best case is that they only both try to grab the same IP address and get an IP address conflict, in the worst case, they both try to mount and write to the same fiberchannel disk at the same time and bollix the file system. If a Remus-based cluster split brains, I can imagine that you'll get mayhem just as awful unless you have a STONITH-like system to prevent it from happening.
Answer:
I 've worked with Remus, so I can answer your question.
It's not "constantly going" into live migration. The backup image is constantly kept in a "paused" state. It doesn't come out of the paused state until communication with the original is broken.
Until the backup goes live, the shadow pages for memory are updated, via checkpoints. The checkpointing interval is somewhat variable, but it's actually hardcoded into the Xen software (at present - this will change), regardless of what the user level utility tells you.
As it is, the subsecond checking doesn't work too well. But intervals of about 1-2 seconds works great. Getting subsecond checkpointing can be done (I've done it), but you need extra code than what Remus currently provides.
Similar comments are applicable to the storage updating. This works absolutely superbly if you're using something like DRBD for the storage replication.
Remus is pretty cool technology, and it serves as a very solid foundation for taking things to the next level.
The folks at UBC have done a superb job here, and should be well congratulated.


Shared storage:
Re:How does it deal with replication latency? (Score:1)
by bcully (1676724) writes: on Thursday November 12 2009, @03:12PM (#30077050)
No, Remus doesn't require shared storage -- just two regular PCs with their own disks will do. It could be used on a SAN, but the disk replication system would have to be modified, to do something like journalling the writes from the primary so that they could be undone on failover up to the point of the last checkpoint.
 
You must log in or register to reply here.
Back
Top