QEMU Migration Phases

QEMU supports only migration protocols 2 and 3 (1 was lacking too many steps). Repeating the protocol sequences from libvirt.c:

Migration protocol v2 API Sequence

Src: DumpXML
  • Generate XML to pass to dst

Dst: Prepare
  • Get ready to accept incoming VM

  • Generate optional cookie to pass to src

Src: Perform
  • Start migration and wait for send completion

  • Kill off VM if successful, resume if failed

Dst: Finish
  • Wait for recv completion and check status

  • Kill off VM if unsuccessful

Migration protocol v3 API Sequence

Src: Begin
  • Generate XML to pass to dst

  • Generate optional cookie to pass to dst

Dst: Prepare
  • Get ready to accept incoming VM

  • Generate optional cookie to pass to src

Src: Perform
  • Start migration and wait for send completion

  • Generate optional cookie to pass to dst

Dst: Finish
  • Wait for recv completion and check status

  • Kill off VM if failed, resume if success

  • Generate optional cookie to pass to src

Src: Confirm
  • Kill off VM if success, resume if failed

QEMU Migration Locking Rules

Migration is a complicated beast which may span across several APIs on both source and destination side and we need to keep the domain we are migrating in a consistent state during the whole process.

To avoid anyone from changing the domain in the middle of migration we need to keep MIGRATION_OUT job active during migration from Begin to Confirm on the source side and MIGRATION_IN job has to be active from Prepare to Finish on the destination side.

For this purpose we introduce several helper methods to deal with locking primitives (described in qemu-threads) in the right way:

  • qemuMigrationJobStart

  • qemuMigrationJobContinue

  • qemuMigrationJobStartPhase

  • qemuMigrationJobSetPhase

  • qemuMigrationJobFinish

The sequence of calling qemuMigrationJob* helper methods is as follows:

  • The first API of a migration protocol (Prepare or Perform/Begin depending on migration type and version) has to start migration job and keep it active:

    qemuMigrationJobStart(driver, vm, VIR_JOB_MIGRATION_{IN,OUT});
    qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_*);
    ...do work...
    qemuMigrationJobContinue(vm);
  • All consequent phases except for the last one have to keep the job active:

    if (!qemuMigrationJobIsActive(vm, VIR_JOB_MIGRATION_{IN,OUT}))
        return;
    qemuMigrationJobStartPhase(driver, vm, QEMU_MIGRATION_PHASE_*);
    ...do work...
    qemuMigrationJobContinue(vm);
  • The last migration phase finally finishes the migration job:

    if (!qemuMigrationJobIsActive(vm, VIR_JOB_MIGRATION_{IN,OUT}))
        return;
    qemuMigrationJobStartPhase(driver, vm, QEMU_MIGRATION_PHASE_*);
    ...do work...
    qemuMigrationJobFinish(vm);

While migration job is running (i.e., after qemuMigrationJobStart* but before qemuMigrationJob{Continue,Finish}), migration phase can be advanced using:

qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_*);