If you're interested, there's a whole lot of really great information on the libvirt migration page about how libvirt handles migrations. After a brief read of this page, a few more searches around the net and a brief study of the man pages I came up with the following command:
virsh migrate --live \ --persistent \ --undefinesource \ --copy-storage-all \ --verbose \ --desturi [destination] [vm_name]
The key to this command is the --copy-storage-all command. This will copy the contents of the source image file to the destination. Before we can run this command, though, we need to create a representation of the image on the destination server, preferably in the same location. Since I use sparsely allocated qcow2 files, I will be using qemu-img with the "qcow2" format flag. If you are using something different, you should create the destination image the same way as you created the original source image. This file should be the same size as the source image:
qemu-img create -f qcow2 /vm_storage/[my_vm].img
Once you've created the file on the destination server, you are almost ready to kick off migration. First we need to make sure you can connect to the remote host. You can test this with virsh:
virsh -c qemu+ssh://[remote_host]/system -l
If all goes well, you should see a list of the VMs on the remote host. You may, however, receive an error or an empty list (assuming it shouldn't be empty). If this happens, you probably need to add the following pkla file which allows the group your user is in to connect to the remote libvirt socket. I've provided the policykit file contents below:
[Remote libvirt SSH access] Identity=unix-group:MY_GROUP Action=org.libvirt.unix.manage ResultAny=yes ResultInactive=yes ResultActive=yes
Change "MY_GROUP" to whatever group you want to use. Wheel is a good choice if your user is a member. On a Redhat/CentOS system, save this file as /etc/polkit-1/localauthority/50-local.d/50-libvirt-remote-access.pkla and try again.
Once you're able to successfully run the test above, you can try to migrate your VM. I've provided an example below. Run this as the user who is a member of the group we added to the pkla file above. You should probably also run this inside of a tmux or screen session as it will take a while to complete:
virsh migrate --live --persistent --undefinesource --copy-storage-all \ --verbose --desturi qemu+ssh://[remote_host]/system [vm_name]
If all goes well, you should receive a progress indicator that will progress slowly. In my tests, a 100GB VM with 4GB of RAM under minimal load took about 15 minute to transfer over a 1GbE network.
This script should be run from the source VM host. It will handle creating the img file on the destination host and do some sanity checking for you:
# Change libvirt default URI to allow enforecement by the pkla file
export LIBVIRT_DEFAULT_URI=qemu:///system
vm="$1"
dest_host="$2"
storage="/directory/of/img/files"
# Are you root?
if [[ "$UID" == "0" ]]
then
echo "You can not run migrate as root"
exit 1
fi
# Exit if not running in screen or tmux
if [[ -z "$TMUX" ]] || [[ "$TERM" != "screen" ]]
then
echo "You must run migrations in either screen or tmux. Aborting."
exit
fi
# Confirmation
read -p "You are about to move ${vm} to ${dest_host}. Are you sure? y/N " -n 1 -r
echo
# Check if VM exists
if ! virsh list --all | awk '{print $2}' | grep -q "$vm"
then
echo "$vm does not exist on this server. Aborting."
exit 1
fi
# Check if remote destination exists
if ! host "$dest_host" > /dev/null
then
echo "Unable to reach $dest_host. Aborting."
exit 1
fi
# Check that we are forwarding ssh agent
if [[ -z "${SSH_AUTH_SOCK}" ]]
then
echo "Please exit and re-connect using ssh's -A flag"
exit 1
fi
# Get VM size (this is ugly)
# TODO: Make this handle multiple disks. Right now it will only work if the
# img file is named the same as the host and there is only one file
disk_size=$(virsh vol-info --pool default "${vm}".img | awk '/Capacity/ {print $2}' | awk -F'.' '{print $1}')
# Create remote disk
if ! ssh "${dest_host}" "[[ ! -f ${storage}/${vm}.img ]] && sudo qemu-img create -f qcow2 ${storage}/${vm}.img ${disk_size}G"
then
echo "Unable to create image file ${storage}/${vm}.img on ${dest_host}. Confirm that this does not already exist and try again."
exit 1
fi
# Migrate VM to new host
virsh migrate --live --persistent --undefinesource --copy-storage-all --verbose --desturi qemu+ssh://"${dest_host}"/system "${vm}"
if [[ $? -eq 0 ]]
then
echo
echo "Migration complete. A copy of the VM image file still resides on the old host inside of ${storage}."
read -n 1 -p "Do you want to delete ${storage}/${vm}.img now? y/N " -r del
if [ "$del" == "y" ]
then
rm -f "${storage}"/"${vm}".img
fi
else
echo "The migration does not appear to have completed cleanly. Cleaning up remote host and exiting."
ssh "${dest_host}" "[[ ! -f ${storage}/${vm}.img ]] && sudo rm -f ${storage}/${vm}.img"
exit 1
fi
exit 0
Invoke the script as follows:
script [vm_name] [destination_server]
Good luck!
