Recently I had to migrate services from a running Debian server to another one, with minimal downtime of services.

I usually do this to P2V or V2V Linux systems, as this allows me to resize the new virtual machine to meet the services requirements (adjust disk size, inodes, partitionning, etc.).

I have done this several times in the past on systems with Grub 1 but this is the first time with Grub 2, so I thought I’d share my process.

Step 1: prerequisites, first copy and validation

During this step, you prepare the new server, copy the data from the current server and reboot the new server to test everything starts correctly. Amongst other things, you check that the bootloader is installed and configured properly, fstab is OK, network interfaces are OK and services start.

At this point, data integrity on the new server is not important (unless it prevents services from starting).

The actual check is performed offline (or in different network segments) to have no impact on the production workloads.

  • On the new server, boot a System Rescue CD.

  • Create the new partition layout.

  • Create the new filesystem(s). Make sure to keep the same inode size, especially with old systems.

  • Mount the new root filesystem to /mnt/gentoo .

  • If needed, create any subdirectory required for other filesystems and mount those filesystems. Example:

    mkdir /mnt/gentoo/boot
mount /dev/sda1 /mnt/gentoo/boot

  • Assign an IP address to the new server.

  • On the old server, make sure to allow connection from the IP address you just set on the new server (routes, iptables, etc.).

  • On the new server, copy the files from the old server. Make sure to exclude any temporary file (/proc, /sys, /tmp, etc.) and all network mounts (nfs, samba, iSCSI, etc.). Example:

    rsync -avPH --numeric-ids --exclude=/proc/ --exclude=/sys/ --exclude=/mnt/sharedfolder 1.2.3.4:/ /mnt/gentoo

  • Mount the dynamic filesystems:

    mount -o bind /proc /mnt/gentoo/proc mount -o bind /sys /mnt/gentoo/sys mount -o bind /dev /mnt/gentoo/dev

  • Chroot into this installation: chroot /mnt/gentoo /bin/bash

  • Update initrd, grub config and install grub on MBR: update-initramfs -t -u update-grub grub-install /dev/sda

  • Edit fstab to match the new partition layout and new UUID’s.

  • Remove or edit the associations between a NIC names and their MAC address:

    rm /etc/udev/rules.d/70-persistent-net.rules

  • Exit from the chroot.

  • Unmount all filesystems:

    umount /mnt/gentoo/proc 
umount /mnt/gentoo/sys 
umount /mnt/gentoo/dev 
umount /mnt/gentoo

  • Sync the filesystems to drives: sync

  • Disconnect the NICs of the new server.

  • Reboot the new server.

It should display the grub menu, boot the approriate entry, start the services and end up with a login prompt.

Be patient with the services because some of them need DNS servers, you need to wait for those requests to timeout.

Step 2: synchronization and migration

During this step, you synchronize the data between the old server and the new server, to guarantee data integrity.

The synchronization is performed online with all services stopped on the old server. That means that if you have a LAMP server, you will need to stop Apache web server, MySQL server and probably other services like FTP, etc.

  • Again, on the new server, boot a System Rescue CD.
  • Mount the new root filesystem to /mnt/gentoo .
  • Mount all other required filesystems.
  • Assign the same IP address from step 1 to the new server.
  • On the old server, stop all services that can change the data you care about. Example: apache2, rsyncd, syslog, ftpd, etc. .
  • On the new server, synchronize the files from the old server.
  • Again, exclude all temporary files and this time, all the files we modified in step 1. Do a dry-run first to check what will be done. Example: 
    rsync -avPH --numeric-ids --delete --dry-run --exclude=/proc/ --exclude=/sys/ --exclude=/mnt/sharedfolder --exclude=/boot/ --exclude=/etc/fstab --exclude=/etc/udev.d/ 1.2.3.4:/ /mnt/gentoo

If you see that it rsync would retransfer the whole filesystem, then you made a mistake somewhere. Fix it and re-try the dry-run.

  • Do the actual sync (remove the --dry-run parameter).

Once it is done, disconnect the NICs of the old server: unplug the cables, or shutdown the switch ports, or disconnect the virtual NIC, anything that makes it unreachable but does not stop the server. In case the new server does not work as expected, you can just shut it down and reconnect the old server.

You can now reboot the new server with its NICs connected into the correct network segments, it will take the place of the old server.

Be aware that you might need to clear some ARP tables in switches and routers since the IP address(es) didn’t change but the MAC address(es) did.