Finding a simple enough tutorial for doing this proved a little disappointing, obviously everyone utilising containers are clearly experts in their field and know everything there is to know about the various functions of LXC/LXD, export, snapshot, copy, etc .. So much so, nothing cohesive could be found.
Because of that I thought I would set myself the problem of creating a container backup server, somewhere other than where the containers are in active use and pausing them would be disadvantageous. The server will be added to the LXC container server as a remote server and images will be copied using LXC COPY (lxc-copy). So worst case scenario, the containers are in two places and can be copied back and forth with relative ease and speed.
All images created by Dave Wise © 2022 . all rights reserved
I have decided to discuss everything I run into along the way including the stuff I do to solve any particular problems along the way.
Creating a Container Backup server
For the backup server I will be starting from a blank slate installation of Ubuntu 22.04 (hopefully if the release date isn’t pushed back some more), no real reason beyond the fact that I fancied having a look at the new version. Yes, yes, very sad I know … you would have thought after 30+ years of continually installing/reinstalling operating systems for fun I would be well over it by now.
The Problem and Solution:
A single server operating LXC containers for different isolated functions with zero crossover between them, they are single instances and if they break there is no recovery for them. For the sake of this exercise we will call this server STUPID.
So we will add another container server and copy all of the containers to that server, we do not need to worry about connecting to any of the services running on those containers as this is purely a one-way backup process rather than having a fail-over. It helps to have a problem to solve, “Why?” I hear you ask. Well that way whoever is reading this nonsense it is easier for them to relate to it and thus, the solution.
For the BACKUP server I have started with 20.04.1 LTS as a clean install with LXC/SNAP 4 enabled. The only system software installed is the absolute minimum and sshd to be able to work the server itself.
Of course if you google (sorry! search using whichever search engine you feel comfortable with, bing, yahoo, whatever) for backing up containers you will get all of the command line specifics but none of the context around it, such as securing the backups, making sure there is very limited access to the backups, securing the underlying OS and so on. Again, this is why this exists.
So here we are, staring at our nice shiny new Ubuntu server, we told it to use the LXD 4 Snap during install, so we have LXC and LXD installed. Now we need to initialize it to our desired settings.
dave@shiny:~# lxd initWhere I select the default, I won’t bother explaining, if you want to know more about the options feel free to once again use your favourite search engine!
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (zfs, ceph, btrfs, dir, lvm) [default=zfs]: lvmI used lvm here, it doesn’t really matter for the limited use I am going to put it through, the default is zfs and that would be perfectly fine.
Create a new LVM pool? (yes/no) [default=yes]:
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]:
Size in GB of the new loop device (1GB minimum) [default=30GB]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:>Would you like the LXD server to be available over the network? (yes/no) [default=no]: yes>Address to bind LXD to (not including port) [default=all]: 80.168.84.221>Port to bind LXD to [default=8443]: XXXX>Trust password for new clients: xxxxxxxxxxxxxxx>Again: xxxxxxxxxxxxxxxRight, this is the important bit, we need the new backup server to be available to STUPID so we can finally back up those containers on there to somewhere else, at least that reduces the risk for the containers a little … we can extend this by creating exports from the backup server (that way the load isn’t service effecting) and then upload those images securely to the cloud somewhere for extra safe keeping … even download them, put them on a disk and put the disk into fireproof/emp protected safe. Again, that’s getting a little ahead of ourselves, lets get back to making sure the LXD server is available for backing up to.
We do want the server to be available over the network, so change the default from no, to YES. Then we supply the IP address for the server we wish to bind to and the port we wish to bind do. I change the port from default, just for the sake of it. I know the port will be firewall protected, but still, better to be safe than sorry, so a little protection through obscuration never hurts.
Make sure you use a strong password here, if in doubt, there are a multitude of strong password generators available online for you to use. If you’re feeling lazy, here’s a quick JavaScript one you can use to generate 16 character strong passwords and link here. There are no symbols in the passwords generated, they irritate me as you can’t just double click them to highlight them for C&P’ing.
<script> const keys = { upperCase: "ABCDEFGHIJKLMNOPQRSTUVWXYZ", lowerCase: "abcdefghijklmnopqrstuvwxyz", number: "0123456789" } const getKey = [ function upperCase() { return keys.upperCase[Math.floor(Math.random() * keys.upperCase.length)]; }, function lowerCase() { return keys.lowerCase[Math.floor(Math.random() * keys.lowerCase.length)]; }, function number() { return keys.number[Math.floor(Math.random() * keys.number.length)]; }]; function createPassword() { const passwordBox = document.getElementById("passwordBox"); const length = document.getElementById("length"); let password = ""; while (length.value > password.length) { let keyToAdd = getKey[Math.floor(Math.random() * getKey.length)]; password += keyToAdd(); } passwordBox.innerHTML = password; } function copyPassword() { const textarea = document.createElement('textarea'); const password = document.getElementById("passwordBox").innerText; if (!password) { return; } textarea.value = password; document.body.appendChild(textarea); textarea.select(); document.execCommand('copy'); textarea.remove(); alert('Password copied to clipboard'); }</script><div class="password"> <p id="passwordBox"></p> <span onclick="copyPassword()" class="dashicons dashicons-clipboard"></span></div><div> <label for="length">Length</label> <input type="number" id="length" min="6" max="32" value="16"></div><button onclick="createPassword()">Get Random Password</button><span class="copyright">© Dave Wise 2022 . use and abuse . feel free to credit</span></div>Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:And that’s pretty much all there is to setting up LXD to be a remote server, with all the cleverness around it all I suspect you thought there was more to it. Alas, it really is that simple. Although we do need to restrict access to it beyond the normal password protection, I for example, only want the server that it’s backing up to be able to see it. So all other traffic will be dropped.
I don’t think I need to set up a storage pool on the new server, again, I hear you calling me names … however, the backup server DOES NOT RUN THE CONTAINERS, it merely stores them as backups. Storage pools are only needed for running containers. I am prepared to be proven wrong, so lets see what happens by the time I get to the end of this exercise.
For the firewall I will just use IPTables … the reality is I could get away with just hosts.allow and hosts.deny here as nothing fancy is needed … in fact …
So that was even quicker, no need to explain any complicated rules to anyone … only that we have two files to edit, /etc/hosts.allow and /etc/hosts.deny. Obviously you don’t want to screw this up as getting around it will be pretty tricky. Test it before you log out of the server.
dave@shiny:~/# vi /etc/hosts.deny# Deny absolutely everythingALL: ALLBe selective, we have still got to do admin/housekeeping on the SERVER, no matter how much you ignore it!
dave@shiny:~/# vi /etc/hosts.allowsshd: a.b.c.d/255.255.255.xxx# Allow everything to/from the server we are backing upALL: a.b.c.dNow the new server is all protected and ready to accept images from elsewhere, lets return to the original server, STUPID to make the additions and changes we need on there.
We ran into a bit of a bump here, STUPID was running a very, very old version of LXD … as such there were some risks taken doing upgrades (after taking copies of everything of course, slowly and manually) to bring it up to a version that could be used with SHINY.
I would love to say it was painless, well to be fair, it was, but definitely arduous … this was something I wanted to avoid, but alas unless I wanted to change the model this whole exercise was based on (ie. tar and compress snapshots and then rsync them onto another server), probably infinitely easier to be honest, but we are where we are. So, without further ado I just created a complete duplicate of the server and disabled all upload services; I am not concerned about logs particularly as they are contained elsewhere. Now I have a replica server to work on to make sure the changes work.
For each LXC container found in /var/lib/lxc we had to create a tarball of the rootfs, create a metadata file, and then image import them into the new lxc system (after installing it via snapd obviously).
dave@stupid:/var/lib/lxc/container/rootfs/# tar zcf /tmp/og-containers/container.tar.gz *
dave@stupid:~/# date +%s
1660235845
dave@stupid:~/# vi /tmp/og-containers/metadata.yamlarchitecture: "amd64"
creation_date: 1660235845
properties:
architecture: "amd64"
description: "CONTAINER (20220101)"
os: "ubuntu"
release: "18.04"dave@stupid:~/# tar zcf ~/metadata.tar.gz metadata.yamlI only needed one metadata file as all of the containers were based on the same OS, so that saved some typing at least. There were only 11 containers, so I didn’t bother writing a script to do it either. Hopefully this wont be an experience I will need to repeat any time soon.
Now to import the images into the new install of LXD/LXC
dave@stupid:~/# lxc image import /tmp/og-containers/container.tar.gz ~/metadata.tar.gz --alias og-container-namedave@stupid:~/# lxc image list +--------------+--------------+--------+-------------+--------------+-----------+----------+------------------------------+| ALIAS | FINGERPRINT | PUBLIC | DESCRIPTION | ARCHITECTURE | TYPE | SIZE | UPLOAD DATE |+--------------+--------------+--------+-------------+--------------+-----------+----------+------------------------------+| og-container | F25fc29a412b | no | | x86_64 | CONTAINER | 402.87MB | Aug 01, 2022 at 0:01am (UTC) | +--------------+--------------+--------+-------------+--------------+-----------+----------+------------------------------+And “Tada“, now to rinse and repeat for all of the other containers and get them into the new image list. Remember, I only need one metadata file as all containers are the same OS, so this can now be replicated for all of the containers.
On a side note, there have been a couple of days between then and now because there is an issue with the old version of systemd for Debian 8 (jessie) which a number of these containers were, so as well as having to move, they have also had to be completely upgraded. So took the opportunity to make them all Bullseye containers. A real ball-ache to be fair, all I could hear whilst doing the work was “If it’s not broken, don’t fix it” … Oh how nice it would have been not to have to. But we now have a fully autonomous LXD 4 container server running 24 containers that are all nice and shiny and new. Now, when time permits I will get back to creating that actual thing I started this for, a backup system on a different server to keep the images and snapshots safe from terminal illness.
Right, now back to it … I have added a password to the LXD backup server
lxc config set core.trust_password xxxxxxxxxxxxxxxxxxxxxxxxxxxxNow, with that done, on the nice shiny new LXD server with all of the containers that need to have backups, I have to add the remove server.
dave@stupid:~# lxc remote add backup xxx.xxx.xxx.xxx:xxxxCertificate fingerprint: e79c64e9dac5ab8a41d1d14af3a154d7bfb44d8a14283907a64a49b2158d9a14ok (y/n)? yAdmin password for backup:Client certificate now trusted by server: backupOf course, you will need the trust_password you just assigned to your backup server.
dave@stupid:~# lxc remote list+-----------------+------------------------------------------+---------------+-------------+--------+--------+| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC |+-----------------+------------------------------------------+---------------+-------------+--------+--------+| images | https://stupid.domain.com | simplestreams | none | YES | NO |+-----------------+------------------------------------------+---------------+-------------+--------+--------+| backup | https://shiny.domain.com:7890 | lxd | tls | NO | NO |+-----------------+------------------------------------------+---------------+-------------+--------+--------+| local (current) | unix:// | lxd | file access | NO | YES |+-----------------+------------------------------------------+---------------+-------------+--------+--------+| ubuntu | https://cloud-images.ubuntu.com/releases | simplestreams | none | YES | YES |+-----------------+------------------------------------------+---------------+-------------+--------+--------+| ubuntu-daily | https://cloud-images.ubuntu.com/daily | simplestreams | none | YES | YES |+-----------------+------------------------------------------+---------------+-------------+--------+--------+I wish the next bit was just simple, bit alas, copying live (running) containers can cause other complications such as freezing or even data corruption, it’s not entirely clear why this happens and it absolutely isn’t consistent, it still happens and as I won’t be doing any of this stuff manually, it’s time to do things slightly differently.
So I need a script that will give me all of the running instances, it’s quite straight forward really, the hardest part remembering the regex to do the trimming … it doesn’t matter how many times I do it, I still have to look up the order of things. To be fair, I create shell scripts maybe once every 3 or 4 months, it’s hardly as if I am living the shell scripting dream.
Given it’s pretty straight forward there is no need to employ anything complicated to get the job done, LXC allows for json output as well as the default/normal stdout ascii. To take advantage of json without employing a more comprehensive language and utilising just a bash script, we will need the additional package “jq” to be installed. Ubuntu and Debian can install it via APT, otherwise it can be found here (https://stedolan.github.io/jq/).
# Old Skoollist = `lxc list | grep -i container | awk 'BEGIN {FS ="|"}; {gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}'`jq isn’t a standard bash tool, so you WILL have to install jq if it isn’t installed
# New Skoollist = `lxc list --format-json | jq -r '.[].name'`Now we need to check if the backup exists, we only want to keep two snap backups, I am not interested in anything overly complicated, we can run monthly exports off of the second lxd server to drump tarballs to a network storage device if that concerned. But right now, I don’t care .. I just need to make sure if the backup exists, its renamed to backup.bak .. if backup.bak exists, it is deleted. Then the backup can be done.
Shell Script Pseudo Code
1. Get list of active containers2. with each container 2.1 create snapshot to backup 2.2 backup snapshot 2.2.1 if destination backup.bak exists delete backup.bak 2.2.2 if destination backup exists move to backup.bak 2.2.3 copy snapshot to destination backup 2.3 delete snapshotI will put the actual shell script here for everyone’s abuse … feel free to use/abuse as required. It would be nice to know if you used it or any part there of. Or even if it just helped you structured your thinking to solve your problems your way.
Shell Script
#!/bin/bash# +--------------------------------------------# | Create remote server backups for LXD Server# +--------------# -- Author: Dave Wise 2022REMOTE="iprism"if [ "$1" == "-h" ] || [ "$1" == "--help" ]then printf "bakbak.sh © Dave Wise 2022 . all rights reserved\n" printf "________________________________________________\n" printf "Usage: bakbak.sh\n\n" printf "Description: A tool for backup up live containers to a remote LXD Server\n\n" exit 0fiif ! jq_loc="$(type -p "jq")" || [[ -z "$jq_loc" ]];then activecontainers=`lxc list | grep -i container | awk 'BEGIN {FS ="|"}; {gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}'`else activecontainers=`lxc list --format=json | jq -r '.[].name'`fiif ((${#activecontainers[@]}));thentodaydate=`date`message="BAKBAK - Container Backup Service\n---------------------------------\n\nContainers processed ($todaydate):\n"for acontainer in $activecontainersdo# Get backup names to checktmessage="${message}$acontainer\n"message=$tmessageif ! jq_loc="$(type -p "jq")" || [[ -z "$jq_loc" ]];then matchingcontainers=`lxc list $REMOTE:$acontainer | grep -i container | awk 'BEGIN {FS ="|"}; {gsub(/^[ \t]+|[ \t]+$/, "", $2); print $2}'`else matchingcontainers=`lxc list --format=json $REMOTE:$acontainer | jq -r '.[].name'`fibackup=""bakbak=""backbackupname="$acontainer-bak"for container in $matchingcontainersdo if [ "$container" == "$acontainer" ] then backup="found" fi if [ "$container" == "$backbackupname" ] then bakbak="found" fidoneif [ "$bakbak" == "found" ]then # Deleting container-bak `lxc delete $REMOTE:$backbackupname`fiif [ "$backup" == "found" ]then # Moving original backup container to second backup container `lxc move iprism:$acontainer $REMOTE:$backbackupname`fi# Now it's time to do the actual backup# Create Snapshot`lxc snapshot $acontainer snap0`# Back up Snapshot to remote server`lxc copy $acontainer/snap0 $REMOTE:$acontainer --quiet`# Delete temportary Snapshot`lxc delete $acontainer/snap0`# In reality this is all one command line but I broke it down # so you can see what each bit is for ... probably pointless tbh# `lxc snapshot $acontainer snap0; lxc copy $acontainer/snap0 $REMOTE:$acontainer --quiet; lxc delete $acontainer/snap0`doneprintf "${message}"fiFinal thing to do is make it run once a week, I don’t know about you but I still think of Sunday’s as the beginning of the week, so I will run this script once now and then set it to run at 1am every Sunday.
0 1 * * 0 /usr/local/bin/bakbak.shYes, I copied the script to /usr/local/bin … I didn’t think I would need to say that, but someone just pointed out I didn’t say it *sighs*
Anyway, that’s enough from me, the script is working and I have backup’s on a different server that update weekly which is more than enough for my requirements. But, the principles are sound to change the frequency, if for example you wanted to run the script daily (bare in mind that large containers will take a while to process and will likely cause significant load whilst processing the snapshots). Maybe extend it further and add a third server to receive weekly backups from the daily backups. You do you, hope this helps someone at least a little even if it is just to help you decide what not to do.
Happy Container’ing #peaceout
Leave a Reply