Backup and restore

Borg does the deduplicated, encrypted storage. Borgmatic adds scheduling, retention, database dumps and monitoring on top, and it runs on the host rather than in a container — a containerised agent would need read access to every volume, and therefore every secret, in the deployment.

Verified 29 Jul 2026

Not exercised: the off-site target. The rehearsal used a repository on the same machine, so append-only enforcement over SSH is written rather than demonstrated.

This page is the general half: installing it, the repository, the keys, the schedule, and how a restore works. What to back up for a particular application lives on that application’s page — which database to dump, whether it needs quiescing, what is different when restoring it. One Borgmatic installation covers every stack on the host; each stack contributes its own few lines to the same configuration file.

Before you start

Needs: root on the Docker host, somewhere to put the repository, and a client for each database engine you run.

Borgmatic’s container: option resolves a container’s IP address and nothing more — the dump command itself runs on the host and connects over TCP. Without the client:

[Errno 2] No such file or directory: 'mariadb-dump'

sudo apt install mariadb-client        # or postgresql-client, mongodb-database-tools

Installation

1. Check what your distribution offers. Version 2.0.8 introduced the container: option this setup depends on, and packaged versions are frequently older — Debian 13 ships 1.9.14:

   apt-cache policy borgmatic

2. If it is below 2.0.8, install from PyPI:

   sudo apt install pipx
   sudo PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install borgmatic
   borgmatic --version

   Caution

   PIPX_BIN_DIR=/usr/local/bin matters. Borgmatic runs as root from a timer, and a default pipx install lands in a user’s ~/.local/bin — which root’s PATH does not include.

3. Create the passphrase and the repository. With repokey the encrypted key lives inside the repository and the passphrase unlocks it, so the passphrase is what reading a backup depends on. Exporting the key adds an independent recovery copy for the case where the repository’s own copy is lost or damaged — restoring from that export needs the export and its original passphrase:

   openssl rand -base64 32 | tr -d '\n' | sudo tee /root/.borg-passphrase > /dev/null
   sudo chmod 600 /root/.borg-passphrase
   
   sudo BORG_PASSCOMMAND='cat /root/.borg-passphrase' \
     borg init --encryption=repokey-blake2 \
     ssh://backup-user@backup.example.com:22/./borg/$(hostname)

4. Export the key and move it off this host:

   sudo BORG_PASSCOMMAND='cat /root/.borg-passphrase' \
     borg key export ssh://backup-user@backup.example.com:22/./borg/$(hostname) \
     /root/borg-key-$(hostname).txt

   Store it with the passphrase, in a password manager or on encrypted offline media. Separating the two protects against nothing and doubles the chance of losing one.

5. Write the configuration, then validate before trusting it:

   sudo mkdir -p /etc/borgmatic
   sudo cp config.yaml.example /etc/borgmatic/config.yaml
   sudo chmod 600 /etc/borgmatic/config.yaml
   sudo borgmatic config validate

6. Run it:

   sudo borgmatic create --verbosity 1 --stats

Add the applications

Everything above is set up once. From here on, each stack you want covered adds a few lines to the same /etc/borgmatic/config.yaml.

Stack	Its backup section
Nextcloud	Backup — MariaDB dump, maintenance mode, what is different on restore
Invoice Ninja	Backup — MySQL dump, the storage volume, and an encryption key that lives outside both
Vaultwarden	Backup before real use — signing keys matter as much as the database
Seafile Pro	Backup — three databases and two file volumes that only restore as a set

The rest of this page is what those sections assume you understand.

Databases need dumping, not copying

Copying a running database’s files captures a torn, mid-transaction state that only reveals itself at restore. Add one entry per database:

mariadb_databases:
    - name: myapp
      container: myapp-db
      username: myapp
      password: "{credential file /srv/blueprint/apps/myapp/.secrets/db_pwd.txt}"
      tls: false

Two things in there are worth understanding.

The password is read from the file the stack already uses. One copy of it exists on the host, nothing is exported into borgmatic’s environment, and rotating the secret needs no second edit.

tls: false is deliberate. The distribution’s client is often newer than the server a stack pins, and a newer client demands TLS the older server has no certificate for:

TLS/SSL error: SSL is required, but the server does not support it

This connection runs from the host to a container over a local bridge. Leaving it unencrypted is an accepted trust assumption about that bridge rather than a statement that TLS would achieve nothing — it would still cover a misconfiguration that put the database somewhere less local than you think. The trade here is a certificate lifecycle for a link that does not leave the machine. For a database that genuinely is remote, configure TLS on the server.

Quiescing, where the files and the database must agree

Some applications keep an index in the database and the data on disk. Capture the two at different moments under load and the restore shows files the index does not know about, or index entries pointing at nothing. Those applications need to be paused for the length of the capture — the application page says whether yours is one of them, and what the pause command is.

The shape is always the same:

commands:
    - before: action
      when: [create]
      run:
          - <put the application into maintenance mode>
    - after: action
      when: [create]
      run:
          - <take it back out>
    - after: error
      run:
          - <take it back out>

Caution

The error hook is not optional. Without it, a run that fails between the other two leaves the application offline until somebody notices. It carries no when:, so it covers every action.

Name the container exactly. docker compose resolves a project from the working directory and can pick the wrong one on a host running more than one deployment; docker exec <name> cannot.

Verify

Check what the archive contains, not that the command exited zero:

sudo borgmatic list
sudo borgmatic list --archive latest --find '*mariadb_databases*'

Then invert the question — search for something that must not be in it. If this host carries anything besides the deployment you meant to back up, that is the check that catches a configuration reaching too far:

sudo borgmatic list --archive latest --find '*the-other-deployment*' | grep -v '^local:'

Empty output is the evidence.

Prove it with a restore

A backup that has never been restored is a hypothesis. Rehearse quarterly, into a throwaway target — never into the live deployment.

1. Bring up a scratch database and create the target. Borgmatic will not create it for you:

   PW=$(openssl rand -hex 16)
   docker run -d --name restore-test -e MARIADB_ROOT_PASSWORD="$PW" \
     --tmpfs /var/lib/mysql:rw mariadb:11.8
   docker exec restore-test mariadb -uroot -p"$PW" -e 'CREATE DATABASE myapp;'

2. Restore into it. --container names the target explicitly, so the configured one cannot be hit by accident:

   sudo borgmatic restore --archive latest --database myapp \
     --container restore-test --username root --password "$PW"

3. Read the result — a table count, and rows in a table you can sanity-check:

   docker exec restore-test mariadb -uroot -p"$PW" myapp -e "
     SELECT COUNT(*) FROM information_schema.tables WHERE table_schema='myapp';
     SELECT COUNT(*) FROM <a_real_table>;"

   A count you can sanity-check is the evidence. A directory tree of correctly-named empty files is a failure mode that looks like success.

4. Clean up:

   docker rm -f restore-test

A restored instance may come back paused

If the backup quiesced the application, that state is part of what was captured — the flag was set at the moment the files were read. A restored configuration file will therefore differ from the live one by exactly that, and the restored instance refuses to serve until it is cleared.

That is the backup working, not a fault. The application page names the command; know it in advance, or the first restore looks like a broken one.

Schedule it, or it will not happen

A backup taken by hand is a demonstration. Check whether the package already ships units before writing your own:

systemctl list-unit-files | grep borgmatic
sudo systemctl enable --now borgmatic.timer
systemctl list-timers borgmatic.timer

A pipx installation ships none — take borgmatic.timer.example from the repository.

Caution

The timer is the point at which this stops being a rehearsal. Until it exists, there is exactly one archive and it is as old as the day you made it.

Going further

Retention

Derive it from how much data loss you can accept and how late a problem might be noticed. A week of dailies does not survive a three-week-old deletion nobody spotted.

keep_daily: 7
keep_weekly: 4
keep_monthly: 6

Append-only, and what it actually buys

Restricting the backup key on the target to borg serve --append-only is worth doing, and it is routinely overstated. From Borg’s own documentation, it affects the low-level structure only — delete and prune still appear to work. A compromised client can still make archives look gone.

What survives is the data, until borg compact runs. So: run retention from somewhere else, prefer storage-level immutability where the target offers it, and above all notice quickly — every recovery path above expires.

Notice when it stops running

The failure this catches is the one nobody sees: the timer stopped six weeks ago and nothing said so.

healthchecks:
    ping_url: https://healthchecks.example.com/ping/your-uuid
    states: [start, finish, fail]

Troubleshooting

[Errno 2] No such file or directory: 'mariadb-dump' — the client is missing on the host. container: does not remove that requirement.

SSL is required, but the server does not support it — client and server versions disagree. See tls: false above.

The application stayed in maintenance mode — a run failed and there is no after: error hook. Add one, then clear the flag by hand.

Archives exist but are older than the schedule — the timer has been failing silently. This is the most common finding in a rehearsal, and the reason the monitoring hook is not decoration.

Repository files

• backup/borgmatic/README.md — full setup, keys, ransomware notes
• backup/borgmatic/config.yaml.example
• backup/borgmatic/RESTORE.md — the rehearsal and the real thing
• backup/README.md — why the agent runs on the host

Official sources

• Borgmatic documentation
• Backing up databases
• Command hooks
• Borg documentation
• Borg on append-only mode