From 9f8d11214cd19709e0bf4ea3dc178634053b800b Mon Sep 17 00:00:00 2001 From: Julian Lackner Date: Sat, 9 May 2026 14:21:13 +0000 Subject: [PATCH] Add README --- README.MD | 211 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 README.MD diff --git a/README.MD b/README.MD new file mode 100644 index 0000000..23b9e74 --- /dev/null +++ b/README.MD @@ -0,0 +1,211 @@ +# Docker Services Recovery + +This document describes how to restore the Docker services (`netbird`, `pocket-id`, `caddy`, `vaultwarden`) from an Internxt backup using `docker-recover.sh`. + +## Overview + +The recovery script does the following: + +1. Lists available backups on Internxt and prompts you to choose one (or accepts one as an argument) +2. Downloads the selected archive from Internxt via rclone +3. Extracts it to a staging directory +4. Stops any existing containers for the four services +5. Moves existing `/opt/` directories aside as `.pre-recovery.` +6. Restores each service's directory from the archive into `/opt/` +7. For Netbird specifically: creates the containers/volumes without starting, copies the management database into the named volume, then starts everything +8. Brings all services back up with `docker compose up -d` + +## Prerequisites + +Before running recovery, the target machine needs: + +- **Docker** with the `compose` plugin (v2) +- **rclone** installed and configured with a remote named `internxt` pointing to the same bucket/path that holds the backups +- **Root access** (the script must be run as root or via `sudo`) +- The `docker-recover.sh` script installed and executable, e.g. at `/usr/local/bin/docker-recover.sh` + +### Setting up rclone on a new machine + +If you're recovering to a fresh VPS, the easiest way to get rclone configured is to copy the config file from the old machine: + +```bash +# On the old machine, find the config file: +rclone config file + +# Copy that file to the new machine at: +# /root/.config/rclone/rclone.conf +``` + +Alternatively, run `rclone config` on the new machine and reconfigure the Internxt remote from scratch. + +Verify it works: + +```bash +rclone listremotes +# should show: internxt: + +rclone lsf internxt:vps-backups/ +# should list the available backup archives +``` + +## Usage + +```bash +sudo docker-recover.sh # interactive: lists backups, prompts for choice +sudo docker-recover.sh latest # restores the most recent backup automatically +sudo docker-recover.sh strato-docker_2026-05-07_03-00-00.tar.gz # restore a specific archive +sudo docker-recover.sh --dry-run # show what would happen, change nothing +sudo docker-recover.sh --dry-run latest # dry-run on the latest backup +sudo docker-recover.sh -n latest # short form of --dry-run +sudo docker-recover.sh --help # show usage +``` + +## Recommended workflow + +### 1. Verify the backup with a dry-run + +Before doing anything destructive, always run a dry-run first to see exactly what the script intends to do: + +```bash +sudo docker-recover.sh --dry-run latest +``` + +Every operation is logged as `WOULD RUN: ` and prefixed with `[DRY-RUN]`. No files are changed, no containers are touched, nothing is downloaded. + +For an even more thorough dry-run, do this two-step: + +1. Start a real recovery and abort at the confirmation prompt — this downloads the archive to `/tmp/docker-recovery/` but doesn't change anything else +2. Run `--dry-run` afterwards — since the archive is now cached locally, the script will list the actual top-level entries from the archive + +### 2. (If migrating to a new VPS) Update DNS first + +If you're recovering to a different machine with a different IP address, update your DNS A/AAAA records **before** running recovery, and ideally wait for propagation. Otherwise: + +- Caddy may try to renew TLS certificates on startup against the old IP +- Netbird clients won't be able to reach the new server until DNS resolves correctly + +### 3. Run the recovery + +```bash +sudo docker-recover.sh latest +``` + +The script will: + +- List available backups +- Ask for confirmation before doing anything destructive +- Stream progress to the terminal and log everything to `/var/log/docker-recover.log` + +### 4. Verify the services came up + +```bash +docker ps +``` + +All four services should be running. Check individual logs if anything looks off: + +```bash +cd /opt/netbird && docker compose logs --tail=50 +cd /opt/pocket-id && docker compose logs --tail=50 +cd /opt/caddy && docker compose logs --tail=50 +cd /opt/vaultwarden && docker compose logs --tail=50 +``` + +### 5. Verify each service's functionality + +- **Caddy**: `curl -I https://yourdomain.example.com` should return a valid response with the existing TLS certificate +- **Pocket-ID**: log in via the web UI; check that previously registered passkeys still work +- **Netbird**: check that existing peers reconnect and the dashboard shows your network +- **Vaultwarden**: log in with an existing account; verify your vault items are present + +### 6. Clean up the pre-recovery directories + +The script preserves the previous state of `/opt/` as `/opt/.pre-recovery.`. Once you've verified everything works correctly: + +```bash +sudo rm -rf /opt/*.pre-recovery.* +``` + +If something looks wrong, you can roll back by stopping the new containers, removing `/opt/`, and renaming the `.pre-recovery.` directory back. + +## What gets restored + +For each of `netbird`, `pocket-id`, `caddy`, `vaultwarden`: + +- The full contents of `/opt//` — `docker-compose.yml`, `.env` files, configuration files, bind-mounted data directories + +For Netbird specifically, the management database is restored separately: + +- The archive contains a `netbird/_netbird-db/` subdirectory holding the database extracted from `/var/lib/netbird/` inside the management container +- The script creates the Netbird containers (and the named Docker volume that backs `/var/lib/netbird/`) without starting them, copies the database in, then starts everything + +This is necessary because Netbird stores its database in a Docker named volume rather than in the bind-mounted `/opt/netbird/` directory, so a plain tarball of `/opt/netbird/` is not enough on its own. + +## Caddy TLS certificates + +Caddy's TLS certificates and ACME account info live in `/opt/caddy/data/` (assuming a bind mount). Restoring this directory means: + +- HTTPS works immediately on the new machine without re-issuing certificates +- No risk of hitting Let's Encrypt rate limits during migration + +If you ever change domain names during a migration, you may need to remove old certificates from `/opt/caddy/data/caddy/certificates/` and let Caddy re-issue them for the new names. + +## Troubleshooting + +### `rclone remote 'internxt' not configured` + +Run `rclone listremotes` as the same user that's running the script (root if using `sudo`). If `internxt:` doesn't appear, the rclone config file isn't where root expects it. Copy it to `/root/.config/rclone/rclone.conf` or set `RCLONE_CONFIG=/path/to/rclone.conf` in the environment. + +### `No backups found matching strato-docker_*.tar.gz` + +Verify the remote contents directly: + +```bash +sudo rclone lsf internxt:vps-backups/ +``` + +If the archives use a different prefix or are in a different folder, adjust `ARCHIVE_PREFIX` and `RCLONE_PATH` at the top of the script. + +### Netbird database not restored + +If the script logs `could not detect Netbird management container; skipping DB restore`, this means it couldn't find a service named `netbird-server` (newer setup) or `management` (older setup) in `/opt/netbird/docker-compose.yml`. Check the compose file and adjust the detection logic in the script if your service has a different name. + +### Containers fail to start after recovery + +Check the logs (`docker compose logs`) for the affected service. Common issues: + +- **Port conflicts**: another service on the host is using the same port +- **Missing environment variables**: the `.env` file wasn't included in the backup or was misplaced +- **Volume permission issues**: file ownership in the restored directories doesn't match what the container expects (especially for non-root containers) + +### Pre-recovery rollback + +If recovery succeeds but a service is broken, you can revert to the pre-recovery state: + +```bash +cd /opt/ && docker compose down +sudo rm -rf /opt/ +sudo mv /opt/.pre-recovery. /opt/ +cd /opt/ && docker compose up -d +``` + +For Netbird specifically, you may also need to remove and recreate the Docker named volume to clear out the restored database: + +```bash +cd /opt/netbird +docker compose down -v # the -v removes the named volumes too +# then bring up the rolled-back state +docker compose up -d +``` + +## Logs + +All recovery operations are logged to `/var/log/docker-recover.log` (in addition to being printed to the terminal). The log persists across runs, so you can review past recoveries. + +## Related files + +- `/usr/local/bin/docker-backup.sh` — the backup script that produces these archives, run nightly via cron +- `/usr/local/bin/docker-recover.sh` — this recovery script +- `/var/log/docker-backup.log` — backup logs +- `/var/log/docker-recover.log` — recovery logs +- `/tmp/docker-recovery/` — staging directory used during recovery (downloaded archives are cached here) \ No newline at end of file