Backups¶
faden backup saves everything the server needs to come back: every database, the .env with its keys, and the
certificate material the machines trust. faden restore puts it back; restoring is still being tested (see Restore). A nightly backup is set up
when you install.
What a backup holds¶
| Part | In the backup | Why it matters |
|---|---|---|
| Every database on the server's Postgres | Yes | Users, settings, providers, machines, clusters, deployments, chats, the request log |
| Langfuse's users, projects, API keys and prompts | Yes, from release 0.7.1 on | Before, Langfuse kept them in the server's own database postgres, which a backup leaves out. faden upgrade to 0.7.1 moves them into their own database; until then, faden backup says that it leaves them out. Langfuse's traces are in ClickHouse (see Logs, metrics, traces below) |
.env |
Yes, unless --db-only |
It holds the key the chats are encrypted with and the keys that seal stored secrets (the Hugging Face token, secret settings, the MCP servers' credentials). A database restored without its .env has chats and secrets nobody can read |
Certificate material (tls/, tls-ca/) |
Yes, unless --db-only |
The served certificate and the server's own certificate authority with its private key. With them, a rebuilt server is one the machines still trust |
| The release, migration versions and checksums | Yes | The release the install ran and the faden version that took the backup, which faden restore shows. It checks the database dumps, the volume archives and the certificate files before it changes anything |
| Files uploaded to the knowledge base | With --include-volumes |
Without it, only the text read from them is in the backup, not the files themselves. With FADEN_DATA_DIR set, the backup reads (and a restore writes) its knowledge folder |
| Files attached to chats | With --include-volumes |
Without it, only the text read from them is in the database, not the files themselves. With FADEN_DATA_DIR set, the backup reads (and a restore writes) its chat-files folder |
| Models | No | They can be downloaded again. Keep a copy of models you imported rather than downloaded |
| Logs, metrics, traces | With --include-volumes, except metrics |
They are kept on the server for viewing; a backup without them loses nothing the server needs |
| Runtime images | No | faden runtime-images fetches them again |
| The GPU machines | Nothing to back up | They hold no state the server needs; their clusters keep running if the server is lost |
Warning
A backup holds private keys and every secret of the install. Encrypt them (see
Encrypted backups), keep them as private as .env, and keep copies off the server: a
backup on the same disk does not survive losing the disk.
On the server, a backup is readable by the user who runs faden only, like .env; faden deploy and
faden upgrade do the same for backups made before.
Back up now¶
It asks for confirmation (-y skips it), then writes ~/fadenstack/backups/<time>/ and lists what it saved.
| Option | What it does |
|---|---|
--output-dir PATH |
Write the backup elsewhere, such as a mounted backup disk. A relative path is taken inside the install directory |
--retain N |
Keep only the N most recent backups of this kind in that directory (default 5). Older ones of the same kind are deleted |
--db-only |
Databases only, without .env and certificate material. Keep a copy of .env apart, as private as the backups |
--include-volumes |
Also the files uploaded to the knowledge base, the files attached to chats, the logs, the traces and Grafana's own data. Not the models |
-y |
Do not ask |
How --retain counts
Each backup records what made it: the nightly timer, you with faden backup, or faden upgrade. --retain
counts and deletes only backups of its own kind, so a manual faden backup --retain 3 leaves the nightly
backups alone, and an upgrade does not push them out. A manual backup keeps 5 unless you pass another number,
the nightly one keeps 7 (see below), and every faden upgrade keeps 5 of its own. Backups made before this
release have no kind and count as manual; a nightly timer set up before it keeps counting as manual until you
run faden backup schedule again.
The nightly backup¶
faden deploy offers a nightly backup at 03:00 that keeps the last 7, and turns it on with -y. To change or
stop it:
Its output goes to ~/fadenstack/backups/backup.log. Read that log now and then: a backup that fails at night
says so only there.
How the nightly backup is started
It runs as a systemd user timer (systemctl --user list-timers), or from your crontab where there is no
systemd.
The nightly backup writes into ~/fadenstack/backups/, on the same disk as the data. Copy the backups elsewhere
with your own tooling, or schedule faden backup -y --output-dir <your backup disk> with your own scheduler
instead.
Every faden upgrade also takes a backup before it changes anything.
Encrypted backups¶
Once the install has a backup key, every backup is encrypted. Each file in it is encrypted to the key's public half, which stays on the server, so the nightly backup runs on its own. Only the recovery key opens a backup: without it nobody reads one, whether they hold the backup disk or administer this server. Fadenstack keeps no copy of it.
faden deploy makes the key. faden upgrade, run by someone at the terminal, offers to make it once;
faden upgrade -y leaves the backups as they were. To make it yourself:
- The recovery key is shown once:
FADEN-BACKUP-KEY1and twelve groups of characters. - Keep a copy off the server, in a password manager or on paper kept somewhere safe, and confirm that you did.
- The screen is cleared, and you are asked for two groups of the key, at random positions, from your copy. Three wrong answers show the key again from the start.
With --write-to PATH the key goes into a file instead, such as one on a USB stick (never inside the install
folder), and is read back to check it.
After an unattended install (faden deploy -y)
Nobody is there to see the key, so it is left in ~/fadenstack/backup-recovery-key.txt, readable by the user
who runs faden only. While it is there, the key is on the same server as the backups: faden backup,
faden upgrade and faden backup key status remind you until you copy it off the server and run
faden backup key confirm. That asks for two groups from your copy, then deletes the file.
| Command | What it does |
|---|---|
faden backup key status |
Whether backups are encrypted, to which key, and what is still to do |
faden backup key check |
Checks that your copy of the recovery key is still this install's; nothing is stored. Do it once a year and with every restore test |
faden backup key create --replace |
A new key. The backups made so far still need the old recovery key |
Only a small backup.json stays readable in an encrypted backup: when it was made, what made it, and which key
opens it. Backups made before the install had a key stay as they are and are restored as before; delete them once
encrypted ones exist.
Warning
Without the recovery key, the backups made with it cannot be restored, by anyone.
Restore¶
Under testing
Restoring is being tested. Try a restore on a copy of the server before you rely on one.
It asks for confirmation (-y skips it). An encrypted backup asks for the recovery key first.
Each database the backup holds becomes exactly the backup's. What came after the backup goes, including tables a newer release added. The databases are restored beside the live ones first, while the install keeps running; only when every copy is complete are the services stopped and the copies put in their place. If anything fails before that, nothing has changed. If the services do not become healthy afterwards, everything goes back as it was before the restore.
What the restore does, in order
- opens an encrypted backup with the recovery key, into a folder inside it that only you can read and that is deleted when the restore ends;
- shows the backup's date, the release it holds and the
fadenversion that took it, and what it holds; - checks the database dumps, the volume archives and the certificate files against the checksums in the backup, and stops if one does not match;
- asks for confirmation (
-yskips it); - checks there is room for a copy of the databases and of the volumes it replaces (see below), and stops if there is not;
- restores every database of the backup into a new one beside the live one (
<name>__restore), while the install keeps running. pg_restore's whole output goes to~/fadenstack/logs/restore-<time>.log. An error stops the restore here: the copies are dropped, the live databases and the services were not touched; - puts up the maintenance page, stops every service except the front door (nginx) and Postgres, and swaps the
names in one step: each live database becomes
<name>__old, each copy takes its name; - keeps a copy of the
.env, the volumes the backup holds and the certificate files, then puts in the backup's (unless told otherwise); - starts everything. Once every service is healthy (within three minutes), it drops the
__olddatabases and the kept copies, takes the maintenance page down and says Restore complete; - if they do not become healthy, it stops them, puts the databases before the restore back in their place,
and the
.env, the volumes and the certificate files too, starts that, and ends with an error that names the services that were not healthy and anything that could not be put back.
| A restore replaces | A restore keeps |
|---|---|
| Every database the backup holds, exactly as it was backed up | Databases the backup does not hold, such as one of a module switched on after the backup |
The .env (not with --db-only or --skip-env) |
The release the install runs: its programs and deployment files |
The certificate material (not with --db-only) |
Models, runtime images and the machines |
The volumes the backup holds (a backup with --include-volumes) |
The volumes the backup does not hold |
Room it needs. Where Postgres keeps its data, room for a copy of each database: as much as the database takes now, or ten times its dump where that is more, plus 1 GB or a tenth, whichever is more, kept free. For a backup with volumes, room for a copy of those volumes beside the install as well. Otherwise the restore says how much it needs and changes nothing.
| Option | What it does |
|---|---|
--verify-only |
Check the backup's checksums, restore nothing |
--db-only |
Restore the databases only |
--skip-env |
Keep the current .env |
--key-file PATH |
Read the recovery key from this file instead of asking for it |
-y |
Do not ask |
Warning
A restore replaces the current databases. Everything done since the backup is lost: chats, users, settings, machines approved since.
| You see | What it means |
|---|---|
Not enough room for the restore ... |
Nothing changed. Free the space it names, or restore on a server with more room. |
The restore failed: pg_restore failed for <database> |
Nothing changed; the install runs as before. The log it names holds pg_restore's whole output. |
The restored install did not become healthy ... |
Everything went back as it was before the restore and is running again. Read faden logs <service> for the services it names. |
An earlier restore left <name>__old ... |
A restore stopped after the swap and before it could drop the databases from before it. Check them; the message gives the command to drop them, then run the restore again. |
Check a backup¶
Run faden restore --verify-only on a copy of a backup now and then. It tells you whether the database dumps, the volume archives and the
certificate files are complete and unchanged, without touching the install.
Rebuild a lost server¶
Restoring is still being tested: try it on a copy of the server first (see Restore).
- Install a new server with the release the backup holds, using
faden deploy.faden restore --verify-onlyshows it as Backup of release … (releasein the backup'smanifest.json). A backup taken before 0.7.0 records only thefadenversion that took it, and says so; for an upgrade's backup that is the release being installed, while the data are those of the release before it. - Copy the backup onto it and run
faden restorewith its path, and the recovery key if the backup is encrypted. The new server encrypts its own backups to the key itsfaden deploymade. - If the new server has another address, run the install line from its console on each machine again, so the agents connect to it (see GPU machines).
- Run
faden runtime-imagesif the machines will need runtime images the server does not hold yet. - Clusters created after the backup are not in it, but their machines still run them. Take them over instead of creating them again: see Taking over clusters.
Because the backup carries the certificate authority, machines that trusted the old server trust the new one.