Upgrading¶
How to move the server to a new release, then the agents on the GPU machines, and how to move a cluster to a newer runtime. Each is a separate step that you start; nothing upgrades itself.
| What | How | What it interrupts |
|---|---|---|
| The server | faden upgrade |
The console and the gateway, for the minutes the services restart. Models keep running |
| A machine's agent | Run the install line again on the machine | Nothing: the agent restarts, the models keep running |
| A cluster's runtime | Runtime on the cluster's page | Every model on that cluster, while it loads again |
How you hear about a new release¶
Once a day the server reads the public list of Fadenstack releases, https://fadenstack.com/updates/v1/index.json.
Nothing about your server is sent: the list is fetched like any file and compared on the server. When a newer
release is out, the administrators see it:
| Where | What |
|---|---|
| Updates in the console | The release this server runs, the newest one and what it fixes, the commands to run for this install, and what each part runs: the services, every machine's agent, every cluster's runtime |
| Needs attention on the dashboard | An item for each newer release; a release that fixes security problems as a warning |
| A banner on the console's pages | For a release that fixes security problems, and when the release this server runs no longer gets fixes |
| One message to the administrators for a release that fixes security problems, when e-mail is set up | |
The faden command |
One line after a command, at most once a day: Fadenstack <new> is available |
Nothing is installed by itself: you run the commands, as described below.
The same daily check also reads the list of Fadenstack apps (the desktop agent, the Office add-ins, the Nextcloud app and the Chrome extension) in the same way: see Fadenstack apps.

When a release is out¶
-
A banner on the console's pages, or an item under Needs attention on the dashboard, says that a release is out. Select See updates, or the item.

-
Updates shows the release this server runs, the newest one, how urgent it is (Update available, Security update, Critical update) and what it fixes. Release notes opens the release on fadenstack.com/releases.

-
Under How to upgrade, copy the commands and run them on the server, as in Upgrade the server. They match how
fadenwas installed on this server, and name the exact release when one cannot be skipped.
-
When the upgrade is done, Updates says Up to date. When the release brings a newer agent for the GPU machines, Updates says so, and the machines show Agent version available: see Update the machines' agents.
A banner for a security update can be hidden until the next release (Hide until the next release, in this browser). A banner for a critical update, or for a release that no longer gets fixes, cannot be hidden.
On the command line, faden says the same after a command, at most once a day:
$ faden status
...
Fadenstack 0.4.2 is available (you have 0.4.1) -- it fixes security problems.
Upgrade: pipx upgrade fadenstack, then faden upgrade
Parts of this install
Below the releases, Updates lists what each part runs against what this release expects: every service, every machine's agent, every cluster's runtime, and the release the command-line tool last installed.
| Status | Meaning |
|---|---|
| Current | It runs what this release expects |
| Update available | A newer one is here to take: a machine's agent, or a cluster's runtime (see below) |
| Other release | A service runs another release than the rest. Run faden upgrade again |
| Unknown | It does not say which release it runs: a service built on this server from source code, or a machine that never reported |
| Stopped | The service is not running |
More about the update check
- Check now on Updates reads the list at once; the daily check needs no one.
faden deployasks whether to check;--update-checkor--no-update-checkanswers without asking. Change it later in the console under Settings, Updates (Check for new releases), and for the command line withupdate_check: falseinfaden.yamlorFADEN_NO_UPDATE_CHECK=1.- Release channel
previewalso announces release candidates, to try a release before it is out. - The list is signed. The server and
fadenaccept only a list signed by the Fadenstack project, and never an older list than the one they have, so a changed list can neither hide a release nor invent one. faden versionandfaden doctorcheck at once and say how to upgrade.- Some releases cannot be skipped. Updates and
faden versionthen name the releases to install one after the other, andfaden upgraderefuses to skip one. - On a command line that is not a terminal (scripts, the backup timer)
fadensays nothing.
A server without internet access¶
- Behind a proxy: set Proxy for the release list under Settings, Updates.
- With a web server of your own: copy
index.jsonandindex.json.sigfromhttps://fadenstack.com/updates/v1/to it, unchanged, and set Release list address to the copy's address. - With neither: download the two files on another computer and upload them on Updates, under No internet on this server?
Or switch the check off: Updates still shows what each part of the install runs.
Before you start¶
- Free disk space. The new images need a few GB, a new runtime image up to about 15 GB per kind of machine,
and the backup the upgrade takes needs room for the databases. Check with
df -h. - Know which install the tool is set up for:
faden config shownames it (install_dir). - Pick a time. People see a maintenance page while the services restart.
Upgrade the server¶
-
Upgrade the command-line tool. It carries the release:
Updates in the console shows this command for the way the tool was installed on this server, and the exact version when a release cannot be skipped.
-
See what the upgrade would do:
It names the release it moves to (
Release: <current> -> <new> (pull published images)). -
Run it:
-
Check it (below).
What the upgrade does, in order
- checks Docker, and that no container it is about to replace was created by something else;
- backs up every database, the
.envand the certificate material into~/fadenstack/backups/<time>/, and stops if nothing could be backed up; - replaces the deployment files with the new release's, and adds to
.envonly the settings the release introduces: every value you set is kept, except the sizing of the services, which is worked out again for the host asfaden tunedoes. A key that seals stored data and is missing gets the value the install ran on, never a new one, and the upgrade says so (see Secrets in .env); - pulls the new release's images;
- behind the maintenance page, first copies the chat attachments that an install from before release 0.4.4 holds
inside the gateway container into their volume, and compares every file by its SHA-256 hash. If a file
cannot be copied or does not match, the upgrade stops here, before anything is replaced: the old services
keep running and the files are untouched (see the table at the end). With
FADEN_DATA_DIRset, it likewise copies the files of the chat and knowledge volumes into the data folders and verifies them (see Where the company data is kept); - restarts the services; the database migrations run here, and the two Postgres
logins with fixed passwords that earlier releases made (
llm_user,faden_backend, used by nothing) are switched off. Before release 0.7.1, Langfuse kept its users, projects, API keys and prompts in the server's own databasepostgres, which a backup leaves out: the upgrade moves them into their own databaselangfuse, once, with Langfuse stopped and every table's rows compared, so backups hold them from then on. Their old copy stays inpostgresfor going back. If the move cannot be made, Langfuse stays where it was and the upgrade says so; - waits for the console API to answer, then lifts the maintenance page;
- fetches the runtime images the new release names. Only what changed is downloaded. A failed pull stops
nothing: retry it with
faden runtime-images.
| Option | What it does |
|---|---|
--dry-run |
Show what would happen, change nothing |
-y |
Do not ask before starting |
--backup-dir PATH |
Put the pre-upgrade backup elsewhere |
--no-backup |
Skip the backup. Only when you have a current one |
--runtime-images ARCHS |
Which runtime images to fetch: all, none, x86_64, aarch64. Remembered |
The chat checks roles since release 0.4.2
Before, every active account could chat. Now people chat with default_user or a role that has the
chat.* permissions. After the upgrade, an account you gave only other roles, such as operator or
viewer, no longer chats: tick default_user for it as well (see Users).
Warning
An upgrade only moves forward: the databases are migrated to the new release. The tool refuses to move an install to an older release, and going back to the release before is not possible yet (see Going back).
Changes you made to the deployment files themselves, such as nginx/nginx.conf, are replaced. Keep your changes
in .env.
What people see meanwhile¶

| Who | What they see while the services restart |
|---|---|
| People in the console | Every page of the console shows "Fadenstack is being updated" and reloads itself into the console once it is back. |
| Scripts and SDKs | The API (/api) and the gateway (/v1) answer 503 with Retry-After: 30, so scripts and SDKs can wait and retry. |
| A console page left open | It notices the new version and offers Reload; a page with text typed and not sent is never reloaded by itself. |
The models on the clusters keep running throughout: a cluster runs on its machines, not on the server.
Maintenance by hand¶
The same maintenance page can be switched on for your own work. The services keep running; only the front door changes.
faden maintenance on -m "Back at 15:00" # the note is shown on the page
faden maintenance status
faden maintenance off
An upgrade leaves maintenance you switched on by hand switched on.
Check it¶
faden status # every service up; healthy where it has a check
curl --cacert ~/fadenstack/tls/ca.crt https://ai.example.internal/api/health # 200
Then sign in to the console. The machines and clusters are as they were.
Upgrade the PII module¶
The resolved detector, pinned revision and locale pack are retained during upgrades, including -y.
A new recommendation does not replace the installed model. Use --pii-model auto to select the current
recommendation explicitly, or --pii-locale german-v1 to restrict content languages. Changing execution
with --pii-profile cpu preserves a catalog-configured detector. For older installations without a saved
detector ID, that explicit CPU option retains its previous meaning: migrate to Patronus ONNX.
An enabled PII module is included in faden upgrade. A disabled module stays disabled. Its existing secrets,
model directory and policies are kept.
- Run
faden upgrade --dry-runto inspect the planned upgrade. This makes no PII changes or model downloads. - Run
faden upgrade. The PII step checks GPU compatibility and offers a mode choice. The current mode is the default. On a CPU-only server with an older installation, chooselegacyto retain English-only detection orcputo adopt the multilingual CPU detector. - Run
faden doctorand review the installed profile in the personal data settings.
faden upgrade -y asks no questions and retains a compatible installed mode. If an installed GLiNER2 detector
cannot use its GPU, it runs on CPU with its existing profile; CPU processing is slower. It does not silently
switch an existing detector to another model. To change explicitly:
faden upgrade --pii-model patronus-onnx-v1 --pii-profile cpu
faden upgrade --pii-model gliner2-multi-v1 --pii-profile gpu-shared
A detector change keeps saved policies and requires explicit adoption of the installed profile in the console.
Without the matching profile, protected requests follow their configured failure policy. A required model
download or verification failure stops the upgrade before service replacement. For offline setup, supply a
verified --pii-model-dir.
Update the machines' agents¶
A new release may expect a newer node agent than the machines run. The Machines list and each machine's page then show Agent version available on that machine, with the version the server expects. Agents never update themselves, and the server never updates them: you pick the window.

- In the console, open Machines and select Add a machine, and copy the install line.
- On each machine, run the line again, the same way you ran it the first time: with
sudofor an agent that runs as a system service, without it for one that runs as a user service.
The installer downloads the agent the server now expects, sees that the machine is already a member, and restarts the agent on the new build. There is nothing to approve. The models on the machine keep running.
Why the models keep running
The agent is the machine's link to the server, and the cluster's containers do not depend on it running.
Update the agents soon after an upgrade. Some features need a recent agent on every machine of a cluster, and the console names the machines to update when one is missing (for example on the cluster's Traffic between machines card, or in a drain plan).
Machines without internet access¶
Machines without internet access install the agent from the server's own copy in ~/fadenstack/agent-binaries/.
Replace those files after every server upgrade with the new release's builds (faden-agent-linux-x86_64,
faden-agent-linux-aarch64, with their .sha256 files), from the project's releases page. Nothing replaces them
for you, and an old copy holds back the machines that rely on it.
Move a cluster to another runtime¶
A runtime is the image a cluster's machines run: Ray and vLLM together, built for that kind of machine. When a newer certified runtime is out, the cluster's Runtime card says so. Nothing changes until you switch.

- Open Clusters, then the cluster.
- On the Runtime card, choose the runtime under Switch to and select Switch.
- Read the warning and select Restart on this runtime.
Each machine's container is made again from the new image (a machine that does not hold it yet fetches it first), and every model on the cluster loads again. Plan a window for it.
More about runtimes
The cluster's Runtime card says whether it is certified: Certified, Partly certified or Not certified. A release ships a Recommended runtime for each kind of machine, can ship a newer Preview, and keeps an older one to go back to. A new cluster gets the recommended one.
A cluster keeps the runtime it was started with: through restarts, through recovery after a fault, and through server upgrades that ship a newer one. It also keeps the settings it was started with, such as where models and caches are mounted. When a release changes them, the Runtime card says so and the models keep serving; Apply and restart takes the new settings, and nothing restarts by itself.
A Preview runs models the recommended runtime cannot; choose it for a cluster that needs those models. The model marketplace says Cannot run here for a model that needs a newer runtime than the cluster runs (see Models).
Going back¶
Under testing
Going back to the release before an upgrade is being tested. The sure way back is still a snapshot of the whole server taken before the upgrade: a virtual machine snapshot or a disk image.
Every upgrade leaves a backup in ~/fadenstack/backups/<time>/ and names it. To go back to the release before
this one (0.7.0), restore that backup with that release's faden, in two steps. Everything done after the
backup is lost: chats, usage, approvals given since.
0.7.0 makes pictures without safeguards
0.7.0 has image generation on, without the safeguards the EU AI Act (Article 5) requires from 2 December 2026 (see Image generation is off). The backup brings back the picture models as they ran before the upgrade: after the restore, stop each one on its deployment page at once, and upgrade again as soon as you can.
-
Install the earlier release's
faden, the wayfadenis installed on the server (FADEN_CLI_INSTALLERin.envsays which): -
Restore the upgrade's backup with it:
It starts the earlier release: the backup's
.envnames it (VERSION), and its images must be on the server or in the registry the.envnames.
| The earlier release's restore replaces | It keeps |
|---|---|
| Each database the backup holds, made exactly the backup's: the tables this release added go | The deployment files as the upgrade wrote them; the earlier release starts with them |
.env and the certificate material |
Databases and volumes the backup does not hold |
The volumes the backup holds (a backup with --include-volumes) |
Models, runtime images, the machines |
A cluster created after the backup keeps running on its machines; take it over instead of creating it again (see Taking over clusters).
When it does not work¶
| You see | What to do |
|---|---|
These containers have names this installation uses, but it did not create them |
A container with one of Fadenstack's names was made some other way, by hand or by another compose project. The message lists them and the command to remove them (docker rm -f <name>). Removing a container keeps its data volumes. Then run the upgrade again. |
This CLI (<version>) is older than the install (<version>). |
The tool is older than the release the install runs. Upgrade the tool first. |
<version> cannot upgrade straight to <version>: <version> has to come first. |
A release in between cannot be skipped. Install the tool of the release named, run faden upgrade, then install the newer tool and run faden upgrade again. The message gives the commands. |
| Updates says the release list could not be read | The server has no internet, or needs a proxy. See A server without internet access. |
Backup failed — aborting upgrade. |
Usually a full disk, or Postgres not running. Free space, check faden status, and run it again. Use --no-backup only if you have a current backup. |
Chat attachments could not be copied ... or ... exist in the data volume with other content ... |
An install from before release 0.4.4 keeps chat attachments inside the gateway container, and the upgrade must copy them into the volume before it replaces the container. It could not, or the volume already holds a file of the same name with other content. Nothing was replaced and no file was overwritten; the old services still run. Free disk space or fix the cause the message names (for a file that differs, move the one you do not want out of the volume), then run faden upgrade again. |
Backend did not become healthy within 120 s. |
The services were restarted but the console API does not answer yet. Read faden logs faden-backend. The maintenance page is lifted anyway, so whatever runs can be reached. |
... this release names a build that is on no registry. |
A runtime image for one kind of machine is not published. The upgrade is not affected and running clusters keep their images. A machine of that kind can still get the image from another machine of its cluster that holds it. |
| The disk is full: Postgres, the log store and the other stores restart over and over | Nothing works until there is space: remove images no container uses (docker image prune), move old backups off the server, then run the upgrade again. The dashboard warns earlier: Disk space low. |