---
name: thermograph-ops
description: Deploying, monitoring and debugging the Thermograph estate — release flow, reading the fleet, and what to do when something breaks.
---

# Thermograph operations

## Release flow

Code stages by branch, and the branch decides the environment:

```
PR into dev  →(required checks)→  dev  →  vps1 (dev.thermograph.org)
                                  dev  →(promotion PR)→ main → beta (vps2)
                                                        main →(promotion PR)→ release → prod (vps2)
```

Promotion is continuous and per-decision — there is no release calendar. Size a
promotion to whatever has actually soaked on the environment below.

Branch protection blocks direct pushes to `dev`/`main`/`release` for everyone.
Forgejo has no auto-merge-on-green, so merges are explicit.

### Who may merge what

Everything is a PR; that is about how a change travels, not who lands it. Who
lands it is three tiers, and they are not symmetric:

| Hop | Whose call | What to do |
|---|---|---|
| anything → `dev` | yours | Merge it. Do not leave outstanding PRs — an open PR into `dev` is unfinished work, not caution. |
| `dev` → `main` (beta) | yours, batched | Promote when a batch is coherent enough to be worth *testing*. Beta is the test environment; not promoting is not testing. |
| `main` → `release` (prod) | **the owner's** | Prepare it, recommend for or against, and ask. Never unilaterally, and never by another route. |

`promote(from, to)` implements exactly this: into `dev` and `main` it opens the
PR, waits for CI and merges; into `release` it prepares everything — what would
ship, the CI evidence, how long it soaked on beta — and hands you a
recommendation to put to the owner. "Ready, but I would wait, because X" is a
legitimate and useful answer.

`forge_pr_merge` enforces the same rule, so there is no back door: it merges
into `dev` and `main`, refuses `main` → `release` under any flag, and requires
`confirm_protected_base: true` for the separate hotfix path (a non-promotion PR
straight at `main`/`release`).

**Commit counts are not deliverable content.** Before promoting, `promote` and
`branch_diff` compare the two tips' *trees*. Identical trees mean the same
changes are already on the target under different SHAs — a promotion there
merges nothing while still firing the target's deploy workflows, including
`infra-sync` re-rendering the rendered env file on prod and beta (two separate
files on vps2 now — `/etc/thermograph.env` and `/etc/thermograph-beta.env`).
`promote` refuses that outright; the ahead/behind numbers will look like real
work.

Expect `dev`, `main` and `release` to be mutually divergent, and expect
`--ff-only` to fail: every promotion merges the source into the target, creating
a merge commit *on the target* that the source never receives, so each promotion
widens the gap it just closed. That is the accumulated design, not drift.

**Image tags are keyed to the last commit that touched a domain**
(`git log -1 -- backend/`), not the branch tip. Both build and deploy compute
the same key, so an infra-only commit at the tip cannot send a deploy chasing an
image no build produced. This is why `deployed_version` can show a tag that is
not the head of the branch — that is correct, not drift.

Rollback is redeploying the previous image tag. Tags are per-commit and GC keeps
the running set.

## Reading the fleet

Start wide, then narrow:

1. `fleet_status` — HTTP reachability plus what is running on each host.
2. `logs_overview since="1h"` — where the volume is, and what has gone quiet.
3. `logs_query service=… host=…` — the actual lines.

**The Swarm trap.** Prod and beta both run Swarm stacks now (colocated on
vps2), and their `job="docker"` streams carry **no `service` label**. A raw
`{host="prod", service="backend"}` query matches nothing and reads as "no
logs" when it means "wrong selector". Use the `logs_query` tool's `service`
argument, which rewrites it to a container name-regex with the churning task
suffix wildcarded. Prod service names are `thermograph_web`, `_worker`,
`_frontend`, `_db`, `_autoscaler`, `_lake`; beta's are the same shape under
`thermograph-beta_beta-*` (`_beta-web`, `_beta-worker`, `_beta-frontend`,
`_beta-lake`, `_beta-daemon` — note the doubled `beta`), with no `_db` of its
own. dev, still compose on vps1, does carry a `service` label.

Retention is 30 days. `thermograph-test_*` containers are rehearsal leftovers —
ignore them.

## Databases

No database is exposed over TCP. Each listens only on its private docker
network, and prod's and beta's are both Swarm **overlays on vps2 that the host
itself cannot route to** — so `ssh -L` is *impossible* for both now (beta used
to be compose, where it worked; it moved). `sql_query` execs into the
container instead, which works identically everywhere.

Beta has no database container of its own: one TimescaleDB instance on vps2
serves both beta and prod, as separate databases (`thermograph_beta` /
`thermograph`) under separate roles. A beta query resolves prod's container
but connects with beta's own role to beta's own database — `sql_query` and
`data_freshness` already do this indirection for you.

Reads run as each environment's own read-only role (`thermograph_ro` for dev
and prod, `thermograph_beta_ro` for beta — NOSUPERUSER, `pg_read_all_data`).
Read-only is enforced by Postgres, not by convention:

```
sql_query(env="prod", sql="create table t(i int)")
→ ERROR: permission denied for schema public
```

`write:true` escalates to the app's superuser role and takes effect immediately
with no confirmation. On prod, know what you are doing.

## The lake

`lake_query` runs DuckDB over the ERA5 Iceberg warehouse in Contabo object
storage — one shared warehouse readable from every environment, so `env` only
picks where the engine runs. Filter on partition columns (tile/year/month): the
scan is partition-pruned, and a predicate on them is the difference between a
fast query and reading the whole warehouse.

## When something breaks

1. `fleet_status` — is it up at all?
2. `logs_query host=prod contains="(?i)error|traceback"` since the incident.
3. `deployed_version env=prod` — did something ship just before it started?
4. `forge_prs` / `forge_ci` — what landed recently, and did CI actually pass?
5. Write down what you found with `notes_write` **before** fixing it. The
   reconstruction is worth more than the fix, and it will be gone by tomorrow.

For anything the typed tools do not cover, `run_on_host` gives a shell as the
`agent` user (passwordless sudo on the VPSes). Every call is audit-logged.

## Things that will bite you

- **Prod and beta container names change on every redeploy** (Swarm task
  suffix, both now that beta is a Swarm stack too). Never hardcode one; resolve
  via `docker ps --filter name=` at call time.
- **Contabo object storage is path-style only**, `region=default`. Virtual-host
  addressing fails outright.
- The **`backups/` prefix** in that bucket belongs to the nightly backup jobs.
  Do not write outside the lake's own prefix.
- **Terraform has no state anywhere.** It is executable documentation; a casual
  `terraform apply` would try to re-provision live prod from scratch.
- **Grafana dashboards are provisioned from repo JSON.** An edit through the UI
  or API is silently overwritten on the next provision — change them with
  `dashboard_write`, which opens a PR.
- Backups are a single copy on the same box as the database, with no offsite
  yet, and restores must handle TimescaleDB's `continuous_agg` circular-FK
  warning (`--disable-triggers`).