Skip to content

FLT-9 · Vikunja hardening + DR

Vikunja (http://localhost:3456, Docker containers vikunja-vikunja-1 + vikunja-db-1, Postgres 16 data in the vikunja_vikunja-db volume) is one of three redundant copies of the fleet's ticket state, alongside Jira Cloud and this repo's own docs/BOARD.md. A Docker restart on 2026-07-14 exposed that it runs on a single PC with no dump schedule and no standby — losing this PC loses real work. This pass hardens the backup/verify/watch story. It is additive only: nothing here reconfigures, restarts, or writes to the live Vikunja or Postgres containers.

Scripts live in ops/vikunja-dr/ in this repo:

ops/vikunja-dr/
  backup/Backup-VikunjaDb.ps1        nightly read-only pg_dump -> OneDrive
  verify/Test-VikunjaRestore.ps1     restore into a THROWAWAY container, assert data
  watchdog/VikunjaWatchdog.psm1      healthz-watch core logic (config/state/log)
  watchdog/Watch-VikunjaHealthz.ps1  healthz-watch Scheduled Task entry point
  install/Install-VikunjaDrTasks.ps1     registers all 3 Scheduled Tasks
  install/Uninstall-VikunjaDrTasks.ps1   removes them

What is backed up

Security warning: each .sql dump is the full Vikunja database. It contains password hashes, API/session tokens, all users, all projects, all tasks, and other application data. By default it lands in %USERPROFILE%\OneDrive\VikunjaBackups\, so it syncs to OneDrive cloud storage unencrypted by these scripts.

Backup-VikunjaDb.ps1:

  1. Confirms vikunja-db-1 is running (docker inspect) — aborts and touches nothing if it isn't.
  2. Runs psql/pg_dump inside the database container over its local Postgres socket, without hardcoding the password or expanding it into a host docker.exe command line.
  3. Reads a read-only row-count baseline from the live db (projects, tasks, users, plus every project's id+title) via a plain SELECT — never a write.
  4. Runs pg_dump -U vikunja -d vikunja --clean --if-exists inside the vikunja-db-1 container (writing to a /tmp path inside the container), then pulls the file out with docker cp and byte-verifies the host copy against the in-container size. This mirrors the claude-mem standby-sync pattern (claude-mem-house/scripts/claude-mem-standby-setup.ps1) and avoids PowerShell console stdout-redirection encoding corruption on non-ASCII task/project titles that a host-side docker exec ... > file risks. The /tmp dump is deleted from the live container in a finally block on success or failure. pg_dump only takes a consistency-snapshot lock; it never writes to the source database. --clean --if-exists only changes the text of the dump (adds DROP ... IF EXISTS statements for a clean restore) — it does not touch the live database.
  5. Sanity-checks the dump size (floor: 2000 bytes) before copying it anywhere, so a broken/empty dump can't silently become "the backup."
  6. Writes the dump to %USERPROFILE%\OneDrive\VikunjaBackups\vikunja-<timestamp>.sql plus a <same-name>.sql.meta.json sidecar containing the exact projects/tasks/users counts and known project titles captured at dump time. The sidecar is what makes restore verification an exact check instead of a live-vs-restored comparison that can drift.
  7. Prunes to the newest 30 dumps (~1 month of nightlies) plus their sidecars.
  8. Logs one JSONL line per run to %USERPROFILE%\.vikunja-dr\backup.log.jsonl.

Where it lands

%USERPROFILE%\OneDrive\VikunjaBackups\ — OneDrive means the dump survives this PC dying, without adding a second-PC dependency (the warm standby is explicitly deferred, see below). This is cloud storage, and the scripts do not encrypt the dump before OneDrive syncs it.

How to restore

Never restore over the live database. To recover Vikunja after a real loss:

  1. Pick a dump from OneDrive\VikunjaBackups\ (newest, or a specific date).
  2. Stand up a fresh vikunja-db-1-equivalent Postgres 16 container / volume (or, if the live container still exists but its data is corrupted, take it down first — this is the one case where restoring is a write, and it is a deliberate human decision, not something these scripts do automatically).
  3. docker cp <dump>.sql <container>:/tmp/restore.sql
  4. docker exec <container> psql -U vikunja -d vikunja -f /tmp/restore.sql
  5. Restart vikunja-vikunja-1 if needed so it reconnects.

Test-VikunjaRestore.ps1 (below) exercises steps 3-4 automatically, just against a throwaway container instead of a real one — read it for the exact command shapes.

How the verification proves the dump is good

"A dump nobody has restored is not a backup." Test-VikunjaRestore.ps1:

  1. Picks the newest dump in OneDrive\VikunjaBackups\ (or -DumpPath).
  2. Loads its .meta.json sidecar for the exact expected counts (falls back to a fresh read-only SELECT against the live db if a sidecar is missing, logged as a weaker baseline since it can drift).
  3. Starts a throwaway postgres:16 container (docker run --rm, no published port, no named volume — nothing survives it) named vikunja-restore-verify, with a freshly generated random password that has nothing to do with the live database's credentials.
  4. Waits for it to be stably ready — two consecutive successful SELECT 1 pings 2 seconds apart, because the official Postgres image runs a temporary bootstrap server during initdb that can make pg_isready report ready for a few hundred milliseconds before it shuts down and the real server starts.
  5. Copies the dump in and restores it with psql -f into that container only.
  6. Asserts: restored projects/tasks/users counts equal the expected baseline exactly, at least one project exists, and every known project (id + title) from the baseline is present after restore. This proves the dump can be restored and those core rows survived, but it would not catch a future dump that silently excluded labels, comments, attachments, tokens, or other tables not covered by these assertions.
  7. Tears the throwaway container down with docker rm -f only after verifying the container has the script-owned Docker label flt9-throwaway=true. No named volume was ever created, so there is nothing else to clean up.
  8. Logs to %USERPROFILE%\.vikunja-dr\restore-verify.log.jsonl.

It never touches vikunja-db-1 or vikunja-vikunja-1 beyond the one read-only SELECT used for the sidecar-missing fallback.

Real proof run (2026-07-15)

Ran directly against the live database, then verified the resulting dump:

BACKUP OK: C:\Users\fives\OneDrive\VikunjaBackups\vikunja-20260715-011308.sql
            (137849 bytes; projects=2 tasks=50 users=1)
    copied to ...\vikunja-20260715-011308.sql
            (137849 bytes on host; matches container byte count)

RESTORE VERIFY OK: vikunja-20260715-011308.sql -> projects=2 tasks=50 users=1
            (baseline: sidecar (exact, captured at dump time))
    restored project titles: 1=Inbox; 2=FLEET — Universal Harness

After the run: docker ps -a --filter name=vikunja-restore-verify returned nothing (throwaway container fully gone), find /tmp ... vikunja-*.sql inside vikunja-db-1 returned nothing, both live containers were still up (vikunja-db-1 healthy, vikunja-vikunja-1 up), and the live vikunja-db-1 counts were re-checked read-only and unchanged (projects=2, tasks=50, users=1).

Healthz watch

Watch-VikunjaHealthz.ps1 (entry point) + VikunjaWatchdog.psm1 (core logic) follow the shape of the existing house watchdog at claude-session-watchdog/src/{watchdog.ps1,Watchdog.psm1}: config-merge from a JSON override file, a JSONL incident/state file, a JSONL append-only log, and an optional Pavlok-paging escalation with a repage budget — rather than inventing a new shape for this ticket.

Deliberate differences from that pattern:

  • No time window / mission marker. The session watchdog only arms during a declared overnight mission; Vikunja is read by humans and agents all day, so this watchdog always checks.
  • Pavlok paging defaults OFF (pavlokEnabled: false in %USERPROFILE%\.vikunja-dr\config.json). FLT-9 asked for a healthz watch; wiring unattended wristband pages is a bigger decision than this ticket's scope. The hook is fully implemented (same incident/repage/budget logic as the session watchdog) and can be turned on by dropping a config.json with "pavlokEnabled": true in that directory — no code change needed.
  • Checks http://localhost:3456/api/v1/info (Vikunja's unauthenticated info endpoint; asserts HTTP 200 and a version field in the JSON body) rather than a dedicated /healthz route, because Vikunja doesn't expose one.

Real run against the live endpoint (2026-07-15):

[2026-07-15T00:48:58] healthy: 200 OK, version v2.3.0

Down-detection was sanity-checked against a deliberately unreachable port (not the real Vikunja endpoint) and correctly produced down → down (incident threshold reached, paging disabled) after 2 consecutive failures, with no page sent (paging is off by default).

Scheduled Tasks

install/Install-VikunjaDrTasks.ps1 registers three tasks (idempotent, -Force; probes for pwsh, falls back to powershell.exe if pwsh resolves to a WindowsApps store alias Task Scheduler can't launch — same logic as claude-session-watchdog/install/install-task.ps1):

Task Schedule Runs
VikunjaNightlyBackup daily 02:15 Backup-VikunjaDb.ps1
VikunjaRestoreVerify weekly, Sunday 03:00 Test-VikunjaRestore.ps1 against the newest dump
VikunjaHealthzWatch every 10 minutes, all day Watch-VikunjaHealthz.ps1

Not yet run via Install-VikunjaDrTasks.ps1 on this PC as part of this pass — the scripts were proven directly (see the real runs above). Run it once ready to put these on the actual schedule:

pwsh -File ops\vikunja-dr\install\Install-VikunjaDrTasks.ps1

Uninstall-VikunjaDrTasks.ps1 removes all three tasks; it does not touch dumps, logs, or any Docker container.

What is still missing (explicitly deferred, per FLT-9 scope for this pass)

  • Second-PC warm standby. The claude-mem DR pattern (hourly sync to a full second stack) is the intended model, but stood up deliberately out of scope for this pass.
  • Mesh exposure (phone/tester access). Depends on FLT-8 (Tailscale/ Headscale), which is still in progress — exposing Vikunja over the mesh before that lands would be premature.
  • API token provisioning for agents and the board-sync script are explicitly FLT-11 (owned by another session), not this ticket.

Known caveats

  • The current dump/verify proof was run manually against the real database and a real dump; the Scheduled Tasks themselves have not yet executed on their actual triggers (no overnight cycle has passed since this pass). The underlying scripts are what will run either way — only the trigger wiring is unexercised.
  • Windows PowerShell 5.1 (what Task Scheduler launches, per the install-task.ps1 probe/fallback) converts native-command stderr into a terminating exception under $ErrorActionPreference = 'Stop' — the same landmine already documented in claude-session-watchdog/src/Watchdog.psm1's Invoke-PavlokPage. Both new scripts here run under $ErrorActionPreference = 'Continue' and check $LASTEXITCODE explicitly for exactly this reason; it was found the hard way while proving Test-VikunjaRestore.ps1 (a routine docker rm -f on an already-gone container was throwing before the fix).
  • Retention (30 nightly dumps) is untested at scale — it prunes by filename sort, which works because the timestamp format sorts lexicographically, but hasn't been exercised with 30+ real dumps yet.