Architecture

Two diagrams: the component layout of the recommended (Compose) path, and the sequence of a Postgres failover (the core DR mechanism this blueprint exists to harden). The k3s learning lab is intentionally out-of-band — see ../../kubernetes/README.md for its own layout.

Component diagram — recommended path

flowchart TB
    subgraph Foundation["foundation/ (do this first)"]
        Hosts[Set-HomelabHostsAliases.ps1]
        Tailscale[Tailscale MagicDNS optional]
        TestRes[Test-NodeResolution.ps1]
    end

    subgraph Primary["Node: current primary"]
        DeployP[Deploy-ComposeStack.ps1]
        PG[(Postgres claude-mem)]
        VK[(Valkey cache)]
        WK[worker]
        DeployP --> PG
        DeployP --> VK
        DeployP --> WK
        WK --> PG
        WK --> VK
    end

    subgraph Standby["Node: warm standby"]
        Backup[Backup-PostgresWarmStandby.ps1 hourly]
        Verify[Test-PostgresRestore.ps1 nightly]
        Promote[Promote-PostgresStandby.ps1 on failover]
        Dumps[(local dump files + manifest.csv)]
        Backup --> Dumps
        Dumps --> Verify
        Dumps --> Promote
    end

    subgraph Marker["Primary-marker split-brain guard"]
        SPM[Set-PrimaryMarker.ps1]
        Alias[hosts alias: claude-mem-primary]
        SPM --> Alias
    end

    subgraph SwarmOpt["compose/swarm/ optional, stateless only, later"]
        InitSw[Init-Swarm.ps1]
        DeploySw[Deploy-SwarmStack.ps1 refuses stateful stacks]
    end

    Foundation -->|stable names required by| Primary
    Foundation -->|stable names required by| Standby
    PG -->|pg_dump over docker exec| Backup
    Promote -->|verify, deploy, restore, claim| Primary
    Promote --> SPM
    Backup -. hourly pull .-> PG

Sequence diagram — promote (failover) walkthrough

Promote-PostgresStandby.ps1 is the tested failover the council required — not an ad-hoc scramble at 2am. Every step below is logged and -WhatIf-able.

sequenceDiagram
    actor Op as Operator
    participant Promote as Promote-PostgresStandby.ps1
    participant Guard as Split-brain guard
    participant Verify as Test-PostgresRestore.ps1
    participant Deploy as Deploy-ComposeStack.ps1
    participant PG as Postgres (this node)
    participant Marker as Set-PrimaryMarker.ps1
    participant Alias as hosts alias

    Op->>Promote: run (elevated, real or -WhatIf)
    Promote->>Guard: is claude-mem-primary reachable and NOT us?
    alt old primary still alive
        Guard-->>Promote: THROW (refuse, avoid split-brain)
        Promote-->>Op: abort - confirm old primary is down, retry -Force
    else old primary down or -Force
        Guard-->>Promote: guard passed
        Promote->>Verify: verify latest dump restores
        Verify-->>Promote: PASS / FAIL
        Promote->>Deploy: bring claude-mem stack up here
        Deploy-->>Promote: healthy
        Promote->>PG: pg_restore latest dump into live primary
        PG-->>Promote: restore complete
        Promote->>Marker: claim primary + update hosts alias
        Marker->>Alias: point claude-mem-primary -> this node
        Promote-->>Op: PROMOTE COMPLETE (RTO: minutes)
    end
    Note over Op,Alias: Old primary, once recovered, runs<br/>Set-PrimaryMarker.ps1 -Action ClearPrimary

Notes

• Valkey is deliberately not part of the DR loop — it is treated as rebuildable cache/derived state (see ../pros-cons.md).
• The worker service in compose/stacks/claude-mem/docker-compose.yml is a placeholder image gated on Postgres/Valkey health; swap WORKER_IMAGE for the real worker.
• The k3s learning lab (kubernetes/) has its own create → deploy → fail-node → reschedule → teardown loop, documented in ../../kubernetes/RUNBOOK.md; it is deliberately not wired into this diagram because production DR does not depend on it.