Skip to content

llm-council

Plain English: you ask one question, and instead of trusting a single AI, this tool asks several different AIs at once (Claude, Codex/ChatGPT, Gemini, and a model running on this PC). They each answer without seeing the others, then grade each other's answers without knowing who wrote what, and finally a "chairman" AI reads everything and writes the best combined answer. You get one solid, cross-checked verdict plus a full transcript of the debate.

A C# take on karpathy/llm-council: ask one question, get answers from Claude, Codex, and Gemini, have them anonymously peer-review each other, and let Claude — the chairman — synthesize a final verdict.

No API keys, no OpenRouter. Every seat talks to its vendor's local CLI using your existing subscriptions:

Seat CLI Plan used
Claude (chairman + member) claude -p Claude Max
Codex codex exec ChatGPT / Codex Max
Gemini agy -p (Antigravity CLI, Gemini CLI's successor) Google AI Pro

Usage

llm-council "Should I use SQLite or Postgres for a desktop app?" --tier fable

llm-council "<question>" [--tier fable|opus|sonnet|haiku] [--config path]
            [--reports-dir dir] [--json]
  • Stage 1 — all seats answer independently, in parallel.
  • Stage 2 — each seat ranks the anonymized answers (A/B/C) and critiques them.
  • Stage 3 — the chairman sees everything (deanonymized) and issues a verdict, resolving disagreements explicitly, ending with per-member "Council notes".

A full markdown transcript of every run is written to reports/. reports/ is deliberately gitignored: transcripts quote whatever the seats said verbatim (which can include project context from the question), so they stay local to each machine. Keep them — they are the council's institutional memory — just don't commit them. --json emits a machine-readable result on stdout (used by the Claude skill). The question can also be piped via stdin (useful for long/multi-line prompts).

Exit codes: 0 every configured seat contributed and the chairman synthesized · 2 usage error · 1 fatal error (no report) · 3 chairman failed or misconfigured — the report and --json output are still written with every seat answer and peer review preserved, so you (or another model) can chair the synthesis manually. The JSON chairmanFailed / chairmanError fields flag this case. · 4 roster degraded — a verdict exists, but at least one configured seat did not contribute (never answered, or answered and then failed peer review), so the verdict rests on less independence than the seat count implies. The JSON rosterDegraded, configuredSeatCount, and degradedSeats[] fields name each one; the report says so above the verdict.

3 and 4 are degraded successes — stdout stays fully usable in both, and one dead seat is still survivable (only a total wipeout is fatal). 3 outranks 4 when both apply, because it names the more urgent action: there is no verdict to weigh at all.

A council is only worth what the independence behind it is worth, so only 0 means a full council answered. If you gate on the council (llm-council ... && push), that gate is exactly as honest as this exit code — which is why a shrunken roster no longer reports success. Callers that would rather proceed on a degraded verdict should say so explicitly (if ($LASTEXITCODE -eq 4) { ... }) instead of learning it by accident.

Tiers

--tier matches the council to the strength of whatever model asked for it. Running from a Fable session? Every seat uses its fable-equivalent. Haiku session? Small/fast equivalents everywhere. Mapping lives in council.config.json:

Tier Claude Codex Gemini
fable claude-fable-5 gpt-5.5 (xhigh effort) Gemini 3.1 Pro (High)
opus opus gpt-5.5 (high) Gemini 3.1 Pro (Low)
sonnet sonnet gpt-5.5 (medium) Gemini 3.5 Flash (High)
haiku haiku gpt-5.5 (low) Gemini 3.5 Flash (Low)

agy quirks (found the hard way): --model takes the exact label from agy models — e.g. "Gemini 3.1 Pro (High)". Unknown ids are silently ignored and fall back to the default (Gemini 3.5 Flash Medium), so a typo won't error — check ~/.gemini/antigravity-cli/log/ for Propagating selected model override to verify. And agy blocks forever if stdin is an open pipe; this app always closes the child's stdin, but when testing by hand use agy -p "..." < NUL.

Vendors rename models constantly — when one breaks, edit the JSON, no rebuild needed. Seats are also fully config-driven (command, args, prompt via stdin/arg, per-tier extra args), so adding a fourth member is a JSON edit.

If a seat fails (not logged in, model rejected, timeout), the council continues with the remaining seats and the report says so.

Build

dotnet publish src -c Release -o publish

publish/ (including llm-council.exe) is not tracked — it is a build artifact; rebuild it with the command above whenever src/ changes.

Test

pwsh -File tests/Invoke-CI.ps1   # xUnit suite + 70% coverage floor
dotnet stryker                    # mutation testing (Stryker.NET, break < 60%)

See TESTING.md for the full 4-layer matrix.

Claude skills

The two Claude Code front-ends for this tool live in this repo (house policy: a coupled skill ships with its tool) and are installed by copying to ~/.claude/skills/:

  • skills/llm-council/SKILL.md — /llm-council: infers the tier from the model powering the current session, runs the binary, relays the chairman's verdict, and saves findings to claude-mem.
  • skills/council-code-review/SKILL.md — /council-code-review: the standard per-commit review gate; fans the repo's uncommitted diff to Codex, Gemini, a local Ollama seat, and the house PC councils, with Codex chairing the synthesis.

Security

  • No secrets in this repo. The tool holds no API keys at all — every seat is a locally-installed CLI already authenticated under your own accounts (Claude Max, ChatGPT/Codex, Google AI Pro, local Ollama). council.config.json contains model names and CLI arg templates only.
  • Seat commands come from the local config file, not from network input; the prompt is passed via stdin or a single argv entry (no shell interpolation).
  • Reports may quote anything a seat said, so reports/ is gitignored.
  • CI runs a Semgrep scan (Docker image) on every push; the repo is checked against the house secret-pattern grep before every commit.
  • Details: docs/technical/SECURITY.md.