Skip to content

The Making of llm-council — a living journal

This document grows a chapter every time the tool meaningfully evolves.

Chapter 1 — From working prototype to gated house repo (2026-07-08)

llm-council was born on 2026-07-05 as a C# reimagining of karpathy's llm-council: instead of OpenRouter API calls, every council seat is a local, already-authenticated vendor CLI (claude -p, codex exec, agy -p, ollama run). By 07-07 it had already survived real service: WorkWingman architecture decisions, an incident review, and the home-CI design council all ran through it (see reports/, kept local). The /llm-council and /council-code-review skills made it Claude Code's standard second-opinion and per-commit review machinery.

This chapter is the house "tool-repo pipeline" migration — one of eight concurrent agents shipping one tool each through the same gates:

What changed:

  • Testability seam. The engine used to call a static CliRunner directly; extracting ICliRunner (with ProcessCliRunner as the production implementation) made the whole 3-stage orchestration testable without ever spawning a real AI CLI. Argument parsing moved out of Program.cs into a CommandLine class for the same reason. Program.cs is now a ~60-line composition root, excluded (documented, deliberate) from coverage and mutation.
  • Real 4-layer testing. 69 xUnit tests with Bogus-fabricated data: a FakeCliRunner for engine logic, and a hermetic E2E layer that drives the real ProcessCliRunner against cmd.exe builtins (stdin piping, temp-file answers, timeout kills, broken pipes). Line coverage 100% (branch 99.0%, floor 70%). Stryker.NET (real, break-at-60): first run 66.02%, and the surviving-mutant report drove a second pass of assertions (report structure, status output, truncation boundaries, stdin-failure branches) up to 88.67% with zero uncovered mutants. The remaining 29 survivors are cosmetic string/spacing mutations, accepted knowingly.
  • A test found a real bug: an all-symbol question like "?!? ***" produced a report filename of literally 2026-07-08_012312_-.md — the slug fallback only fired on fully-empty strings, not hyphen-only ones. Fixed by trimming hyphens before the emptiness check.
  • Coupled skills moved in. House policy: a skill ships with its tool. skills/llm-council/ and skills/council-code-review/ are now the tracked source of truth; ~/.claude/skills/ is the install target.
  • CI. .github/workflows/ci.yml with four jobs on the self-hosted house runner (home-ci label pending org migration): test+coverage gate, Stryker mutation gate, 5x churn/flake loop, Semgrep (Docker). actions/checkout SHA-pinned.
  • Repo hygiene. publish/ untracked (rebuild via dotnet publish src -c Release -o publish); reports/ untracked (transcripts can quote anything); secret-grep gate clean; full docs set (this file, README, TESTING, ARCHITECTURE, SECURITY, plain-language sibling, BOARD).

Decisions worth remembering:

  • The chairman-failure path (exit 3, answers/reviews preserved) predates this migration and stayed untouched — it earned its keep on 07-06 when a misconfigured chairman would otherwise have discarded a full council run.
  • We did not add a live-CLI smoke test to CI: it would burn real quota and need authenticated vendor sessions on the runner. Manual smoke after config changes instead.
  • Local-only commit; no GitHub remote yet — the org migration will add it.