Skip to content

Testing

This is the top-level entry point for how Odysseus is tested. It exists mainly to satisfy the house tool-pipeline's "every repo has a TESTING.md" convention and to point at the (much more detailed) testing documentation that already lives under tests/:

  • tests/README.md - concrete helper reference and how to run focused subsets (tests/run_focus.py, taxonomy markers).
  • tests/TESTING_STANDARD.md - the standard the suite is refactored toward: determinism, isolation, taxonomy, behavioral-first assertions.
  • docs/security-ci.md - what the security-focused CI jobs (gitleaks, zizmor/actionlint, hadolint, Trivy, dependency review) check and why.

The four layers (house pipeline convention)

Layer Status in this repo
Unit tests + coverage pytest, ~1,300+ test files under tests/. pyproject.toml sets testpaths = ["tests"], asyncio_mode = "auto", and a taxonomy of area_*/sub_* markers registered by tests/conftest.py. Coverage is not currently gated in CI; see the coverage run captured for this audit below.
Mutation testing Gap. No mutmut (or other Python mutation tool) config exists yet. Given the suite's size (~1,300 files), a full-repo mutmut run is not "quick" — it was not run as part of this audit. Documented here as an open item rather than silently skipped; a good first step would be scoping mutmut to a single high-value module (e.g. core/auth.py, src/tool_security.py) rather than the whole tree.
Churn (repeat-run flake detection) Gap, now filled. .github/workflows/ci.yml's python-tests job already runs continue-on-error: true because "the suite has known flaky / environment-dependent failures" (tracked under the ROADMAP "fresh install smoke tests" item). This audit adds .github/workflows/churn.yml, which runs the fast lane (-m "not slow") five times and fails on any pass/fail disagreement between runs, to give that known flakiness a concrete signal instead of leaving it purely anecdotal.
Automation / E2E Covered indirectly: tests/streaming/*.mjs (Node-backed streaming/markdown invariant tests) and the many *_js.py tests that exercise static JS via a JS harness are the closest thing to browser-level behavior tests. There is no hermetic full-browser E2E harness (e.g. Playwright) checked into CI; the project instead leans on the very large pytest suite for route/service-level coverage. Not added in this audit — out of scope for an audit pass, flagged as a possible future ROADMAP item.

Real numbers (this audit, 2026-07-08)

Run from a fresh pip install -r requirements.txt on Windows / Python 3.13.11 (the repo's own CI pins 3.11; this audit ran on the box's installed 3.13 since that is what was available - see caveat below):

python -m pytest -q --cov=. --cov-report=term-missing --cov-report=xml

Result: 115 failed, 4353 passed, 20 skipped, 21 errors in 12m52s; coverage 65% total (100,693 statements, 35,361 missed - measured over the whole tree including tests/ itself, so treat it as an overall floor, not a src/-only figure).

Caveat on the failures: this was a Windows run and the failure profile is dominated by Windows-environment issues, not logic regressions. 70 of the 115 failures are *_js.py harness tests all failing with Node's ERR_UNSUPPORTED_ESM_URL_SCHEME (importing a bare C:/... path as an ES module without a file:// scheme - Node 24 on Windows rejects this), and all 21 errors are Windows FileNotFoundError/path issues in tests/test_code_nav_tools.py and tests/test_signature_route_hardening.py. Most of the remainder cluster in Windows-path-sensitive files (test_tool_path_confinement.py, test_shell_routes.py, test_agent_migration_manifest.py). This matches the repo's own disclosure: CI's python-tests job (ubuntu, Python 3.11) is continue-on-error: true because "the suite has known flaky / environment-dependent failures", and the ROADMAP tracks Windows under "fresh install smoke tests". The CI job output is the authoritative number for the pinned environment.

Known, pre-existing gaps (not introduced by this audit)

  • python-tests in CI is continue-on-error: true - it is informational, not a blocking gate, until the flaky subset is fixed (ROADMAP: "fresh install smoke tests").
  • No coverage floor is enforced in CI.
  • No mutation testing.