Skip to content

WorkWingman — Testing (Technical)

How WorkWingman proves its behavior today, and the disciplined testing program it is building toward. This document is deliberately honest about the line between what exists now and what is planned (labelled Roadmap — do not treat roadmap items as working).

What exists now: backend and frontend unit/integration test counts grow with every merged feature (xUnit + NSubstitute + Bogus; Flurl HttpTest for the REST clients on the backend; vitest + @faker-js/faker + HttpTestingController; a spec per screen and per service), 6 Playwright.NET UI E2E journeys (tests/WorkWingman.E2E) driving the built Angular app against a stubbed API, an Electron shell smoke E2E (tests/electron-e2e), a determinism/churn CI job (each suite re-run ×5), and mutation testing on both backend (Stryker.NET) and frontend (StrykerJS), scheduled weekly. Still roadmap: 100% coverage and broader E2E journeys.

Testing philosophy

The rules every test in this repo is held to:

  • Test behavior, not implementation. A test asserts on observable outcomes (a returned value, a status code, a thrown exception, rendered DOM) — never on private internals. You should be able to refactor the implementation without touching the test.
  • Deterministic, always. A test either passes every time or it is a bug. We do not retry-mask, sleep-and-hope, or depend on wall-clock time, network reachability, or execution order. Flakiness is treated as a defect in the test, not a nuisance to paper over.
  • Mock the edges, exercise the core. External systems (HTTP APIs, browsers, the OS) are simulated so the logic under test is the only variable. In-memory hosting (WebApplicationFactory) runs the real API pipeline without a socket, Kestrel, or a browser.
  • Fast enough to run on every change. The whole suite is designed to run in CI on every push and pull request (see CI).

Tooling in use today

Tool Role Docs
xUnit Backend test framework ([Fact], [Theory], [InlineData]) xunit.net · GitHub
Microsoft.NET.Test.Sdk Test host / VSTest runner integration GitHub
coverlet Cross-platform code-coverage collector (XPlat code coverage in test.runsettings) GitHub
WebApplicationFactory<Program> In-memory ASP.NET Core integration host (no Kestrel, no browser) learn.microsoft.com
Flurl.Http testable HTTP Fluent HTTP client whose calls are simulated in ResiliencePipelineTests flurl.dev testable-http · GitHub
vitest Frontend test runner (Angular 22 uses it via @angular/build) vitest.dev · GitHub
Angular TestBed Component/service test harness angular.dev/guide/testing

The test pyramid

flowchart TD
    U["Unit tests<br/>(pure logic: parsing, classification, vault rules)"]
    I["Integration tests<br/>(WebApplicationFactory — real API pipeline, in-memory)"]
    E["End-to-end tests<br/>(Roadmap: Playwright.NET journeys, Electron shell)"]
    M["Mutation tests<br/>(Stryker.NET + StrykerJS — do the tests actually catch bugs?)"]

    U --> I --> E --> M

Most tests are cheap, fast unit tests at the base. A thin band of integration tests exercise the wired-up API. E2E and mutation testing sit at the top; both are now wired (mutation runs weekly on a schedule).

What exists today — backend (xUnit tests)

Location: tests/WorkWingman.Tests (targets net10.0).

Test file What it proves
ResiliencePipelineTests The Polly transient-fault predicate classifies 429, 500, 503, and network failures (no response) as retryable, and 400 / 401 / 404 as non-retryable. A separate test drives the default pipeline through two transient failures and asserts it retries then succeeds on the third attempt (result 42, exactly 3 attempts).
AtsDetectorTests ATS detection from a job description page: Workday is detected and its tenant host is extracted (cvshealth.wd1.myworkdayjobs.com); Greenhouse, Lever, and iCIMS are each detected from their host markers; unmarked HTML resolves to Unknown.
VaultServiceTests A locked vault refuses access — listing entries or resolving a secret throws InvalidOperationException. After unlock, CreateAtsEntry names the entry from the tenant (workday-acme), files it under the Ats category, and stores a resolvable secret. Resolving an unknown entry throws KeyNotFoundException.
ApiSmokeTests In-memory API via WebApplicationFactory<Program>: seven GET endpoints return 200 (jobs, connections, settings, applications, intake, story, vault status); the locked-vault entries endpoint does not return 200 (never leaks secrets); the dependency check (GET) is open; install is rejected (401) without a token or trusted origin (hostile-page simulation); and install passes the auth gate from the trusted dev origin http://localhost:4200 (then 400 on empty id list).
DependencyDoctorTests + VersionParsingTests + DependencyManifestTests Version extraction pulls the first semver out of noisy --version output (git version 2.49.0.windows.1 → 2.49.0); comparison enforces minimums; the manifest is valid (every dependency has an id, purpose, min-version, install metadata, and an https:// info URL); the doctor returns a status for every manifest entry (never Unknown); installing an unknown id fails gracefully without throwing; and optional gaps do not block launch while a missing required dependency does.
LinkedInPostingSignalExtractorTests + SelectBySelectorLikeClassTests + CollectCandidateTextTests Repost-count/original-date/current-age parsing from fixture LinkedIn job-page HTML (see job-signals.md): repost count with and without an explicit number, original date in both rendered formats, case-insensitivity, and graceful degradation to null/unavailable on empty, malformed, or classless markup — no real network, no live LinkedIn page.
WarnNoticeSourceTests + SecEdgarSourceTests + LayoffsFyiSourceTests + CompanyLayoffServiceTests Each layoff source adapter's fetch/normalize logic against Flurl HttpTest stubs (Texas WARN Socrata feed, SEC EDGAR full-text search, and the intentionally-no-op layoffs.fyi adapter — see job-signals.md for why); CompanyLayoffService merges across sources, dedupes same-company/same-date events keeping the highest-priority source, and returns results newest-first, degrading gracefully when a source is empty or unreachable.
IncomeTaxEstimatorTests Exhaustive federal/state/local bracket boundary math (at-cap, one-dollar-over, no-tax states, flat states, progressive states), FICA wage-base and Additional-Medicare-threshold boundaries, local-tax city/state matching, and guard clauses, against a synthetic fixture dataset — plus sanity checks that the real default 2026 dataset loads all 51 state entries and behaves correctly for known cases (see col-tax.md).
CostOfLivingServiceTests + CostOfLivingSourceTests Per-field merge across mocked ICostOfLivingSources, unconfigured sources skipped and never called, first-configured-source-wins on a field conflict, and graceful degradation when a source throws — plus config-gating (IsConfigured) for EiaGasSource, BlsCpiSource, and the paid, config-stubbed NumbeoSource, none of which ever reach the network when unconfigured.
CityComparisonServiceTests Raw take-home delta math, cost-of-living-adjusted delta (full data, partial data, one side missing entirely → null, equal indexes → adjusted equals raw), and resilience when the cost-of-living lookup throws.
ConceptVisualPromptBuilderTests + StudyVisualGeneratorTests + MagnificSourceTests + StudyVisualsControllerTests + StudyVisualServiceTests See study-visuals.md: the prompt builder is deterministic and never references a copyrighted work (regression-guarded against a forbidden-terms list); GeminiImageSource/OpenAiImageSource never call their API without a configured key (asserted via an empty Flurl HttpTest call log), hit the exact verified endpoint/verb/header, and degrade gracefully (never throw) on an empty/malformed response or an HTTP error; MagnificSource drives its async create-then-poll upscale flow to completion, failure, and a bounded timeout with no hang; the image-serving endpoint never serves a path outside its own directory; StudyVisualService orchestrates generate/regenerate/delete/enhance against a real temp-directory LocalJsonStore, scoping each gallery to its study item and falling back across enhancers. Stryker.NET mutation score: 92.75% on **/Visuals/*.cs.

What exists today — frontend (2 vitest tests)

Location: frontend/src/app/app.spec.ts. Angular 22 runs these through vitest via @angular/build.

Test What it proves
should create the app The root App standalone component instantiates under TestBed with router + HTTP providers.
should render the sidebar brand After the fixture stabilizes, the sidebar .brand-name renders and contains "Work Wingman".

How to run the tests

Backend (Windows, .NET 10):

# from the repo root — the unit/integration suite CI gates on
dotnet test tests/WorkWingman.Tests/WorkWingman.Tests.csproj --settings test.runsettings

The ScraperLab tier: compiled always, run on request

The 13 WorkWingman.ScraperLab.*.Tests projects are in WorkWingman.slnx and stay there, so every one of them compiles on every build — including CI's. That is the point: a lab that stops compiling has to break something. It used to break nothing, and WorkWingman.ScraperLab.Google.Tests sat with 20 compile errors while dotnet build WorkWingman.slnx reported 0 Error(s), because the solution had never heard of it. SolutionMembershipTests now fails if any .csproj drifts out of the solution again.

Membership is not the run-selection lever. Compilation is exhaustive; execution is selective. Each lab test assembly carries [assembly: AssemblyTrait("Category", "Lab")], so:

# fast loop — everything except the lab tier
dotnet test WorkWingman.slnx --filter Category!=Lab

# full loop — labs included
dotnet test WorkWingman.slnx

Lab tests drive a real headless Chromium against offline loopback fixtures. Browser-launching test classes take a context from SharedBrowser rather than launching their own — one browser per test assembly, not one per test. It lives in WorkWingman.TestSupport because assemblies are what parallelise, so a helper only one assembly can reach caps only that one.

A bare dotnet test (or dotnet test WorkWingman.slnx) also picks up WorkWingman.E2E, which serves the built Angular app and fails fast with DirectoryNotFoundException: Angular build output not found until you have run npm run build in frontend/. That red is a missing prerequisite, not a broken test — the UI E2E is a separate CI job that builds the frontend first. To run it locally, follow tests/WorkWingman.E2E/README.md.

The suite runs at xunit's default parallel width. It used to cap at 0.25x of the machine's cores (tests/WorkWingman.Tests/xunit.runner.json, since deleted): the apply-engine tests each launched their own headless Chromium, so at full width the browsers oversubscribed the box and the wall-clock waits inside them — and inside the engines under test — expired, producing timeouts in a rotating cast of tests that all passed in isolation. The cap read as "greener and faster," which is the tell that it was never a CPU cap: a thread cap relieving CPU contention does not speed a suite up. It was rate-limiting browser spawns. Once SharedBrowser bound each assembly to one browser (WW-93/WW-92), the storm was gone and the cap became pure cost — measured, interleaved, capped vs uncapped on the same tree: uncapped ~74s vs capped ~119s, both 3654/3654 green, uncapped green even at 1.6 GB free. So it was deleted (WW-95); the real fix was bounding browsers, not threads.

Coverage is collected by coverlet via the XPlat code coverage data collector declared in test.runsettings.

Frontend (Angular, vitest):

cd frontend
npm test

Continuous integration

.github/workflows/ci.yml runs on every push and pull_request to master / main:

  • Backend job (windows-latest): installs the .NET 10 SDK, builds Release, then runs dotnet test --no-build --settings test.runsettings.
  • Frontend job (ubuntu-latest): installs Node 24, npm ci, npm run build, then npm test.
  • Security (dependencies) job (ubuntu-latest): dotnet list package --vulnerable --include-transitive and npm audit --audit-level=high gate every push/PR on known-vulnerable packages — see security.md for the exact parse logic and the allowlist.

All jobs must be green to merge.

Shipped (was roadmap)

Backend coverage with mocked dependencies

Services, clients, and pipelines are driven with substituted collaborators: NSubstitute for Core interfaces, Bogus for seed-controlled fake data (TestData.cs), and Flurl HttpTest to stub the REST clients (asserting URL/verb/bearer with no live network). Learn more: mocking lets you assert on how a unit uses its collaborators (test doubles).

Frontend unit coverage

A spec per component and service across all 10 screens plus the services, using @faker-js/faker view-model builders and the mockApi() helper (spec-support.ts), with HttpTestingController for the API service.

UI E2E in C# (Playwright for .NET)

tests/WorkWingman.E2E drives the built Angular app against a stubbed API (Playwright route interception, CORS-hardened) via a loopback static server, using the Page Object Model. Journeys: shell + navigation, theme persistence across routes, the job queue, the judgement-call pause→resolve flow, and connections. Runs as its own CI job. Selenium remains the documented fallback only.

Electron shell E2E (TypeScript)

Playwright's .NET binding cannot launch Electron, so the shell smoke suite (tests/electron-e2e) is TypeScript (_electron): it boots the packaged shell, confirms the renderer loads, and that it reaches the loopback API. Runs headless under xvfb in its own CI job.

Mutation testing — backend

Stryker.NET proves the tests catch bugs by injecting faults and checking a test fails. Config: stryker-config.json. It runs weekly (and on demand) via mutation.yml, uploading the HTML report — mutation is too slow for per-push.

Scope decision (documented survivors): mutation targets the pure-logic files the unit tests cover — Services, Clients, Http, Vault, DependencyManifest, VersionParsing, AtsDetector. It excludes the I/O-bound classes — DependencyDoctor and ProcessRunner (winget/npm/Playwright subprocess paths) and the Playwright browser automation (BrowserSession, LinkedInJobsScraper, WorkdayAutomationEngine). Those spawn real processes/browsers and belong to integration tests, not unit mutation — mutating them only produces no-coverage/timeout noise. additional-timeout is raised so slow-but-real Polly/async mutants aren't mis-scored as timeouts.

Mutation testing — frontend

StrykerJS does the same for the Angular frontend via its vitest-runner. Config: frontend/stryker.conf.json. Like the backend it runs weekly (and on demand) via mutation.yml.

The Angular-22 blocker and the fix. Angular 22 runs unit tests through the @angular/build:unit-test builder, which owns the vitest environment and injects a virtual TestBed-bootstrap module — so ng test works, but a standalone vitest run (which is what StrykerJS's vitest-runner drives) has no Angular bootstrap and fails to load. The fix is a standalone Angular vitest config that reproduces what the builder provides:

  • frontend/vitest.config.ts — a plain vitest config that compiles Angular sources with @analogjs/vite-plugin-angular, runs in jsdom, and loads the setup file below. This is additive: npm test still uses the Angular builder (ng test); only Stryker and any bare npx vitest run use this config.
  • frontend/src/test-setup.ts — initializes the Angular TestBed (BrowserTestingModule + platformBrowserTesting), mirroring the builder's generated bootstrap. The app is zoneless, so it deliberately does not import zone.js or add provideZoneChangeDetection() — the TestBed uses default zoneless change detection, exactly as under ng test. All 15 existing spec files pass unchanged under npx vitest run.

Config notes. coverageAnalysis: "perTest" with vitest test isolation on (isolate: true) is what lets Stryker attribute mutant coverage correctly across every mutated file; each isolated worker builds its own TestBed once (guarded by an ANGULAR_TESTBED_SETUP symbol). vitest.related is set to false so Stryker runs the full spec set rather than relying on vitest's module-graph "related files" resolution, which the Analog compile step does not expose.

Scope decision (documented survivors): mutate targets src/app/**/*.ts minus the specs, the testing/ faker fixtures, and the pure-wiring files (main.ts, app.config.ts, app.routes.ts) — mutating bootstrap/route tables only produces no-coverage noise. The pure-logic services score well (e.g. theme.service ~91%); feature components carry many no-coverage mutants because their specs assert rendered output and service interactions rather than every internal branch — a known gap tracked under coverage on the Roadmap, not a runner failure.

Determinism / churn testing

The Determinism (churn) CI job re-runs both suites ×5 and fails if any result flips — a flaky test is a bug, never retry-masked (test independence). Bogus and faker data are seeded for reproducibility.

Roadmap (planned)

  • 100% coverage targets — expand backend + frontend coverage to every branch, including the I/O classes via integration tests (real ProcessRunner/doctor against a sandbox). On the frontend this also means killing the component no-coverage mutants that StrykerJS surfaces (see Mutation testing — frontend) by asserting more internal branches, not just rendered output.
  • Broader E2E journeys — extend the Playwright.NET suite across the full flow (onboarding steps → story → docs → approve → applied → settings) with a Page Object per screen.
  • Randomized test order — add xUnit/vitest order randomization on top of churn to surface hidden inter-test coupling.