Skip to content

WorkWingman — Audio Harness (Technical)

Provider-agnostic, BYO-key audio for WorkWingman: text-to-speech "audio overviews" of a job's study material, and optional AI music for study soundtracks / podcast stings. This implements the tiered recommendation in ../ai-audio-providers.md and honors every licensing flag in it.

Friendlier version: ../plain/audio-harness.md

Related design/research: study-audio.md (the broader ffmpeg/Piper studio plan), study-visuals.md (the visual analog this feature mirrors line-for-line).

Tiers (from the research doc)

Tier Speech (ISpeechHarness) Music (IMusicHarness) Key?
0 — keyless local DEFAULT Kokoro-82M (Apache-2.0, local ONNX; falls back to the offline OS voice until the model is fetched). Piper (GPL-3.0) low-end fallback. none — no keyless music by design (MusicGen weights are CC-BY-NC → unlicensable for shared output) ✗
1 — affordable BYO-key OpenAI gpt-4o-mini-tts, Amazon Polly Neural Stable Audio 2.5, Beatoven.ai ✓
2 — premium BYO-key ElevenLabs, Google Chirp 3 HD ElevenLabs Music v2, Google Lyria 3 ✓

Config-gating rule (identical to IStudyVisualGenerator): a cloud adapter whose key is absent reports IsAvailableAsync() == false and, if called anyway, returns AudioResult.Unavailable without ever hitting the network. Kokoro carries on with no key. Keys resolve through the same vault-first IApiKeyProvider; sandbox never falls back to appsettings/env keys.

Interfaces (WorkWingman.Core)

  • ISpeechHarness.SynthesizeAsync(text, SpeechOptions, ct) → AudioResult — bytes + format + a contained local file path + an AI-voice disclosure.
  • IMusicHarness.GenerateAsync(MusicRequest, ct) → AudioResult — Tier 1+ only.
  • IAudioOverviewService — composes the selected ILlmHarness to write a grounded script, then an ISpeechHarness to render it; persists a per-study-item gallery; serves everything path-free.
  • IStudyOverviewSource — gathers the job's existing study material (gap analysis, prep bank, flashcards) into the plain-text ground truth the script is written from.

ILlmHarness / ILlmHarnessSelector are a frozen contract owned by a parallel branch (feat/llm-harness). This branch ships a minimal copy of the interface file plus a thin ClaudeCliLlmHarness + LlmHarnessSelector so the feature works and compiles until merge.

Audio overview flow (no fabrication)

  1. IStudyOverviewSource.GetMaterialAsync(jobId) → plain text from the study plan + prep bank + flashcards. Empty material → the overview refuses ("nothing to narrate"), never invents filler.
  2. ILlmHarness.GenerateAsync writes the narration script. The system prompt binds it to the provided material only — "do not invent facts, company details, statistics, or technical claims not present in that material." (Anti-hallucination sits alongside the app-wide IStudyContentVerifier gate from study-audio.md.)
  3. ISpeechHarness.SynthesizeAsync(script, …) renders the voice.
  4. Persisted; a path-free AudioOverviewView is returned; audio bytes are served by id.

AI-voice disclosure (ToS)

OpenAI's TTS ToS requires telling listeners the voice is AI-generated. Rather than special-case OpenAI, every speech result carries AudioDisclosure.Voice and every music result carries AudioDisclosure.Music. The disclosure rides on AudioResult, is stored on AudioOverview, and is surfaced in the Study UI (under the <audio> player) and in the generation toast.

Kokoro — bundling / obtaining the model (no large binary committed)

The Kokoro ONNX weights (kokoro-v1.0.onnx + voices-v1.0.bin, ~300 MB) are not committed. They are fetched to %USERPROFILE%\Wingman\models\kokoro\ at first use (or bundled by the installer). Until they are present, KokoroSpeechHarness falls back to the keyless, offline OS voice (ILocalVoiceSynthesizer → Windows SAPI via System.Speech), so the Tier-0 default always emits real speech with no key and no download — exactly the research doc's "Windows SAPI — last-resort keyless" tier. Kokoro is the higher-quality upgrade that lights up once its model is on disk. IKokoroOnnxEngine is the (not-yet-registered) ONNX inference seam the Microsoft.ML.OnnxRuntime implementation plugs into when the bundling step ships.

Fetch step (documented): download the two files from https://huggingface.co/onnx-community/Kokoro-82M-v1.0-ONNX into the model directory above; the harness picks them up automatically on the next synthesis. Verify SHA before use.

Live verification (this machine, Windows)

LocalVoiceLiveTests exercises the real WindowsSapiVoiceSynthesizer + SystemProcessRunner (no mocking) and asserts a genuine spoken WAV (>4 KB) is produced. Verified live on the build host: a 272 KB spoken .wav was written to %USERPROFILE%\Wingman\data\audio\ through the Kokoro fallback path.

API (api/audio-overviews)

  • GET speech-providers / GET music-providers — availability per provider (open, drive the UI).
  • GET {studyItemId} — the gallery (path-free views).
  • GET {overviewId}/audio — streams bytes (containment-checked under the audio dir; never a file:// path).
  • POST generate, DELETE {overviewId} — app-token-gated (may call a paid cloud voice), same cross-origin protection as StudyVisuals.

Registry rows (ApiKeyProviderRegistry)

Nine BYO-key cloud rows added exactly per the research doc's section-C table: tts-openai, tts-polly (+tts-polly-secret for Polly's SigV4 secret half), tts-elevenlabs, tts-google, music-stableaudio, music-beatoven, music-elevenlabs, music-lyria. IsFree mirrors the doc ("free to obtain a key" vs usage-billed). Tier 0 (Kokoro/Piper) has no row — it needs no key.

Licensing safeguards honored

  • No keyless music. MusicGen's CC-BY-NC weights are never shipped; music is Tier-1-and-up BYO-key only. AudioRegistryTests.Registry_HasNoKeylessMusicRow asserts this.
  • Kokoro (Apache-2.0) is the bundled default, not Piper (GPL-3.0) or Coqui/XTTS (non-commercial weights, vendor defunct — never used).
  • OpenAI AI-voice disclosure surfaced uniformly.
  • No voice cloning — narration uses stock voices only; consent capture is out of scope.
  • Dependency/CVE hygiene: Polly uses a hand-rolled AWS SigV4 signer (PollySpeechHarness.SignV4, unit-tested) instead of the AWS SDK, avoiding the AWSSDK.Core advisory and keeping the surface small.

Tests

Backend (WorkWingman.Tests): cloud-adapter HTTP mocked (request shape, key-absent → unavailable + no call, graceful degradation, never fabricates); Kokoro local path (fake voice) + a live SAPI path; Polly SigV4 determinism; audio-overview composition (fake ILlmHarness + fake ISpeechHarness + fake source); controller app-token gate + path-free serving; registry rows. Frontend (study.spec.ts): voice picker, load/generate/delete, in-flight guard.