Metrics and telemetry (technical)

Canonical technical map of what WorkWingman measures, where events are collected, where they are stored, and the privacy stance encoded in code and specs. Primary sources: repo-root METRICS.md, docs/technical/value-metrics.md, docs/testing/user-testing-telemetry-spec.md, docs/technical/byok-efficiency.md, and the implementations under src/ and frontend/.

Scope note: This is a measurement inventory. It does not invent production retention SLAs, commercial analytics products, or venture outcomes that are not declared elsewhere.

1. Systems overview

WorkWingman has several separate measurement lanes. They share honesty/Kerr rules in places but must not be collapsed into one stream.

Lane	Purpose	Primary types / API	Storage
Value Metrics	Job-hunt facts + dashboard rollup (“what the app did for the hunt”)	MetricsEvent / MetricsEventTypes; GET /api/metrics/summary	Local monthly shards metrics-events-{yyyy-MM} via MetricsEventLog / LocalJsonStore
WING-208 UX telemetry + surveys	Usage-not-value instrumentation for user testing; micro-surveys	Client types via TelemetryService + SurveyService → POST /api/metrics/events; server apply-engine types append to same log	Same metrics-events-* locally; optional stdout → Cloud Logging → BigQuery when WorkWingman:UserTesting=true
AI usage metering (WW-66 / TROI)	Content-free meter of every ILlmHarness call	AiUsageEvent via MeteringLlmHarness → IAiUsageRecorder	Separate shards ai-usage-{yyyy-MM} via AiUsageLedger
LLM routing decisions	Router decision log (TROI-related)	RoutingDecisionLog	Separate routing-decisions-* shards (isolated from both above)
Local analytics engine	Funnel / posting / salary-style aggregates over domain data	IAnalyticsEngine / AnalyticsController	Derived from job/application stores — not the event log
Onboarding local event log	First-run funnel milestones	OnboardingEventsService	Browser localStorage only — no network path
Venture evidence	Falsifiable business claims	Markdown ledger	docs/business/venture-evidence-ledger.md (in-app VentureEvidence aggregate not implemented)

No Application Insights, Mixpanel, Segment, PostHog, or AppCenter product SDK integration was found in app code. OpenTelemetry appears only as a transitive npm lock entry — not application instrumentation. contracts/ has no dedicated telemetry schema (feature-flag defaults only).

2. What is measured

2.1 Value Metrics event taxonomy

Defined in src/WorkWingman.Core/Models/ValueMetrics.cs (MetricsEventTypes) and documented in docs/technical/value-metrics.md.

Server-emitted (not client-writable):

Type	Emitter (documented / code)	Typical props / notes
scrape.session.completed	JobQueueService	scrapedCount, addedCount
fit.analyzed	JobQueueService	ats, fitLevel, matchedCount, gapCount, adjacentCount
apply.session.started	ApplicationRunService	ats, company, title, mode
apply.session.parked / apply.session.resumed	Apply driver at resume gate	Brackets user-turn pauses for active-time math
apply.session.readyForReview	Driver after drive	rawMs, outcome, fieldsFilled, resumeAttached
apply.session.failed	Driver	rawMs, outcome=failed
apply.submitted	ApplicationTrackerService	ats, company, title — user confirmation, app never clicks Submit
application.status.changed	ApplicationTrackerService	status
offer.created / offer.decision.recorded	OfferService	Decision journal facts (event happened; not “good decision” scoring)
apply_step_mode	Apply engine (server-only)	runId*, ats, step, mode ∈ parser / page_advisor_ai / copy_paste_manual
apply_escalation	Apply engine (server-only)	runId*, ats, step, from, to, reason enum
document.generated / document.adopted	Reserved for drafter / TROI WW-71	Stable names; durable-outcome wiring is later work

Client-writable allowlist (MetricsEventTypes.ClientWritable):

• Study: study.linkOpened, study.session.started, study.session.stopped, study.selfReport
• Baselines: baseline.manualTimed
• Painted-door: rewards.compareOffers.clicked
• WING-208 UX: ui_interaction, page_time, mode_switch, mode_time, onboarding_source, live_run_start, live_run_end, survey_answer, survey_skipped
• Review-flow: review_diff_opened, review_back, review_submit

MetricsEvent shape: Id, Ts, Type, EntityIds, Props (string primitives), SchemaVersion. Events are immutable facts; derived values (e.g. time saved) are never stored.

2.2 Value Metrics rollup (dashboard)

ValueMetricsService recomputes ValueMetricsSummary on demand against metrics-baselines.json (per-ATS ranges). Headline rules (Kerr audit, value-metrics.md):

• Qualified application = user-confirmed submit whose fit analysis was Good / TopApplicant.
• Hero counts and savings use qualified gating; gross savings only in methodology context.
• Active minutes = drive wall-clock minus parked intervals.
• Savings: max(0, baseline − activeMinutes) as a low–high range; low is the headline.
• Study: clicks = intent only; timer and self-report are separate series; self-report excluded from marketing-facing aggregates by design.
• UI: /metrics (Angular features/metrics).

API: GET /api/metrics/summary, GET/POST /api/metrics/baselines[/manual], POST /api/metrics/events.

2.3 WING-208 UX telemetry

Spec: docs/testing/user-testing-telemetry-spec.md. Implementation: frontend/src/app/core/telemetry/telemetry.service.ts, time trackers, SurveyService, server validation in MetricsController.

Common envelope (client): type, tsUtc, sessionId (random per launch, not persisted), page (route), uiMode (simple|power), payload — enums/ids/numbers only; no free-text user content, no PII, no job titles/company names.

type	payload (summary)	when
ui_interaction	control, kind, optional enum value	Instrumented control (data-track)
page_time	route, activeMs, idleMs (idle = 30s no input)	Route leave / blur flush
mode_switch	from, to, onRoute	UiModeService set/toggle
mode_time	mode, activeMs, idleMs	Switch + session end
onboarding_source	source ∈ linkedin/indeed/usajobs; action ∈ connected/scraped/skipped	Connections / first scrape
live_run_start	entry ∈ recommended/saved; runId entity	startRun
live_run_end	outcome ∈ completed/paused_abandoned/stopped/error	Terminal run state
survey_answer / survey_skipped	surveyId, trigger, answer (option id or 1–5)	Survey lane
review_*	counts only + runId	Live-run review surface (diff UI may still be partial)

Surveys (frontend/src/app/core/survey.service.ts):

• One question, skippable; skip is first-class data.
• Frequency: max 1 prompt / 10 min segment, max 4 / session, each surveyId at most once per user (persisted).
• Triggers documented in the WING-208 spec (mode switch, abandon, completed run, onboarding complete, 3rd session).
• Armed when GET /api/metrics/user-testing-enabled reports enabled: true (WorkWingman:UserTesting).

Client pipeline: buffer → batch POST every 15s or 25 events; flush on visibilitychange / pagehide / unload (keepalive-style). Failure = silent drop. Document-level delegated data-track listener (not per-component wiring).

Server validation: rate limit 120 events/min/IP; size caps; WING-208 types get strict prop/entity key allowlists and enum checks; non-allowlisted types rejected.

2.4 AI usage metering (WW-66)

Canonical honesty record: METRICS.md.

• Every call through any ILlmHarness is wrapped by MeteringLlmHarness at DI registration (LlmHarnessRegistration.cs) — single door.
• AiUsageEvent fields: Id, Ts, Provider, Model, TaskClass (rejects Unknown), Status (Ok|Empty|Failed), LatencyMs, InputTokens/OutputTokens (nullable), SchemaVersion.
• Content-free by construction — no free-text property; reflection tests ban prompt/response/content/text/message/body property names (AiUsageEventContentFreeTests).
• Tokens: Anthropic / OpenAI / Ollama report real usage via AiCallContext; CLI harnesses report null tokens (null = not measured, never zero).
• Explicit non-claims: no etcUsd, no plan pricing table, no MEASURED-vs-MODELED savings in this slice (later WW-67+).

Hostname gate tests (ProviderHostnameGateTests) fail the build if provider host literals appear outside src/WorkWingman.Infrastructure/Llm/.

2.5 Onboarding events (local only)

First-run milestones (e.g. first_flight / step / hub / first_job_scraped patterns described in board/docs) live in localStorage via OnboardingEventsService — zero telemetry upload for that log. Distinct from WING-208 onboarding_source events on the metrics pipeline.

2.6 Local AnalyticsEngine

AnalyticsController / AnalyticsEngine compute funnel/trend/salary-style views from job queue + application tracker domain data. Not third-party analytics; not the MetricsEvent stream.

2.7 Lab metrics (non-product)

Scraper lab tooling (LabMetrics, ATS labs docs) tracks selector hit rates for engine research under tools/ — not end-user product telemetry.

3. Collection points

Layer	Components
Angular client	TelemetryService, PageTimeTracker / ModeTimeTracker, SurveyService, study/equity/live-run/queue/simple-home appendMetricsEvent / track call sites
Local API	Default client base http://127.0.0.1:5211; MetricsController under api/metrics
Server services	Apply/scrape/tracker/offer (+ engine escalations) → IMetricsEventLog
LLM path	MeteringLlmHarness → IAiUsageRecorder / IAiUsageReader (plans/capacity UI reads usage)
Config gate	IAppEnvironment.UserTesting from WorkWingman:UserTesting (default false in registration)
Cloud mirror	EmitTelemetryLogLine only when UserTesting — single-line JSON stdout log_type=ww_telemetry
Marketing site	UserTesting notice page content — not the app event pipeline

Operator rollup: GET /api/metrics/usage-insights — token-gated (RequireLocalToken) and 404 unless UserTesting.

4. Storage and sinks

Destination	Contents	When
LocalJsonStore collections metrics-events-{yyyy-MM}	Value Metrics + accepted client telemetry events	Always (local append-only; never-throws contract)
LocalJsonStore ai-usage-{yyyy-MM}	AI usage ledger	Always, isolated from metrics-events
LocalJsonStore routing-decisions-*	Router decisions	When routing path records them
metrics-baselines store	Per-ATS baseline ranges + manual samples	Editable via API
Browser storage	Onboarding events; survey asked/session counters; session run-entry map	Client-side only
Cloud Logging → BQ dataset nifty-condition-503506-j1.ww_telemetry	Mirrored client-accepted events	User-testing cloud builds only
SQL views	docs/testing/bq-dashboard-views.sql (v_events, v_events_clean, prod filters, QA tenant exclusion)	Operator analytics
Looker Studio	Report linked from docs/BOARD.md for WING-208 (Overview / Modes & Pages / Apply Automation / Reviews & Surveys)	Post-sink dashboard

Append contracts: both MetricsEventLog and AiUsageLedger swallow storage failures so metering never breaks the feature that produced the event.

5. Privacy stance

Documented and/or enforced:

1. Local-first default for Value Metrics: dashboard computes on-device; design doc states nothing leaves the machine for that rollup path.
2. Telemetry capture ships in client code but stays local unless a user-testing build enables cloud mirror (UserTesting=true); only those builds are expected to have event logs collected off-box (user-testing-telemetry-spec.md).
3. Surveys and usage-insights require the UserTesting flag; insights are operator-only.
4. No free-text / PII in WING-208 envelopes — enums, ids, counts; field names/values forbidden on review events; server key allowlists + enum validation.
5. Server-stamped tenantId / audience / environment on cloud mirror lines — never client-asserted (MetricsController.EmitTelemetryLogLine, tenancy middleware comments).
6. AI ledger cannot hold prompts/responses by type design + tests.
7. Separate shard families so different PII postures cannot co-mingle files (ai-usage-* vs metrics-events-*).
8. Kerr labels: usage ≠ value; survey completion rate is not a target; durations split active/idle.
9. Onboarding milestone log: no network.
10. Venture-evidence boundary: do not put interviews/hypotheses into MetricsEvent; telemetry may support Observed/Outcome only with explicit provenance (venture-evidence-ledger.md).

Documented tensions / unknowns

• docs/business/adaptive-solutions-pitch.md states candidate data is local-only, “No telemetry,” crash reports opt-in and redacted. That pitch language conflicts with shipped WING-208 telemetry code and cloud mirror for user-testing builds. Treat pitch text as stale or narrower marketing wording until reconciled — TODO(verify) intended public wording.
• Crash-report pipeline: not found in code search; pitch-only claim — TODO(verify).
• End-user opt-out toggle for local event capture: not found — TODO(verify).
• BigQuery / Cloud Logging retention TTL, access control, and whether Looker is limited to operators: partially specified by views; full policy — TODO(verify).
• Exact on-disk root path for LocalJsonStore data directories: convention-based — TODO(verify) document path layout per edition (desktop vs Cloud Run volume).
• Multi-tenant caveat: process-wide singleton stores on cloud can co-mingle shards; server stamp is the segmentation strategy until stronger tenancy isolation lands — TODO(verify) current production tenancy state.

6. Related docs

Doc	Role
METRICS.md	Shipped AI-usage honesty record (WW-66)
docs/technical/value-metrics.md	Value Metrics design + Kerr rules
docs/plain/value-metrics.md	Plain twin of Value Metrics
docs/technical/byok-efficiency.md / docs/plain/byok-efficiency.md	TROI nine-part design
docs/testing/user-testing-telemetry-spec.md	WING-208 taxonomy + predeclared assumptions
docs/testing/bq-dashboard-views.sql	BQ view definitions
docs/business/venture-evidence-ledger.md	Claims, thresholds, telemetry boundary
docs/discovery/ww-117-onboarding-concierge-v0.md	Predeclared onboarding experiment metrics
Token-ROI parent standard	Referenced as token-roi/docs/METRICS-STANDARD.md (path outside this repo) — TODO(verify) availability on every machine

7. Explicit non-claims

• No cost/dollar savings claims from AI metering until later TROI tickets land (METRICS.md §6).
• Telemetry does not establish customer demand, willingness to pay, or market prevalence by itself.
• Usage dashboards are not authorized to invent traction for decks without a ledger ID and predeclared thresholds.