WING-208 — Cloud User-Testing v3: Telemetry + Surveys Spec¶
Goal: understand why external testers behave as they do — not raw engagement counts. Every metric here is "usage, not value" (Kerr constraint). Builds on existing MetricsEventLog (monthly shards via LocalJsonStore), POST /api/metrics/events (MetricsController.AppendEvent, ClientWritable-type allowlist, size-capped), and OnboardingEventsService funnel pattern.
Event taxonomy (all client-writable, additive to existing allowlist)¶
Common envelope fields on every event: type, tsUtc, sessionId (random per app launch), page (route path), uiMode (simple|power), payload (type-specific, small, NO free-text user content, NO PII, NO job titles/company names — ids only).
| type | payload | fired when |
|---|---|---|
ui_interaction |
{ control, kind, value? } — kind ∈ button/toggle/radio/checkbox/filter/select/tab/link; control = stable data-attr id; value = chosen option for toggles/radios/filters (enum values only, never free text) |
any instrumented control activated |
page_time |
{ route, activeMs, idleMs } — idle = no pointer/key/scroll for 30s |
route leave (and app blur/close flush) |
mode_switch |
{ from, to, onRoute } |
UiModeService.set/toggle |
mode_time |
{ mode, activeMs, idleMs } |
on switch + session end flush |
onboarding_source |
{ source } ∈ linkedin/indeed/usajobs, { action } ∈ connected/scraped/skipped |
connections + first scrape |
live_run_start |
{ entry } ∈ recommended/saved, { runId } |
startRun call site |
live_run_end |
{ runId, outcome } ∈ completed/paused_abandoned/stopped/error |
run terminal state |
survey_answer |
{ surveyId, trigger, answer } — answer = option id or 1–5 scale |
survey submitted |
survey_skipped |
{ surveyId, trigger } |
survey dismissed — skip IS data |
Server-emitted apply-engine events (NOT client-writable — engine emits directly to MetricsEventLog)¶
| type | payload | fired when |
|---|---|---|
apply_step_mode |
{ runId*, ats, step, mode } — mode ∈ parser/page_advisor_ai/copy_paste_manual |
each apply step/page resolves — which automation tier actually handled it |
apply_escalation |
{ runId*, ats, step, from, to, reason } — from/to = same mode enum; reason ∈ unknown_control/low_confidence/advisor_error/engine_error/user_requested |
tier hand-off (parser→AI, AI→copy-paste, etc.) |
review_diff_opened |
{ runId*, changedFieldCount } |
user opens the pre-submit review diff |
review_back |
{ runId*, fieldsEdited } |
user goes back from review to change answers (each round trip) |
review_submit |
{ runId*, edited, editRounds, editedFieldCount } |
final submit — edited=false means submitted as-is (trust signal) |
* = entityIds. Counts only — never field names/values (a field NAME can leak the question, a VALUE can leak the answer; neither ships). Predeclared assumptions added: parser+Playwright suffices for most steps (if page_advisor_ai handles >50% of steps on a Big-4 ATS, parser coverage is the roadmap gap); users mostly submit as-is (if >60% of submits have edited=true, draft quality isn't earning trust → drafting iteration before scale).
Frontend architecture¶
TelemetryService(core, providedIn root): buffers events, batches POST to/api/metrics/eventsevery 15s or 25 events, flush onvisibilitychange/unload via sendBeacon-style call. Failure = silent drop (telemetry must never break the app).- Interaction capture: single document-level delegated listener keyed off
data-trackattributes (data-track="queue.filter.remote") — NOT per-component wiring. Components opt in by adding the attribute. Instrument: mode toggle, queue filters, compare controls, live-run controls (pause/resume/stop/judgement), onboarding hub cards, settings toggles, recommended/saved job actions. PageTimeTracker: Router events + idle detector (30s no input) + visibility API.ModeTimeTracker: subscribes UiModeService + same idle detector.
Surveys (moment-triggered micro-prompts)¶
- One question per prompt, bottom-sheet/toast style, always skippable, skip recorded.
- Frequency guard: max 1 prompt per 10-minute segment, max 4 per session, each surveyId asked at most once per user (persistedSignal).
- Triggers v1: after first mode switch ("Why did you switch?" — 4 option chips); after live-run abandon ("What made you stop?"); after first completed run (1–5 "How much effort did this save vs applying yourself?"); after onboarding complete ("Which job source do you expect to use most?"); on 3rd session start ("What's the one thing you came back for?").
- Answers are fixed option chips or 1–5 scale — no free-text in v1 (keeps events PII-free and analyzable).
Backend¶
- Extend
MetricsControllerClientWritable allowlist with the 9 types above; validate payload size + enum values server-side. - New rollup in
ValueMetricsService(or siblingUsageInsightsService): per-day aggregates — page time by route (active only), mode split, source usage, run entry split, survey response table. Exposed atGET /api/metrics/usage-insights(token-gated).
Kerr guardrails (non-negotiable)¶
- Durations always split active/idle; UI labels say "usage, not value".
- No dashboard/rollup ranks features by raw click count; interaction counts always shown beside a task-outcome column (runs completed, applications submitted).
- Survey completion rate is NOT a target metric; skips are first-class data.
- Tests assert taxonomy shape, allowlist enforcement, frequency guards, PII-absence — never target numbers.
Predeclared assumptions (venture-evidence — declare BEFORE tester sessions)¶
| Assumption | Metric | Threshold / decision |
|---|---|---|
| Testers prefer Flow (simple) mode for real work | mode_time active split | If ≥60% active time in Studio, simplify-first roadmap is wrong → revisit |
| LinkedIn is the dominant onboarding source | onboarding_source | If Indeed+USAJobs > LinkedIn connections, reprioritize source polish |
| Recommended jobs drive live runs more than saved | live_run_start entry split | If saved ≥ recommended, recommendation ranking isn't earning trust → iterate ranking |
| Live runs mostly complete once started | live_run_end outcomes | abandon rate >40% → run UX/reliability blocker before scaling testers |
| Surveys at moments get answered | survey answer vs skip per trigger | skip >75% on a trigger → that trigger is mistimed; remove it, don't nag harder |
No invented traction: report observed numbers with denominators; confounders (small N, invited testers, demo sessions) stated alongside.
Multi-tenant caveat (cloud)¶
Legacy app stores are process-wide singletons on cloud (WING-202 handoff): MetricsEventLog shards co-mingle all tenants. Envelope MUST carry tenantId/workspace id when the server can resolve it (server-side stamp in MetricsController, never client-asserted) so analysis can segment. usage-insights stays operator-only (token + UserTesting flag); never surface cross-tenant aggregates to testers.
Cloud edition¶
Same code ships everywhere; cloud user-testing build is config: WorkWingman:UserTesting=true enables survey triggers + insights endpoint (telemetry capture itself ships everywhere but stays local; only user-testing builds are expected to have their event log collected). Branch off internal-testing build if divergence needed.