PageAgent as a real-time advisor (hybrid, one app)

Plain-language version: ../plain/pageagent-advisor-design.md

Supersedes any earlier "replace Playwright with PageAgent" framing on this branch. That framing was explored and abandoned: PageAgent's DOM-simplification model (browser-use-style indexed elements, aria-label/name/role by default — see Findings) is a plausible fill engine but reinvents, worse and more expensively, what the existing per-ATS Playwright selector chains already do reliably. The thesis that survives is narrower and stronger: use PageAgent (Claude-backed) as an advisor Playwright consults only at the exact moment its own automation engine already pauses for a judgement call.

TL;DR

• Playwright stays the driver, for all 11 ATS bases (Workday, ADP, Dayforce, Greenhouse, iCIMS, Lever, Oracle/Taleo/ORC, Paycom, Paycor, SuccessFactors, UKG). It navigates, resolves selectors, handles iframes/timing, and owns the submit gate. Nothing about that changes.
• PageAgent (Claude) becomes an advisor, consulted through one new interface, IPageAdvisor, at the single call site every engine already has: the moment it can't confidently auto-fill an ambiguous field and would otherwise pause and wait for the human.
• One interface, ATS-agnostic. IPageAdvisor.AdviseAsync(AdvisorRequest) takes a field label, the candidate's target value, the options actually on the page, and optional context — no selector, no automation-id, no ATS name. It returns ranked, explained interpretations (AdvisorHint) that slot directly into the existing JudgementCall.Options model.
• Advisor informs, never decides finality. The calling engine still applies its own confidence threshold to whatever the advisor says. Below threshold, the engine still raises JudgementCall and pauses for the user — exactly as it does today with no advisor at all.
• Model-agnostic too. IPageAdvisor has two implementations behind it — ClaudePageAdvisor and CodexPageAdvisor — sharing one prompt/parse contract (AdvisorPromptBuilder) and differing only in their injected IAdvisorTransport. The engine picks an implementation via DI/config and never branches on which one it got.
• Token cost is bounded by the pause rate, not the field count. Advisor calls track each ATS's already-measured judgement-call frequency (roughly: iCIMS 13/20 iterations, SuccessFactors ~60% on the degree field, Lever spiking to 18/20 in one specific run, Workday/ADP/Paycom/UKG "every iteration" on the degree case) — not every field on every page, which is what made the full-replacement framing cost 45-80 calls/application.

---

Why the pivot

The original ask ("prove PageAgent can replace the selector engine, filling a whole Workday application from one natural-language prompt") was built and run against a local Workday fixture in this same branch (tools/pageagent-lab). It worked, but the process surfaced three structural findings that argue against full replacement and for a narrower advisor role:

Findings from the full-fill prototype

1. PageAgent's default DOM view doesn't expose data-automation-id. Its DEFAULT_INCLUDE_ATTRIBUTES list (title, type, name, role, value, placeholder, aria-label, id, …) omits the exact attribute the production Workday selector chains (WorkdaySelectors.cs) are built around — data-automation-id had to be added explicitly via PageControllerConfig.includeAttributes, and even then, PageAgent's DOM simplifier truncates long attribute values (legalNameSection_firstName renders as legalNameSection_fir...). A real LLM tolerates this by inference; a selector engine cannot and does not need to — it already has the exact attribute value.
2. Container-vs-descendant automation-ids have no PageAgent analog. The degree combobox's data-automation-id="formField-degree" lives on the wrapping <div>, which PageAgent's simplifier never lists (only interactive leaf elements are indexed) — so PageAgent can only ever act on the inner button's own visible text ("Select Degree"), never the automation-id the Playwright chain leads with. This is a genuine capability gap, not a tuning problem.
3. The one place PageAgent unambiguously added value was the ambiguous-option judgement call (the degree dropdown with no exact match for "B.S. Game Development") — reading the rendered options and ranking them against the target value with an explained rationale. That is precisely the moment the existing Playwright engines already isolate into JudgementCall/RunStatus.AwaitingJudgement — because it is the one moment that is genuinely a language-understanding problem, not a DOM-traversal problem.

Put differently: DOM traversal, selector fallback chains, iframe descent, and hydration waits are things Playwright already does well and cheaply (deterministic, sub-second, zero LLM calls). Interpreting an ambiguous label against a tenant-curated option list, or a free-text tenant screening question with no fixed schema, is a language problem an LLM is well-suited for and Playwright fundamentally is not. The advisor role routes each kind of problem to the tool built for it.

The pause point already exists — and is already cross-ATS

A cross-repo research pass (see Per-ATS survey) confirmed the judgement-call pause exists in some form in all 11 ATS automation engines. The pattern isn't identical everywhere yet, but the underlying shape — a field label, a set of candidate options, and a confidence threshold — is the same everywhere, and the shared WorkWingman.Core.Models.JudgementCall/RankedOption type (already present, already commented "Claude's ranked guesses") is a proven fit: 4 of the 11 engines (Workday, iCIMS, Lever, SuccessFactors) already construct it directly today.

---

The IPageAdvisor seam

namespace WorkWingman.Core.Interfaces;

public interface IPageAdvisor
{
    Task<AdvisorHint?> AdviseAsync(AdvisorRequest request, CancellationToken ct = default);
}

public sealed class AdvisorRequest
{
    public string FieldLabel { get; set; } = string.Empty;
    public string TargetValue { get; set; } = string.Empty;
    public List<string> CandidateOptions { get; set; } = [];
    public string? PageContext { get; set; }
    public int ConfidenceThreshold { get; set; } = 90;
}

public sealed class AdvisorHint
{
    public List<RankedOption> RankedOptions { get; set; } = [];
    public string Reasoning { get; set; } = string.Empty;
    public RankedOption? TopChoice => RankedOptions.Count > 0 ? RankedOptions[0] : null;
}

(Full source: src/WorkWingman.Core/Interfaces/IPageAdvisor.cs.)

Deliberately excluded from AdvisorRequest: selectors, automation-ids, tenant/ATS name, frame references. The advisor only ever sees what a human reading the rendered page would see — a label, the candidate's intended value, and the options actually present. This is what makes the same interface usable from Workday's data-automation-id-keyed selectors, ADP's formcontrolname, iCIMS's inside-an-iframe id-suffix chain, or any future ATS, without the advisor code ever knowing which one it's talking to.

AdvisorHint.RankedOptions is shaped to drop straight into the existing JudgementCall.Options — no translation layer needed between "what the advisor said" and "what the pause-for-user UI already renders".

The shared call site

Every engine's pause today looks structurally like this (Workday's is the clearest, already using the shared model — WorkdayAutomationEngine.cs):

// BEFORE (today, all 11 engines, in one of three shapes — see Per-ATS survey):
if (confidence < threshold)
{
    run.PendingCall = new JudgementCall { FieldLabel = fieldLabel, Options = alternatives, ... };
    run.Status = RunStatus.AwaitingJudgement;
    return false; // pause, wait for the user
}

The seam is a single insertion before that existing check:

// AFTER (additive — the existing pause-for-user path is UNCHANGED and still the fallback):
if (confidence < threshold)
{
    var hint = await advisor.AdviseAsync(new AdvisorRequest
    {
        FieldLabel = fieldLabel,
        TargetValue = proposedValue,
        CandidateOptions = alternatives.Select(a => a.Label).ToList(),
        ConfidenceThreshold = threshold,
    }, ct);

    if (hint?.TopChoice is { Confidence: var advisedConfidence } top && advisedConfidence >= threshold)
    {
        // Advisor is confident enough — Playwright still executes the fill/select,
        // never the advisor. Log the advisor's reasoning for auditability.
        return await FillWithChoiceAsync(top.Label, advisorReasoning: hint.Reasoning);
    }

    // Advisor unavailable, or not confident either — fall through to the existing,
    // unmodified pause-for-user behavior. This is the ONLY safety net that matters;
    // the advisor is never allowed to be the last word.
    run.PendingCall = new JudgementCall { FieldLabel = fieldLabel, Options = hint?.RankedOptions ?? alternatives, ... };
    run.Status = RunStatus.AwaitingJudgement;
    return false;
}

Three invariants this preserves, all load-bearing:

1. Playwright still executes every fill. The advisor returns a ranked opinion; it never calls .FillAsync()/.ClickAsync() itself. advise() in the JS/TS prototype (tools/pageagent-lab/src/advisor/advise.ts) enforces this structurally, not just by convention: it constructs PageAgent with customTools: { click_element_by_index: null, input_text: null, select_dropdown_option: null, execute_javascript: null }, so a prompt-injected page cannot trick the advisor into acting even if it wanted to.
2. The threshold still gates everything. The advisor doesn't bypass RunStatus.AwaitingJudgement — it only gives the engine a better-informed answer to compare against the same threshold that already exists.
3. Never-auto-submit is untouched. The advisor is never consulted anywhere near the Review & Submit step; nothing about the submit gate changes in any of the 11 engines.

Per-ATS survey

A cross-repo pass over all 11 sibling lab repos (WorkWingman-scraper-lab-*) found the pause implemented three different ways today — meaning IPageAdvisor is the right abstraction to unify around, but wiring each engine into it is not zero-effort per engine:

ATS	Pause pattern today	Uses shared JudgementCall?	Selector strategy	Notable archetype
Workday	run.PendingCall = new JudgementCall {...} in TryFillFieldAsync	Yes	data-automation-id	Selector-first (container-vs-descendant automation-id)
iCIMS	run.PendingCall = new JudgementCall {...} in DecideDegree	Yes	iframe id-suffix (input[id$='FirstName'])	Two-level FrameLocator (outer #icims_content_iframe, nested inner frame); 13/20 iterations pause on the degree field
Lever	run.PendingCall = new JudgementCall {...}	Yes	name	18/20 pauses observed in one bug-discovery run
SuccessFactors	run.PendingCall = new JudgementCall {...}	Yes	id-substring (input[id*='firstName' i])	~60% of iterations pause on the degree field
ADP	AmbiguousFields.Add(...) + free-text log	No (ad hoc list)	formcontrolname	Degree judgement fires every iteration
Dayforce	AmbiguousFields.Add/MissedFields.Add + free-text log	No (ad hoc list)	data-testid	Timing/hydration: polls #hydration-host[aria-busy] via WaitForFunctionAsync before touching any field
Paycom	AmbiguousFields.Add(...) + free-text log	No (ad hoc list)	id then name	Degree judgement every iteration
UKG	AmbiguousFields.Add(...) + free-text log	No (ad hoc list)	id → name → aria → label → placeholder	Degree judgement every iteration
Oracle (Taleo/ORC)	free-text log only, no accumulator	No	Taleo=name, ORC=id	iframe switch rate 80% (Taleo) vs 0% (ORC)
Paycor	JSONL event log only, no accumulator	No	id then name	—
Greenhouse	own local GreenhouseJudgement record + OnJudgement callback	No (own type, not Core's)	#id	~1/3 of core-field resolutions fall back off #id

Implication: the interface itself (IPageAdvisor/AdvisorRequest/AdvisorHint) is provably one shape that fits every archetype — proven directly by driving the identical advise() call site against three deliberately different archetypes (Workday's selector problem, Dayforce's hydration-timing problem, iCIMS's nested-iframe problem; see Cross-archetype demo below). Wiring the other 7 engines to emit the shared JudgementCall model in the first place (instead of their current ad hoc AmbiguousFields lists or bare log lines) is separate, mechanical follow-up work per engine — not a design question, a migration task.

---

Model-agnostic advisor implementations

Per an explicit requirement to prove the seam isn't Claude-only, IPageAdvisor has two concrete implementations, both built on one shared, model-agnostic prompt/parse layer:

IPageAdvisor
├── ClaudePageAdvisor(IAdvisorTransport transport)
└── CodexPageAdvisor(IAdvisorTransport transport)
        │
        └── both call AdvisorPromptBuilder.BuildPrompt / .ParseResponse (identical logic)
                │
                └── IAdvisorTransport.SendAsync(prompt) -> raw text response
                        ├── (Claude) in production: in-page PageAgent inside the Electron
                        │   Live-Run BrowserView, driven over IPC — see below
                        └── (Codex) StubCodexAdvisorTransport — throws NotImplementedException;
                            a real transport (codex exec CLI, or an OpenAI-compatible endpoint)
                            is explicitly left for follow-up (tracked in parallel on the
                            research branch), but the DI seam and prompt contract are real today

AdvisorPromptBuilder (in WorkWingman.Infrastructure.Automation.Advisor) builds the exact same strict-JSON ranking prompt for both, and parses the response the same way — filtering out any option the model invented that wasn't actually on the page, clamping confidence to 0-100, and returning null (never throwing) on anything unparseable. Neither ClaudePageAdvisor nor CodexPageAdvisor contains model-specific prompt logic; the only thing that differs between them is which IAdvisorTransport they were constructed with.

tests/WorkWingman.Tests/PageAdvisorTests.cs proves this concretely: given the same IAdvisorTransport response, ClaudePageAdvisor and CodexPageAdvisor produce byte-identical AdvisorHints (top choice, confidence, ranked-option count) — a calling engine coded against IPageAdvisor literally cannot tell which one it got. This means WorkWingman can compare token cost and hint quality for Claude vs. Codex on the same fixture once a real Codex transport lands, by swapping one DI registration.

---

Cross-archetype demo (Workday, Dayforce, iCIMS)

To prove the call site is genuinely cross-cutting and not just "works on the fixture it was designed against", tools/pageagent-lab/src/harness/runAdvisorDemo.ts drives the identical adviseOnAmbiguousField() call against three purpose-built local fixtures, each reproducing one archetype confirmed in the survey above:

Fixture	Archetype	What's different from Workday
workday-fixture.html (existing, reused)	Selector problem	data-automation-id-based degree combobox, container-vs-button gotcha
dayforce-fixture.html (new)	Timing/hydration problem	#hydration-host[aria-busy] flips true→false ~1.2s after step navigation, mirroring DayforceAutomationEngine.WaitForHydrationAsync's poll
icims-fixture.html + icims-inner-fixture.html (new)	Cross-origin/nested-iframe problem	Degree field lives inside <iframe id="icims_content_iframe">; the advisor is injected into the child Frame, not the top-level Page

All three ask the same question ("B.S. Game Development" against that ATS's own tenant-curated degree list) through the same advise.ts function, with only the host object (Page vs. Playwright Frame) varying — enforced by the function's actual TypeScript signature, which takes a minimal structural EvaluateTarget (addScriptTag + evaluate) that both Page and Frame satisfy:

export interface EvaluateTarget {
  addScriptTag(options: { path: string }): Promise<unknown>;
  evaluate<T, Arg>(fn: (arg: Arg) => T | Promise<T>, arg: Arg): Promise<T>;
}
export async function adviseOnAmbiguousField(
  target: EvaluateTarget, model: ModelAdapter, request: AdvisorRequest,
): Promise<AdvisorHint | null> { /* ... */ }

Observed mock-model run (npm run demo:advisor in tools/pageagent-lab), one call each:

--- Workday (selector problem) ---
  Top choice: Bachelor of Science (BS) (confidence 40)
  Full ranking: Bachelor of Science (BS)=40, Bachelor of Arts (BA)=17, MBA=0, AA=0
  LLM calls used for this advisor turn: 1

--- Dayforce (timing/hydration problem) ---
  Top choice: Bachelor of Science (confidence 40)
  LLM calls used for this advisor turn: 1

--- iCIMS (cross-origin iframe problem) ---
  Top choice: BS (confidence 50)
  LLM calls used for this advisor turn: 1

Same call site, same ranking logic, same "1 call per ambiguity" cost shape — regardless of whether the underlying ATS problem is selectors, timing, or iframes.

---

Token cost: advisor-only vs. full replacement

The abandoned full-replacement framing estimated 45-80 LLM calls per application (one PageAgent macro-tool turn per meaningful DOM interaction across every step of a multi-page Workday application — confirmed directly in this branch's own full-fill prototype, which took 20+ steps/turns to fill contact info, one work-history block, and education).

The advisor role fires only at the existing judgement-call pause, which per the survey above happens at a rate far below "every field":

ATS	Observed pause rate	Advisor calls per application (approx.)
iCIMS	13/20 iterations (65%) on the degree field alone	~1 (just the degree ambiguity)
SuccessFactors	~60% of iterations, degree field	~1
Lever	up to 18/20 in one specific run (bug-discovery, not steady-state)	~1 (typically)
Workday, ADP, Paycom, UKG	"every iteration" — but scoped to the one degree field per application	~1
Oracle, Paycor, Greenhouse	no numeric pause rate documented yet; qualitatively similar (one ambiguous field per app, not per-field)	~1

In every observed case, the judgement call is triggered by one field per application (the degree dropdown) in the current fixtures — a tenant with several genuinely ambiguous custom-screening questions would raise this proportionally, but even a pessimistic estimate of 3-5 ambiguous fields per application puts advisor cost at 3-5 calls/application, roughly 10-25x cheaper than the 45-80 calls/application full-replacement estimate, while keeping 100% of Playwright's existing selector/iframe/timing reliability for everything that isn't ambiguous.

---

What stays the same

• Every ATS's Playwright selector chain, iframe/FrameLocator handling, and hydration-wait logic — completely untouched.
• The never-auto-submit contract — the advisor is never invoked anywhere near Review & Submit, and nothing about the submit gate changes.
• The existing JudgementCall/RunStatus.AwaitingJudgement pause-for-user UX — it is the fallback whenever the advisor is unavailable or itself not confident, exactly as it is today with no advisor at all.
• docs/technical/scraper-automation-learnings.md and its Workday-specific findings remain fully valid — this document adds a seam on top, it does not revise them.

What's additive (this branch)

• src/WorkWingman.Core/Interfaces/IPageAdvisor.cs — the seam.
• src/WorkWingman.Core/Interfaces/IAdvisorTransport.cs — the transport abstraction both concrete advisors share.
• src/WorkWingman.Infrastructure/Automation/Advisor/ — AdvisorPromptBuilder, ClaudePageAdvisor, CodexPageAdvisor, StubCodexAdvisorTransport.
• tests/WorkWingman.Tests/PageAdvisorTests.cs — proves both advisors are interchangeable.
• tools/pageagent-lab/ — the TypeScript/PageAgent side: src/advisor/advise.ts (the real advisor call, structurally prevented from acting on the DOM), src/harness/runAdvisorDemo.ts (cross-archetype proof), plus the three fixtures.

Nothing existing was modified or removed. WorkdayAutomationEngine.cs and every sibling ATS engine are untouched; wiring any of them to actually call IPageAdvisor at their pause site is the next, separate, per-engine step (see Per-ATS survey for which ones need the shared JudgementCall model plumbed in first).

Electron / IPC path for the real Claude transport

The production ClaudePageAdvisor's IAdvisorTransport would not run PageAgent in a headless Playwright page (as the prototype does, for a self-contained CLI demo) — it would run inside the same Electron WebContentsView that already hosts the Live-Run browser the user watches (electron/main.js currently spawns the .NET API and loads the Angular shell; the Live-Run view is the browser surface Playwright drives today). Sketch of the real path:

1. The .NET engine (via IAdvisorTransport.SendAsync) sends the built prompt over an IPC channel to the Electron main process (a small addition to electron/preload.js / main.js — not yet built, out of scope for this prototype).
2. Electron's main process forwards the prompt into the Live-Run WebContentsView's renderer, where PageAgent is already injected (reading the same live DOM the user is watching, not a detached fixture) and configured exactly as advise.ts does — customTools disabling every mutating tool, customFetch pointed at a ClaudeModelOptions-equivalent adapter reading ANTHROPIC_API_KEY from the Electron main process's environment (never the renderer, never hardcoded).
3. PageAgent's single-turn answer streams back over the same IPC channel to .NET, which ParseResponses it exactly as the prototype does.

This keeps credentials host-side (Electron main process env, same trust boundary as the existing vault-backed automation), keeps the advisor read-only in the renderer via the same customTools-disabling technique already proven in advise.ts, and requires no new network surface — everything stays inside the already-running Electron process tree.