Skip to content

Scraper Automation Learnings — Google Careers Apply Flow

Technical write-up. Companion plain-language version: ../plain/scraper-automation-learnings-google.md.

This documents what the local, offline Google careers scraper lab (tools/WorkWingman.ScraperLab.Google) taught us about automating google.com/about/careers' bespoke, custom-built apply flow — the hardest kind of base in this series, because it is NOT a vendor ATS at all. Everything here was learned against a loopback HttpListener fixture — there is no real network, no real google.com, no account creation, and the final submit is a no-op. See "Safety model" at the end.

What the lab is

Same shape as the Workday/Dayforce/iCIMS labs — a fixture, a variation randomizer, a selector chain, an adapter, a loop, learnings, and its own test project:

Piece File Role
Fixture FakeGoogleSite.cs Loopback HttpListener reproducing a realistic Google careers apply flow from public knowledge. Parameterized per iteration.
Variation SiteVariation.cs Randomizes anchors, labels, hydration delays, rotated ids, and missing/ambiguous fields to stress fallbacks + the judgement pause.
Selectors GoogleSelectors.cs Per-field fallback chains — six rungs deep (more than Workday or Dayforce need).
Selector chain SelectorChain.cs Ordered fallback resolution, each candidate tried with a wait-for-visible; records hit/miss and wait/timeout telemetry.
Adapter GoogleAutomationEngine.cs Playwright-driven engine with a hydration gate before every step; fills from an IntakeProfile; raises a real RunStatus.AwaitingJudgement pause on an ambiguous screening question; logs every selector attempt, hydration wait, judgement call, and the suppressed submit.
Loop LearningLoop.cs Iterates fresh randomized fixtures; rolling JSONL learnings + selector/hydration/failure stats; per-iteration watchdog; --iterations N / --minutes M; log-and-continue.
Entry point Program.cs CLI arg parsing + Playwright-not-installed hint.
Tests tests/WorkWingman.ScraperLab.Google.Tests/ xUnit + Bogus, fully offline (no browser). 39 tests.
Mutation config tools/WorkWingman.ScraperLab.Google/stryker-config.json Scoped to the browser-free logic (GoogleSelectors.cs, SiteVariation.cs, SelectorChain.SelectorTelemetry). 100% mutation score.

Run it:

dotnet run --project tools/WorkWingman.ScraperLab.Google -- --iterations 20
dotnet run --project tools/WorkWingman.ScraperLab.Google -- --minutes 30
dotnet run --project tools/WorkWingman.ScraperLab.Google -- --iterations 500 --minutes 60 --out run.jsonl

Why Google is different from every other base in this series

Every other lab in this series (Workday, Dayforce, iCIMS, Greenhouse, Lever, ADP, Oracle, Paycom, Paycor, SuccessFactors, UKG) automates a vendor ATS — a product built by a company whose business is applicant-tracking software, sold to thousands of employers. Google's careers site is different in a way that matters a great deal to the selector strategy: it is a bespoke, in-house application built by Google's own engineers, not a purchased ATS product. That has two consequences:

  1. No vendor automation attribute exists at all. Workday ships data-automation-id on essentially every control. Dayforce ships data-testid/data-automation. Google's careers apply flow has neither — there is no third-party test-automation convention to lean on, because there is no third party. The best the adapter can do is a "best-guess lead anchor": a custom, semantically-named attribute (modeled here as data-fieldid, loosely evoking Google's own public jsname/jscontroller-style conventions) that the fixture can drop or rename per iteration.
  2. The DOM shape is unpredictable and can change on any deploy. A vendor ATS's markup is contractually stable for its customers (Workday can't casually rewrite the DOM its enterprise clients depend on). Google's own team can restructure their careers site whenever they want — there is no external dependency forcing markup stability. This is the single biggest reason Google is the prime case for an IPageAdvisor (see the dedicated section below): a static selector chain, however deep, is fundamentally reactive to a DOM that can shift without notice.

The Google careers apply flow, from public knowledge

The fixture reproduces the salient, publicly-observable traits of google.com/about/careers' application flow (it does not copy proprietary markup):

  • Client-rendered fields, no framework fingerprint. The server ships a near-empty shell; a bootstrap script hydrates each step's fields after the shell paints — same category of problem as Dayforce's Angular hydration, but the fixture deliberately avoids any specific framework's markup conventions (no mat-form-field, no Angular-isms) to model the "we don't actually know what frontend stack a bespoke app uses" reality.
  • A best-guess lead anchor, not a guaranteed one. Controls carry data-fieldid (the lead guess), name (an ordinary HTML-forms convention that tends to outlive custom-attribute churn), and aria-label. The variation can drop the lead anchor and/or rotate ids to a build-generated token, forcing the chain deeper.
  • A five-step apply flow. Job posting → Resume/CV upload → Contact information → Job-specific & screening questions → EEO / self-identification → Review & Submit. Google's own postings commonly include a job-specific free-text screening question ("tell us about a relevant project") with no deterministic mapping from a candidate's structured profile data — this is the lab's judgement- pause trigger, described below.

Parameterized variation

Each loop iteration builds a SiteVariation that shuffles:

  • Primary anchor (leadattr / substring / name / aria / labelsibling / placeholder) — which rung of the chain would hit first if everything were present.
  • Label style (plain / verbose / localized, incl. Spanish).
  • DropLeadAnchor — drops/renames the best-guess data-fieldid hook so the chain must fall back to substring/name/ARIA/label.
  • RotateGeneratedIds — rotates id attributes to a build-generated token (a bespoke app's component framework commonly reissues ids per deploy), so a naive #id selector can never lead.
  • BootstrapDelayMs / StepHydrationMs — hydration jitter; always non-zero, because the flow is always client-rendered.
  • ResumeUploadVariant — offers resume upload first, forcing the manual-entry fallback.
  • AsyncEeoChunk — the self-identification module lazy-loads as a separate chunk (extra delay).
  • DropOneRequiredField — one required field never renders (missing-field stress).
  • AmbiguousScreeningQuestion — renders the free-text "relevant project" question, the trigger for the judgement pause.

The selector + hydration strategy that works

Wait-for-app, then wait-for-element (same discipline as Dayforce)

Like Dayforce, Google's bespoke apply flow cannot be read eagerly. GoogleAutomationEngine uses the identical two-layer wait Dayforce needed:

  1. Hydration gate. WaitForHydrationAsync waits for #gc-hydration-host to flip aria-busy from true to false via page.WaitForFunctionAsync(...) polling (default 6s timeout, 8s for the EEO step to absorb the async chunk). If no hydration host exists (the eagerly-rendered job posting), it's a fast no-op (an 800ms Attached-state probe that times out immediately).
  2. Per-field visible-wait. SelectorChain.ResolveAsync calls WaitForAsync(State = Visible) per candidate, timed, so the loop reports avg-hit-wait/max-hit-wait/timeouts.

Selector fallback priority — six rungs, deeper than any vendor-ATS base

GoogleSelectors.Field(...) tries, in order:

  1. [data-fieldid='…'] — the best-guess lead anchor. Most useful when present, least stable across deploys (no vendor guarantees it survives).
  2. [class*='gc-field--…'] — a substring/contains match on a class fragment carried directly on the control. Survives a lead-anchor rename because a bespoke app rarely renames every hook convention simultaneously. (Caught in review: an earlier draft wrote this as a descendant selector, [class*='...'] input, which never matched because the fixture stamps the class on the control itself, not a wrapper — fixed, and pinned with an exact-selector-shape test.)
  3. [name='…'] — an ordinary HTML-forms attribute. Tends to survive frontend rewrites even when custom attributes churn, because name often carries real semantic/backend meaning.
  4. [aria-label='…'] — the most durable anchor across redesigns; accessibility requirements tend to outlive any particular frontend implementation.
  5. label:has-text('…') + input/select/textarea — visible-label association.
  6. [placeholder='…'] — last resort, brittle (localized away easily; not always present).

This is a deliberately longer chain than Workday's (one stable id) or Dayforce's (data-* then ARIA): a vendor-less bespoke app has no single dominant, guaranteed anchor, so the chain has to hedge across several weak signals instead of leaning on one strong one. id selectors never lead the chain at all — SiteVariation.RotateGeneratedIds models ids as a build-generated token that can change on every deploy, so an #id selector is actively unsafe to rely on, let alone lead with.

20-iteration hit-rate summary

Run on this development machine (contended by other concurrent agent sessions at the time — see "Failure modes" for what that reveals):

completed=6  failed=14  submit-suppressed=6  judgement-total=2
selector attempts: hits=119 misses=103 fallback-hits=20 timeouts=79
hydration waits: avg-hit-wait=4ms max-hit-wait=44ms total-hydration-wait=12031ms

Reading the per-selector table (most-missed first), the pattern matches the design intent:

  • [data-fieldid='…'] primaries hit consistently whenever DropLeadAnchor didn't fire for that field — e.g. contact-first-name, contact-email, eeo-gender all show 6 hits / 0-2 misses across the 6 completed runs, i.e. the lead anchor won essentially every time it was present.
  • [name='…'] is the first fallback that actually earns its keep. When the lead anchor was dropped, [name='contact-email'], [name='contact-phone'], etc. picked up 100% of those misses (2 hit / 0 miss on several fields) — confirming name is a stronger second rung than aria-label for this bespoke app, because our fixture (realistically) always emits name regardless of variation, while aria-label/label-sibling/placeholder rungs saw zero hits in this run (0 hit across all of [aria-label='Continue'], label:has-text(...), [placeholder='...']) — they exist to catch a tail this particular 20-run sample didn't happen to trigger enough to land on those exact rungs, not because they're dead code (the offline tests directly exercise every rung).
  • The ambiguous screening question (q-custom-project) is the most-missed field by far (0-2 hit / 6 miss across its four candidate selectors) — expected, since AmbiguousScreeningQuestion only fires ~30% of iterations and, when it does, the field only appears in the subset of completed runs.
  • enter-manually (the resume-upload-first escape hatch) shows a similar pattern — it only exists in iterations where ResumeUploadVariant fired, so most attempts against it are structural misses (the button legitimately isn't there), not chain failures.

Failure modes observed

  • Hydration timeout. Same as Dayforce: when a step's hydration delay approaches the wait budget (worst with AsyncEeoChunk, +400ms), a per-field visible-wait can time out. Classified as timeout (element/nav never appeared — hydration lag or missing field).
  • Missing required field. DropOneRequiredField removes the phone input; its whole chain misses and phone lands in MissedFields rather than crashing the run.
  • Upload-first experience. ResumeUploadVariant would trap a naive adapter into a file dialog. The engine detects the "Enter information manually" affordance first (via GoogleSelectors.EnterManually.ResolveAsync with a short 900ms timeout so it doesn't stall the common manual-entry case) and falls through to the manual fields.
  • Ambiguous free-text screening question (judgement call). This is the lab's headline failure mode and its advisor-readiness case — see the dedicated section below.
  • Watchdog-timeout and browser-process-lost (the environment lesson). In the representative 20-run captured for this doc, the dev machine was concurrently running other Playwright/dotnet work from unrelated sessions. The per-iteration 60s watchdog tripped once (watchdog-timeout), and — new in this lab compared to Dayforce — a shared browser process can itself crash under sustained resource contention, not just individual contexts. LearningLoop.RunAsync now checks browser.IsConnected before each iteration and relaunches Chromium if the process died, classifying the relaunch as browser-crashed (relaunched); if an iteration still can't recover (browser dies mid-iteration), it's classified browser-process-lost and the loop keeps going rather than aborting. This is a strengthening of the Dayforce lab's log-and-continue guarantee: Dayforce's watchdog assumed force-closing one context was always sufficient; Google's lab assumes the whole browser process can go away and still keeps the run alive. All 20 iterations wrote a valid JSONL record and the final summary printed even though 14/20 iterations failed for environmental reasons that afternoon — the safety and telemetry guarantees held throughout.
  • Bare-filename --out path (a real bug found and fixed while building this lab). The initial LearningLoop.RunAsync called Directory.CreateDirectory(Path.GetDirectoryName(_jsonlPath)) unconditionally; when --out is a bare filename (the documented default, e.g. google-scraper-learnings.jsonl), Path.GetDirectoryName returns "" and Directory.CreateDirectory("") throws ArgumentException before a single iteration runs. Fixed to only create a directory when one was actually specified, with a regression test (LearningLoop_RunAsync_AcceptsABareFilename_WithNoDirectoryComponent) covering it.

Why Google is a prime PageAgent-advisor case

The judgement pause in this lab is deliberately wired to the real production seam, not a lab-only approximation:

  • GoogleAutomationEngine.RaiseJudgementPause(fieldLabel, question, context, options) sets ApplicationRun.Status = RunStatus.AwaitingJudgement and populates ApplicationRun.PendingCall with a JudgementCall { FieldLabel, Question, Context, Options } — the exact same model classes the production Workday/iCIMS engines already use (src/WorkWingman.Core/Models/ApplicationRun.cs). Nothing about the pause's shape is Google-specific.
  • The trigger is FillQuestionsAsync's handling of the job-specific "relevant project" free-text screening question (AmbiguousScreeningQuestion variation, ~30% of iterations). Once the field is located (the selector chain's job), there is no deterministic way to decide what to write in it — the right answer depends on reading the actual job posting's language and matching it against the candidate's work history, something a static rule can't do.
  • The lab's loop auto-resolves the pause (picks the top-ranked option) so an unattended run never blocks forever — but in production, this is exactly the pause point where an IPageAdvisor (being built on feature-pageagent-swap) plugs in unchanged: it reads ApplicationRun.PendingCall (field label, question, context, ranked options) and the live page DOM, and returns a chosen answer or a ranked recommendation — without either the engine or the advisor needing to know anything Google-specific.

Why Google, specifically, is the prime case for this seam (more than any other base in this series): every other base's judgement calls are closed-set problems — iCIMS's degree dropdown has a fixed list of options to fuzzy-match against, Dayforce's degree box just needs consistent formatting. Google's bespoke screening questions are open-ended free text on an arbitrary, unpredictable DOM — there is no fixed option list to fuzzy-match, and the DOM shape asking the question can change on any deploy with no vendor stability guarantee. A page-reading advisor that can look at whatever the page actually says right now and reason about it is not just useful here, it is close to necessary — a hardcoded rule table cannot keep up with a bespoke site that can change its own screening questions and markup at will. This lab's judgement pause is the seam where that advisor plugs in, and this document is the evidence that the seam already works end-to-end (pause raised, state populated, loop unblocked) without an advisor present yet.

Hardening notes

  • Per-step timeout budgets, not one global value (EEO gets 8s vs 6s default, mirroring Dayforce).
  • Wait for visible, never just attached — a bespoke SPA can attach-then-reveal same as Angular does.
  • Lead with the weakest-but-most-useful anchor, then degrade through progressively more durable but less specific signals (data-fieldid → class substring → name → aria-label → label sibling → placeholder). This ordering trades "most likely to exist this deploy" first against "most likely to survive any deploy" last.
  • Never lead with #id on a bespoke app with no stability guarantee on generated ids.
  • Record wait durations, not just hit/miss — avg/max-hit-wait tell you whether the timeout budget matches this tenant's hydration behavior.
  • Degrade on missing fields; never let one absent control abort the run.
  • A judgement pause should populate the SAME production model (ApplicationRun/JudgementCall) every other engine uses, not a lab-local approximation — that is what makes the pause advisor-ready with zero extra wiring.
  • Watchdog the browser process, not just the context. Under real resource contention the whole browser can die; detect IBrowser.IsConnected and relaunch rather than assuming a context-level force-close is always enough.

Safety model (non-negotiable)

  • No real network. The fixture binds http://127.0.0.1:<free-port>/ via HttpListener. Nothing leaves loopback. Google host detection in the loop is a string check against a local string (google.com/about/careers), never a real lookup, and AtsKind gains no Google entry (Google is a bespoke flow, not a vendor ATS — production models are untouched).
  • No account creation. Google's real careers flow doesn't require account creation ahead of applying in the way iCIMS/Workday sometimes do, so this lab has no simulated account step; nothing in the fixture or engine performs one.
  • Submit is a no-op. The only submit target is the fixture's loopback POST …/submit, which sets a flag and returns a "LOCAL NO-OP — nothing was sent" page. The engine never clicks submit: it resolves the control via GoogleSelectors.Submit, logs "Reached Review & Submit — would submit (SUPPRESSED). No click issued; no real endpoint.", and stops. After each iteration the loop asserts SubmitWasRequested == false and raises a SAFETY VIOLATION InvalidOperationException if it were ever true. Two dedicated offline tests (FakeSite_SubmitEndpoint_IsALocalNoOp_ThatNeverRepresentsARealSend, FakeSite_NonSubmitPost_DoesNotFlagSubmitWasRequested) prove the guard's behavior and specificity.
  • Local profile only. Fill input is the Andrew Jones IntakeProfile (reused from src/WorkWingman.Core/Models); it is never transmitted. Tests use Bogus-generated fill data for the non-canonical-profile cases so the suite doesn't rely solely on one hardcoded example.
  • Production untouched. The lab adds no Google enum to AtsKind and edits no production models (it reads the existing ApplicationRun/JudgementCall/RunStatus shapes, but adds nothing new to them); it is fully self-contained under tools/ and tests/WorkWingman.ScraperLab.Google.Tests/.