Skip to content

Scraper Automation Learnings — Dayforce (Ceridian) Candidate Experience

Technical write-up. Companion plain-language version: ../plain/scraper-automation-learnings-dayforce.md.

This documents what the local, offline Dayforce scraper lab (tools/WorkWingman.ScraperLab.Dayforce) taught us about automating the modern Dayforce (Ceridian) "candidate experience" apply flow, and how it differs from the Workday adapter the lab is modeled on. Everything here was learned against a loopback HttpListener fixture — there is no real network, no real *.dayforcehcm.com, no account creation, and the final submit is a no-op. See "Safety model" at the end.

What the lab is

Same shape as the Workday and UKG labs — a fixture, an adapter, a loop, learnings, and tests:

Piece File Role
Fixture FakeDayforceSite.cs Loopback HttpListener that reproduces a realistic Dayforce candidate-experience SPA apply flow from public knowledge. Parameterized per iteration.
Variation SiteVariation.cs Randomizes hooks, labels, async-load delays, and missing fields to stress fallbacks + wait logic.
Selector chain SelectorChain.cs Ordered fallback selectors per field, each tried with a wait-for-visible; records hit/miss and wait/timeout telemetry.
Adapter DayforceAutomationEngine.cs Playwright-driven engine with robust waits for client-rendered fields; fills from an IntakeProfile; logs every selector attempt, wait outcome, judgement call, and the suppressed submit.
Loop LearningLoop.cs Iterates fresh randomized fixtures; rolling JSONL learnings + selector/wait/failure stats; --iterations N / --minutes M; log-and-continue.
Entry point Program.cs CLI arg parsing + Playwright-not-installed hint.
Tests ../../tools/WorkWingman.ScraperLab.Dayforce.Tests/DayforceLabTests.cs xUnit, fully offline (no browser).

Run it:

dotnet run --project tools/WorkWingman.ScraperLab.Dayforce -- --iterations 20
dotnet run --project tools/WorkWingman.ScraperLab.Dayforce -- --minutes 30
dotnet run --project tools/WorkWingman.ScraperLab.Dayforce -- --iterations 500 --minutes 60 --out run.jsonl

The Dayforce candidate experience, from public knowledge

The modern Dayforce candidate experience is a single-page Angular application. The fixture reproduces its salient, publicly-observable traits (it does not copy proprietary markup):

  • Client-rendered fields. The server ships a near-empty shell; Angular hydrates each step's fields after the shell paints. The fixture models this literally: every route past /apply returns an empty #hydration-host with aria-busy="true", and a bootstrap script injects the real fields after a per-step delay (BootstrapDelayMs / StepHydrationMs).
  • data-* test hooks + ARIA. Controls carry data-testid / data-automation hooks plus aria-label, role, and aria-required. These are the primary automation anchors.
  • Angular Material markup. Fields render inside mat-form-field / mat-input / mat-select wrappers; some tenants render the profile step as a mat-step stepper whose panels animate in.
  • A clean, linear wizard. Job posting → Apply → Contact/Profile → Experience → Education → Screening questions → EEO / voluntary self-id → Review & Submit. Cleaner and more consistent than legacy ATSes (iCIMS, Taleo), and client-routed (no full page loads between steps).

Parameterized variation

Each loop iteration builds a SiteVariation that shuffles:

  • Primary anchor (testid / automation / aria / id / placeholder) — which hook the fields lead with.
  • Label style (plain / verbose / localized, incl. Spanish) — Dayforce tenants reword and localize heavily.
  • DropTestHooks — drops the data-* hooks so the chain must fall back to ARIA/label.
  • BootstrapDelayMs / StepHydrationMs — hydration jitter; always non-zero because Dayforce is always client-rendered.
  • MatStepper — wraps the contact step in an animated Material stepper.
  • ResumeUploadVariant — offers resume upload first, forcing the manual-entry fallback.
  • AsyncEeoChunk — the voluntary self-id module lazy-loads as a separate chunk (extra delay).
  • DropOneRequiredField — one required field never renders (missing-field stress).

The wait + selector strategy

Wait-for-app, then wait-for-element

The single most important lesson: on Dayforce you cannot read the DOM eagerly. A scrape that runs at DOMContentLoaded sees an empty container. The adapter uses a two-layer wait:

  1. Wait-for-app (hydration gate). Before touching any field on a route, WaitForHydrationAsync waits for the #hydration-host to flip aria-busy from true to false via page.WaitForFunctionAsync(...) polling. This is the "Angular has painted" signal. If no hydration host exists (the eagerly-rendered job posting), it's a fast no-op.
  2. Wait-for-element (per-field). Each field's SelectorChain.ResolveAsync calls WaitForAsync(State = Visible) per candidate selector — not merely attached, because Angular may attach a hidden node first and reveal it later. Every wait is timed, so the loop reports avg-hit-wait, max-hit-wait, and timeouts.

This is belt-and-suspenders: the hydration gate handles the common case cheaply, and the per-field visible-wait absorbs the residual jitter (async EEO chunk, stepper animation).

Selector fallback priority (Dayforce-specific)

SelectorChain tries candidates in this order, recording each attempt:

  1. [data-testid='…'] — Dayforce's own test hook (most stable when present).
  2. [data-automation='…'] — alternate automation hook.
  3. [aria-label='…'] — ARIA label (Dayforce stamps these widely; survives when data-* is dropped).
  4. label:has-text('…') + input/select — visible-label association.
  5. #id — only some tenants emit ids.
  6. [placeholder='…'] — last-resort, brittle (localized away easily).

The loop's per-selector hit/miss table confirms the priority pays off: in a 20-iteration run the [data-testid=…] primaries hit on the large majority of iterations, ARIA labels caught every case where DropTestHooks fired, and the [data-automation=…] / #id / [placeholder=…] rungs mostly recorded misses (they exist to catch the tail, not the mainline).

How Dayforce differs from Workday (the core learnings)

Dimension Workday Dayforce
Rendering Server-rendered; fields present at DOMContentLoaded. Client-rendered Angular SPA; fields hydrate after the shell paints. Must wait-for-app then wait-for-element.
Primary anchor Universal, stable data-automation-id on essentially every control. data-testid / data-automation when present, but not universal — ARIA is the essential fallback.
Wait discipline Mostly wait-for-navigation; elements are there once the page is. Wait-heavy: an explicit hydration gate + per-field visible-wait on every step. Timeouts are a real, expected failure mode.
Routing Multi-page / server round-trips per step. Client-side routing; Continue swaps the route and re-hydrates.
Markup Workday's own component DOM. Angular Material (mat-form-field, mat-select, mat-step).
Localization Present. Heavy; ARIA labels and visible text both localize, which is why data-*/aria beat placeholder.
Cleanliness Verbose, deeply nested. Cleaner, more linear wizard — easier once you solve the wait problem.

The one-line takeaway: Workday is a selector problem; Dayforce is a timing problem. Workday hands you a stable hook and asks you to find it in a dense DOM. Dayforce hands you a clean DOM but only after you wait for Angular — get the waiting right and the selectors are comparatively easy.

Failure modes observed

  • Hydration timeout. When a step's hydration delay approaches the wait budget (worst with AsyncEeoChunk, which adds ~400ms), the per-field visible-wait can time out. Classified as timeout (element/nav never appeared — hydration lag or missing field). The loop logs and continues; the fix is generous, per-step-tunable timeouts (EEO gets 8s vs 6s default).
  • Missing required field. DropOneRequiredField removes the phone input; its whole chain misses and the field is recorded in MissedFields rather than crashing the run. Real tenants omit/rename fields, so the adapter must degrade, not throw.
  • Upload-first experience. ResumeUploadVariant would trap a naive adapter into a file dialog. The engine detects the "Enter manually" affordance first and falls through to the manual fields.
  • Ambiguous free-text degree (judgement call). The profile carries "Bachelor of Science" with field of study "Game Development" separately, but a free-text degree box may want a combined "B.S. in Game Development". The engine flags this as an AmbiguousField and logs a JUDGEMENT line rather than silently guessing the format — mirroring the Workday adapter's degree-ambiguity handling.
  • Client-side navigation race. The first working version clicked "Continue"/"Apply" and then separately awaited the new URL. Against the SPA's location.href route swaps this intermittently missed the navigation and timed out (~1 iteration in 3). The fix: bind the click and the navigation-wait into one atomic RunAndWaitForNavigationAsync (ClickAndNavigateAsync). This is a general SPA lesson — never issue a navigating click and its wait as two separate awaits.
  • Iteration hang under load (watchdog-timeout). A single Playwright wait can stall far past its own timeout when the host is starved of CPU (e.g. other builds/tests running). A per-iteration 60s watchdog cancels the iteration (force-closing the browser context so in-flight waits throw), records it as watchdog-timeout, and lets the loop continue. In a representative 20-run on a heavily-loaded machine, 14 iterations completed and 6 tripped the watchdog purely from contention — none were logic failures, and the loop still finished cleanly with a full summary. The lesson: log-and-continue must cover hangs, not just exceptions.

Hardening notes

  • Per-step timeout budgets, not one global value. Steps known to lazy-load (EEO) get more.
  • Wait for visible, never just attached — Angular attaches-then-reveals.
  • Gate on the app's own busy signal (aria-busy) when the framework exposes one; it's cheaper and more reliable than blindly retrying selectors.
  • Lead with framework-agnostic anchors (data-*, then ARIA). Placeholder/CSS-class selectors are localization- and refactor-fragile; keep them as a last rung only.
  • Record wait durations, not just hit/miss. The avg/max-hit-wait and timeouts numbers are what tell you whether your timeout budget is right for a given tenant.
  • Degrade on missing fields; never let one absent control abort the run.
  • Bind navigating clicks to their navigation (RunAndWaitForNavigationAsync) so a click and its URL-wait can't race on an SPA route change.
  • Watchdog every iteration. A single hung step must never stall a long loop; cap and continue.

Safety model (non-negotiable)

  • No real network. The fixture binds http://127.0.0.1:<free-port>/ via HttpListener. Nothing leaves loopback. AtsDetector is never pointed at a real host; Dayforce host detection in the loop is a string check against a local string (dayforcehcm.com / ceridian).
  • No account creation. Sign-in is simulated locally; no credentials are used or stored.
  • Submit is a no-op. The only submit target is the fixture's loopback POST …/submit, which sets a flag and returns a "LOCAL NO-OP — nothing was sent" page. The engine never clicks submit: it resolves the control, logs would submit (SUPPRESSED), and stops. After each iteration the loop asserts SubmitWasRequested == false and raises a SAFETY VIOLATION if it were ever true.
  • Local profile only. Fill input is the Andrew Jones IntakeProfile (reused from src/WorkWingman.Core/Models); it is never transmitted.
  • Production untouched. The lab adds no Dayforce enum to AtsKind and edits no production models; it is fully self-contained under tools/.