Skip to content

Scraper automation learnings — ADP Workforce Now / ADP Recruiting (technical)

Plain-language version: ../plain/scraper-automation-learnings-adp.md

This documents what the local, offline ADP scraper lab taught us about automating ADP Workforce Now / ADP Recruiting candidate-portal apply flows, and how an AdpAutomationEngine adapter is hardened against them. It is the sibling of the Workday/UKG/Paycom labs, built for a fourth ATS family whose markup conventions — a genuine Angular SPA — differ from all three.

Everything in the lab is local and offline. The harness drives a loopback HttpListener fixture on 127.0.0.1. There is no real network, no real workforcenow.adp.com/myjobs.adp.com site, no account is ever created (the "Create myADP account" step is a purely local simulation with no external identity call), and no real submit endpoint. The final "Submit" is a no-op that logs would submit (suppressed) and restarts with a new randomized variation.

Where the lab lives

tools/WorkWingman.ScraperLab.Adp/
  Program.cs                     CLI entry (--iterations N | --minutes M | --out PATH)
  FakeAdpSite.cs                  loopback HttpListener serving the ADP apply flow (8 steps)
  SiteVariation.cs                per-iteration randomization (anchors, labels, missing fields, slow loads)
  SelectorChain.cs                priority fallback chain + hit/miss telemetry
  AdpAutomationEngine.cs          Playwright adapter that fills each step from an IntakeProfile
  LoopRunner.cs                   the loop runner: JSONL log + selector tallies + failure modes
  SampleProfile.cs                the Andrew Jones seed (reuses Core IntakeProfile shapes)
tools/WorkWingman.ScraperLab.Adp.Tests/
  FakeAdpSiteTests.cs             fixture + detection + safety tests
  SelectorLogicTests.cs           selector-chain + variation-randomization tests

Also touched (additive only, reused by every ATS lab): AtsKind.Adp was added to src/WorkWingman.Core/Models/JobPosting.cs, and a workforcenow.adp.com / myjobs.adp.com detection branch was added to src/WorkWingman.Infrastructure/Automation/AtsDetector.cs — mirroring how Workday/Greenhouse/Lever/iCIMS/UKG/Paycom are already recognized there.

How ADP differs from Workday/UKG/Paycom (the core lesson)

The Workday adapter leans on data-automation-id, a stable semantic attribute stamped on essentially every control. Paycom and UKG have no such attribute and fall back to plain id/name and aria-label respectively. ADP is different again: it is a genuine Angular single-page application, and its reactive-forms controls commonly expose a formcontrolname attribute — the Angular binding key that ties a DOM control to its FormControl in the component class. From public knowledge of the ADP candidate portal (https://workforcenow.adp.com/mascsr/default/mdf/recruitment/recruitment.html?cid=...&ccId=... and the newer https://myjobs.adp.com/... shape):

Concern Workday UKG Pro Recruiting Paycom ATS ADP Workforce Now / Recruiting
Primary stable hook data-automation-id on every control none (leans on ARIA) none — plain id/name only formcontrolname (Angular reactive-forms binding key)
Auto-generated id durability n/a — data-automation-id used instead secondary, fairly stable fairly stable, semantic least durable of the four — framework-assigned, can shift between renders
Accessibility anchors present but secondary widely stamped present but inconsistent present but inconsistent — similar to Paycom, never primary
Framework Workday's own SPA React/Angular, client-side routing mostly server-rendered, light hydration genuine Angular SPA — full client-side routing + reactive forms
Candidate account required required required (self-service portal) required — "myADP" candidate account (simulated locally here, never real)
Voluntary EEO inline step collapsed accordion step OR folded inline at Review, per tenant "Voluntary Disclosures" — step OR folded inline at Review, per tenant (same structural shape as Paycom, independently modeled)
Resume vs manual resume-optional resume-upload-first common resume-upload-first common resume-parse-first is the ADP default, more aggressively nudged than Paycom/UKG
Custom questions fixed step schema fixed step schema fully tenant-defined "Additional Questions" — fully tenant-defined per requisition, same shape as Paycom
URL identity tenant subdomain tenant subdomain single clientkey= query param two-part cid= (company id) + ccId= (client company code) query pair

The practical consequences for the adapter:

  1. Lead with formcontrolname, not id. This is the opposite priority from every other lab in this family (Workday leads with its automation id; Paycom leads with plain id). ADP's formcontrolname reflects the Angular FormControl binding and survives re-renders that reshuffle the DOM's auto-generated ids; leading with id here would have been the wrong lesson to carry over from Paycom.
  2. The Voluntary Disclosures step's location is itself a variable, not just its content — exactly the Paycom lesson, independently reproduced: some tenants render it as its own step, others fold it into the tail of Review. The adapter must detect where it is, not just fill it.
  3. Additional Questions have no fixed schema. Same as Paycom: the adapter cannot hard-code "salary" / "start date" fields; it must probe for whichever questions a given requisition happens to render.
  4. Two-part tenant identity (cid + ccId) means detection and telemetry track both — a single clientkey-style token (as Paycom uses) is not enough to reconstruct a realistic ADP apply URL.

The selector fallback chain (adapter approach)

SelectorChain tries candidates in priority order, waiting for the first visible match. The priority order encodes the ADP lesson — the inverse of Paycom's and a new tier ahead of Workday's:

  1. [formcontrolname='...'] — the Angular reactive-forms binding key. ADP's most stable anchor when present, and it is present on every control in this fixture because it is what actually drives the binding — a real ADP tenant customization is far less likely to strip it than to strip an auto-generated id.
  2. #id — auto-generated id. Demoted to 2nd tier here (unlike Paycom, where plain id led) because ADP's id values are framework-assigned and the least durable of the four labs.
  3. [name='...'] — the form field name; the workhorse fallback once again, same as every other lab in this family.
  4. label-sibling (label:has-text('…') + input) — visible-label association.
  5. [aria-label='…'] — a weaker signal on ADP, similar to Paycom; still occasionally the only anchor left standing.
  6. [placeholder='…'] — last-resort, brittle.

Two structural refinements matter for ADP specifically:

  • Short probe on lead candidates, full budget on the last — identical timing discipline to the UKG/Paycom labs: ~400ms probes on early candidates, ~1.5s on the last (which may be waiting on the personal-info step's Angular route-resolver hydration).
  • Detecting where the Voluntary Disclosures step lives, not just its fields. ADP's fixture reuses the exact same page-detection discipline the Paycom lab had to learn the hard way (see the failure-mode table below) — carried forward correctly from day one this time, because the Paycom lab's fix generalized cleanly.

What the loop measured (validation run, --iterations 20)

A clean 20-iteration run produced 20/20 completed, 0 failures, every iteration detected as AtsKind.Adp, every one submit-suppressed, and no field's whole fallback chain went unresolved (MissedFields was empty on every iteration). Selector telemetry highlights:

  • [formcontrolname='continue'] — the shared "Continue" button binding across every step — was the single highest-hit selector (87 hits in one run), confirming formcontrolname as ADP's most reliable anchor, the same role [name='continue'] played for UKG/Paycom.
  • Every [formcontrolname='firstName']-style selector showed hits alongside misses on iterations where a different attribute was the iteration's PrimaryAnchor (e.g. anchor=name, anchor=aria) — these are expected probe attempts on a non-primary tier, not full-chain misses; every one of those iterations still resolved the field via [name='firstName'] or another later tier. This is the fallback chain working exactly as designed, and it is why "per-selector hit/miss" and "whole-chain miss" are tracked as two separate numbers — conflating them would make normal fallback exercising look like a defect.
  • #id-style selectors showed a roughly 55/45 hit/miss split, matching the ~45% DropStableIds probability built into SiteVariation (ADP's highest of the four labs, reflecting that its auto-generated id is the least durable anchor) — every one of those misses fell through cleanly to formcontrolname or name.
  • Tenant Additional-Question selectors (#aq_* / [formcontrolname='customQ*']) only appear in the ~60% of iterations where HasCustomQuestions is true — the adapter's generic-probe approach handled the other 40% (no questions rendered) without any special-casing, same as Paycom.
  • Resume-parse-first was the majority path (ResumeUploadVariant ~55% probability, ADP's highest of the labs modeled) and the "Enter manually" fallback fired cleanly every time it was offered.

Failure modes anticipated and pre-empted (carried forward from the Paycom lab)

Unlike the Paycom lab — which discovered its EEO-page-detection bug live during its own validation runs — the ADP lab's FillVoluntaryEeoAsync probe was written with the Paycom fix already applied, because the underlying shape (a shared gender/formcontrolname='gender' control appearing both on a dedicated Voluntary Disclosures step and folded inline at Review) is structurally identical:

# Anticipated symptom (if the Paycom lesson were ignored) Root cause it would reproduce Pre-emptive fix applied from day one
1 Probing #eeoGender, [formcontrolname='gender'] to decide "is Voluntary Disclosures on its own step?" would misfire Review's inline fold-in also renders a control bound to gender (by design, so the two code paths share a mapping function) — the probe cannot tell the two apart on gender alone Probe on an anchor that exists only on the dedicated step (veteranStatus/disabilityStatus), never the shared gender binding
2 An id-only version of fix #1 would misroute whenever DropStableIds=true removed the dedicated step's ids The probe would wrongly conclude "folded inline," skip the real Voluntary Disclosures step, and leave Playwright stranded when Review's submit-locator ran Probe using both the id and the formcontrolname/name form of veteran/disability, so the check survives DropStableIds the same way every other field's fallback chain does

The 20/20 clean run with zero missed fields and zero page-detection failures confirms the pre-emptive fix holds. The lesson generalizes across ATS families: any "which branch of the flow am I on?" probe is itself a selector-chain problem — it needs the same formcontrolname→id→name fallback discipline as an ordinary field, not a single hard-coded anchor, and this discipline transfers cleanly to a brand-new markup family (Angular vs. Paycom's mostly-server-rendered DOM) as long as the underlying shape of the ambiguity is the same.

Judgement calls (ambiguity, not guessing)

Mirroring the Workday/UKG/Paycom "degree" case, the education step raises a judgement call every iteration: the profile carries Degree = "Bachelor of Science" and FieldOfStudy = "Game Development" separately, but some ADP tenants render a single free-text degree box that expects a combined "B.S. in Game Development". The adapter flags this (AmbiguousFields, logged JUDGEMENT — degree format ambiguous) rather than silently picking a format.

Tenant-defined Additional Questions are a second, structural source of ambiguity: because there is no fixed schema, the adapter maps its three synthetic fixture questions from direct IntakeProfile fields (earliest start date, salary expectation, relocation preference) when present, and simply skips the step when the fixture renders none — it never invents an answer to a question shape it doesn't recognize.

Safety model (identical discipline to the Workday/UKG/Paycom labs)

  • No real network. The fixture is loopback-only (http://127.0.0.1:<port>/), and its ApplyUrl mimics the public .../mascsr/default/mdf/recruitment/recruitment.html?cid=...&ccId=... shape purely as a string — it never resolves to a real host.
  • No account is ever created. The fixture's "Create account" path (/mascsr/default/mdf/recruitment/create-account) is a local-only simulated redirect; it sets an AccountCreationSimulated flag for tests to observe and nothing else. The engine's SignInAsync always takes the "Sign In" path against the local fixture with a throwaway password, never "Create account," and never touches any real identity system (no real myADP account, ever).
  • Submit is a no-op. The Review step's submit control is located and verified to exist, then the adapter logs would submit (SUPPRESSED) and never clicks it. The fixture's /mascsr/default/mdf/recruitment/submit handler only sets a flag and returns a "LOCAL NO-OP" page.
  • Hard assertion. Each iteration asserts FakeAdpSite.SubmitWasRequested == false after the run; a true value throws a SAFETY VIOLATION and is counted as a failure. Across all validation runs this never fired.
  • Realistic data, local only. The fill-input is the Andrew Jones seed (reusing Core's IntakeProfile shapes: Full Sail University B.S. Game Development, Equifax). It is never transmitted anywhere.

Detection

AtsKind.Adp and a workforcenow.adp.com / myjobs.adp.com detection branch were added to the production AtsDetector (additive only — no existing detection logic was changed). Because the fixture itself must stay loopback-only, each loop iteration synthesizes a realistic https://workforcenow.adp.com/mascsr/default/mdf/recruitment/recruitment.html?cid=<synthetic>&ccId=<synthetic> string (never resolved, never dialed) and confirms the detector classifies it as AtsKind.Adp before driving the flow — the same "confirm detection matches the real router" discipline the UKG/Paycom labs use.

Running it

Short validation:

dotnet run --project tools/WorkWingman.ScraperLab.Adp -- --iterations 20

Long local hardening run (time-bounded, e.g. an hour), writing a rolling JSONL log:

dotnet run --project tools/WorkWingman.ScraperLab.Adp -- --minutes 60 --out adp-run.jsonl

Each JSONL line is one iteration: variation, detected ATS, outcome, steps completed, ambiguous count, missed fields, submit-suppressed flag, elapsed ms, and the per-iteration log. The console prints a rolling summary: selector hit/miss table (most-missed first), fields that missed the whole chain, and failure-mode counts.

Playwright's Chromium is required once:

pwsh tools/WorkWingman.ScraperLab.Adp/bin/Debug/net10.0/playwright.ps1 install chromium

Tests:

dotnet test tools/WorkWingman.ScraperLab.Adp.Tests

Takeaways for a production ADP adapter

  1. Lead with formcontrolname, fall back to id, then name, then label, then aria-label. This priority order is the opposite of Paycom's (which leads with id) — carrying Paycom's id-first lesson into ADP unmodified would have been the wrong call. Always confirm the actual most-durable anchor for a new ATS family rather than assuming the last lab's ordering transfers.
  2. Treat "which branch of the flow am I on?" as a selector-chain problem, not a single-selector check. The Voluntary-Disclosures-location probe generalizes cleanly from the Paycom lab's EEO-location bug: any branch decision needs the same multi-tier fallback discipline as filling a field, or it will silently mis-route on exactly the fuzzed variations the lab exists to catch.
  3. Additional/custom questions have no fixed schema. Probe generically for whichever questions a requisition renders; never assume a specific set exists.
  4. Never create a real candidate account during automated hardening. Only ever sign in against a local fixture with throwaway credentials — there is no such thing as a "test" myADP account that's safe to create against the real site.
  5. Flag ambiguity, don't guess — carry the Workday/UKG/Paycom judgement-call discipline into ADP (degree format, unmapped custom questions).
  6. Keep submit suppressed and asserted — the safety boundary is a hard invariant, not a convention.