Cross-base scraper automation learnings — summary (technical)¶
Plain-language version: ../plain/scraper-automation-learnings-summary.md
This is the consolidated blueprint distilled from eleven separate offline learning labs, one
per major Applicant Tracking System (ATS): Workday, UKG, Greenhouse, Lever, Oracle (Taleo + Oracle
Recruiting Cloud), SAP SuccessFactors, iCIMS, Paycom, Dayforce, ADP, and Paycor. Each lab is
documented in full in its own worktree (scraper-automation-learnings-<ats>.md); this doc pulls
out the pattern that generalizes across all of them, for hardening the app's real per-ATS adapters.
Every lab ran against a loopback-only fake site built from public knowledge of that ATS's markup — never a real tenant, never a real account, never a real submit. See each base's own learnings doc, and the making-of chapter, for the full safety-by-construction detail.
The law: there is no universal "best" selector¶
Line up all eleven bases and one law falls out — the right lead anchor is a property of the ATS
platform, not a fixed priority order to apply everywhere. Workday's data-automation-id is the
best possible anchor on Workday and is absent everywhere else. ADP's formcontrolname is its most
durable anchor and is the exact opposite priority from Paycom, where plain id leads and
formcontrolname doesn't exist. Assuming the previous base's ordering transfers to the next base is
the mistake the labs exist to catch — it happened at least once mid-run (Paycom's EEO-location
probe bug) and the fix generalized cleanly to ADP's near-identical ambiguity precisely because it
was re-derived, not copy-pasted.
The actionable rule: detect the ATS first (AtsDetector / AtsKind), pick that platform's lead
anchor, and always fall back through the full chain anyway. A chain that only tries the "right"
anchor and gives up if it's missing is exactly as fragile as hard-coding one selector — the whole
value of a fallback chain is that it degrades gracefully when a tenant's next release drops or
renames the primary hook.
The blueprint table¶
| ATS | Lead anchor | Fallback chain (best → last resort) | iframe-aware? | Notable quirk |
|---|---|---|---|---|
| Workday | data-automation-id (exact) |
id exact → id substring (*=) → aria/name/type → generic tag |
No (single document) | .First must be scoped >> visible=true — repeated ids across wizard steps caused 100% iteration failure until fixed; degree control's clickable target is a descendant of the id, not the id itself |
| UKG | #id |
#id → [name] → [aria-label] → label-sibling → [placeholder] |
No | name is the real workhorse fallback; short-probe lead candidates + full timeout budget on the last (hydration); wait on URL or anchor (client routing) |
| Greenhouse | #id (semantic) |
#id → label text (GetByLabel) for custom questions |
Sometimes (careers-page embed via #grnhse_app) |
Custom questions must resolve by visible label text — the numeric index and question_id are unstable per posting |
| Lever | [name] |
[name] → [data-qa] → [type] → name*= substring → [aria-label] |
No | name is Lever's real backend-binding contract; custom questions keyed by an unstable per-posting card UUID, matched positionally |
| Oracle — Taleo (legacy) | [name] (id is dead weight) |
name → generic |
Yes, deeply (iframe-in-iframe) | id-hit 0/80; name-hit 77/80 — a naive id-first strategy fails completely here |
| Oracle — ORC (modern) | #id |
id → later tiers only when a section needs them |
Less than Taleo | id-hit 57/80 — the opposite lead anchor from Taleo despite being the same vendor |
| SAP SuccessFactors | id substring token (id*='firstName') |
substring token → alternate token spelling → [aria-label] → [name] → [placeholder] |
Yes — ~24–30% of matched fields in an iframe | Reused ids across steps demand first-visible-match resolution; visible-text (:has-text) selectors are last resort due to heavy per-tenant localization |
| iCIMS | id suffix/substring token | suffix (id$=) → substring → [name] → [aria-label] |
Yes — deeply, sometimes 2 levels; frame resolution is its own fallback chain | Hardest of the eleven: generated ASP.NET ids, table layout, forced account gate; no bare blind-iframe fallback (would wrongly descend on single-frame tenants) |
| Paycom | #id |
id → [name] → label-sibling → [aria-label] (weak) → [placeholder] |
No | aria-label inconsistent enough to demote below label-sibling — opposite of UKG; routing-decision probes (e.g. "is EEO its own page?") need the same fallback discipline as a field |
| Dayforce | [data-testid] |
data-testid → [data-automation] → [aria-label] → label-sibling → #id → [placeholder] |
No | Not a selector problem — a timing problem: client-rendered Angular SPA needs a hydration gate (wait for app's busy signal to clear) plus per-field wait-for-visible, not just wait-for-attached |
| ADP | [formcontrolname] |
formcontrolname → #id → [name] → label-sibling → [aria-label] (weak) → [placeholder] |
No | Angular reactive-forms binding key is the most durable anchor of all eleven bases; id is the least durable here (framework-assigned) — inverse of Paycom |
The "pick lead anchor per detected ATS, always fall back through the chain" rule¶
In practice, this means the production adapter shape is:
- Detect the ATS from the URL/host (
AtsDetector→AtsKind) before choosing any selector strategy. - Select that platform's lead anchor from the table above — not a hardcoded global default.
- Always try the full fallback chain, in order, scoped to the first visible match — never
trust
.Firstwithout a visibility filter; every lab that skipped this hit near-100% failure until it was added. - If the ATS embeds iframes (iCIMS, SuccessFactors, Taleo, and sometimes Greenhouse), treat
frame resolution itself as a fallback chain — resolve the outer frame, wait for it to actually
load before probing, descend into a nested frame only when a probe genuinely misses in the outer
one, and fall back to the top document for tenants that render inline. Never blindly descend into
any
<iframe>— that misroutes single-frame tenants. - If the ATS is a client-rendered SPA with real hydration delay (Dayforce, and to a lesser extent UKG/ADP), gate on the app's own busy/loading signal first, then wait-for-visible per field — don't rely on wait-for-navigation alone.
- Treat "which branch of the flow am I on?" as a selector-chain problem too, not a single-selector check — the Paycom/ADP Voluntary-Disclosures-location bug shows a routing probe needs the same multi-tier fallback discipline as an ordinary field, or it silently mis-routes under exactly the fuzzing these labs exist to catch.
- Never guess on genuine ambiguity (a degree dropdown with no exact match, a tenant's novel custom question) — raise a judgement call, rank the options, and pause below the confidence threshold. This is the same discipline the hybrid PageAgent-advisor architecture extends further.
Per-base labs¶
Each ATS base has its own offline learning lab with a full write-up (technical + plain):
| Base | Technical | Plain |
|---|---|---|
| Workday | learnings | plain |
| ADP | learnings-adp | plain |
| Dayforce | learnings-dayforce | plain |
| Greenhouse | learnings-greenhouse | plain |
| iCIMS | learnings-icims | plain |
| Lever | learnings-lever | plain |
| Oracle (Taleo/ORC) | learnings-oracle | plain |
| Paycom | learnings-paycom | plain |
| Paycor | learnings-paycor | plain |
| SuccessFactors | learnings-successfactors | plain |
| UKG | learnings-ukg | plain |
| Amazon | learnings-amazon | plain |
| learnings-google | plain | |
| Meta | learnings-meta | plain |
| Microsoft | learnings-microsoft | plain |