Skip to content

Scraper Automation Learnings — SAP SuccessFactors Recruiting

Source: the offline learning harness at tools/WorkWingman.ScraperLab.SuccessFactors. Everything below was learned by driving a Playwright engine against a loopback-only fake SuccessFactors (SF) apply site, iteration after iteration, with randomized markup. No real network, no real *.successfactors.com, no candidate account, and no submit is ever touched.

What the harness is

A .NET 10 console loop, same shape as the Workday and UKG labs:

Piece File Role
Fake site FakeSuccessFactorsSite.cs Loopback HttpListener reproducing an SF apply flow from public knowledge.
Engine SuccessFactorsAutomationEngine.cs Playwright-driven, iframe-aware field resolver + filler.
Variation FixtureVariation.cs Per-iteration knobs: id styles, iframe presence, locale, step order, missing fields.
Selectors SelectorModel.cs Ordered primary→fallback selector chains per field.
Loop LoopRunner.cs Walks the full flow, logs metrics, suppresses submit, restarts with new variation.
Metrics LabMetrics.cs Rolling JSONL + selector hit/miss + iframe-switch rate + failure modes.

Run: dotnet run --project tools/WorkWingman.ScraperLab.SuccessFactors -- --iterations 20 (also --minutes M, --seed N, --headed, --output <dir>).

The SF apply flow reproduced

Career-site landing → candidate account / login (SF forces this) → multi-step apply: personal info → resume (upload vs. manual) → work experience (repeater) → education (school + degree <select>) → screening questions → EEO / voluntary disclosures → review → submit.

SF markup traits (and WHY they matter for automation)

  1. SAP-generated long, underscore-separated element ids. Fields render with ids like apply_form_personalInfo_firstName_input, apply_form_navigation_nextButton, apply_form_submitApplication_button. The meaningful part is a token buried inside a long, tenant/render-dependent wrapper. Why it matters: you cannot pin an exact #id; it changes across tenants and even across renders. The durable signal is the substring token (firstName, nextButton), so every primary selector is input[id*='firstName' i], not #….

  2. Frequent iframes. SF embeds whole steps (legacy modules, resume parsing, disclosures) in same-origin iframes. In the harness ~24–30% of matched fields lived inside an iframe. Why it matters: a main-frame-only query silently finds nothing. The engine must search the main frame and every child frame for each field.

  3. Reused ids across steps. Because SF renders one step at a time, several steps share the same generated id — every step's Next button is apply_form_navigation_nextButton. Why it matters: "first DOM match" is wrong; a hidden earlier step's control matches first. The engine must select the first visible match. (This was the single biggest bug the loop surfaced: all 20 iterations timed out re-clicking a hidden step's Next until the resolver was changed to skip invisible matches.)

  4. Table layouts. Fields sit in <table><tr><td>label</td><td>input</td></tr>. Why it matters: label→input association is positional, not <label for>. Placeholder/aria fallbacks are more reliable than DOM-proximity label matching.

  5. Heavy localization. Visible labels and aria-labels are translated per tenant locale (the harness cycles en/de/fr/es). Why it matters: any fallback that matches on visible English text (button:has-text('Next')) is locale-fragile and must be the last resort. The language-independent signals are the SAP id token and the input type.

Selector + iframe strategy

Each field is an ordered SelectorChain tried until one matches:

firstName:  input[id*='firstName' i]      ← SAP id token (language-independent, primary)
            input[id*='first_name' i]      ← alternate token spelling
            input[aria-label*='first name' i]
            input[name*='first' i]
            input[placeholder*='first' i]  ← positional/table fallback, last

Resolution algorithm (ResolveLocatorAsync):

  1. Build the frame list: main frame first, then every child frame.
  2. For each selector in chain order, for each frame, take the first visible match.
  3. On a hit inside a non-main frame, fire OnIframeSwitch and log the switch.
  4. On a hit at chain index > 0, log which fallback carried the field (SAP token missed).
  5. Emit a SelectorAttempt(label, chainIndex, selector, matched, frameLabel) for every try — this is what feeds the hit/miss + iframe-rate metrics.

Language-independent selectors lead; ARIA/label/placeholder fallbacks trail; visible-text (:has-text) is dead last because it breaks under localization.

Failure modes observed

Mode Cause Hardening applied
TimeoutException on Next (initial) Reused id + "first DOM match" re-clicked a hidden step. Resolve to first visible match.
judgement-call:degree-ambiguity Degree <select> lacks an exact "B.S. Game Development" option (~60% of iterations). Raise a ranked judgement call; lab auto-resolves to top option, production pauses for the user.
Field-not-found on optional fields Address/phone absent in some variations. Optional fields skip quietly; required fields log a warning, loop continues.
Iframe not yet loaded Slow-render variation. Per-step visible-wait tolerates timeout; frame search retries across chain.

Log-and-continue throughout: one iteration's fault never stops the loop.

How SF differs from Workday (practical automation deltas)

Dimension Workday SuccessFactors
Stable hook data-automation-id (purpose-built, fairly stable) No automation attribute. Only long generated ids → substring-token matching.
Frames Mostly single-document React app Frequent iframes per step → must switch frame context.
Account Sign-in exists, but flow is one app Forced candidate account (register/verify/login) before apply — simulated locally here.
Localization Present but automation-ids are English & stable Labels and the hookable text are localized → visible-text selectors are fragile.
Layout Flex/React containers Table layouts; positional label→input.
Degree/EEO Custom button-comboboxes + listbox promptOptions Native <select>/<option> — easier to enumerate, but option text is localized.

Net: SF pushes you harder toward language-independent, token-based, frame-aware resolution than Workday does, because SF gives you no purpose-built automation attribute and scatters fields across iframes.

Hardening checklist (portable to production)

  • Match on id substring tokens, never exact #id.
  • Search all frames; treat iframe-switching as a first-class step, not an edge case.
  • Always pick the first visible match (steps reuse ids).
  • Order fallbacks language-independent → ARIA/label → visible-text last (localization).
  • Treat the forced candidate account as a required pre-apply step.
  • Below the auto-fill confidence threshold (degree, screening), raise a judgement call rather than guess.
  • Log every selector attempt, iframe switch, and judgement call; never fault the whole loop on one field.