Skip to content

Scraper automation learnings — iCIMS (offline lab)

Companion to the Workday / UKG / Greenhouse / Lever learning labs. This one targets iCIMS, the hardest ATS base to automate. Everything here was learned from a fully local, offline harness — no real network, no *.icims.com, no account creation, and no submit ever fired.

What the lab is

tools/WorkWingman.ScraperLab.Icims is a .NET 10 console app that drives a real Playwright browser against a loopback-only fake iCIMS site (FakeIcimsSite, an in-process HttpListener on 127.0.0.1). Each iteration randomizes the markup (iframe topology, element ids, dropdown options, render pacing, page-skip), fills the form from the seed profile (Andrew Jones), logs every selector attempt and iframe switch, resolves the ambiguous-degree judgement call, then at Review & Submit logs would submit (suppressed) without clicking, and restarts.

tools/WorkWingman.ScraperLab.Icims/
  Program.cs                 # CLI (--iterations / --minutes / --seed / --headed / --output)
  LoopRunner.cs              # per-iteration orchestration; log-and-continue
  IcimsAutomationEngine.cs   # THE iframe-aware adapter (drives the browser)
  FakeIcimsSite.cs           # loopback fixture: nested iframes + multi-page + legacy ids
  FixtureVariation.cs        # per-iteration randomization knobs (incl. iframe topology)
  LabMetrics.cs              # JSONL log + selector/iframe hit-miss tallies + failure modes
  LabProfile.cs / LabVault.cs# local fixture profile + in-memory vault stub (no real KeePass)
src/WorkWingman.Infrastructure/Automation/
  IcimsSelectors.cs          # ordered fallback selector chains + degree similarity scoring
tests/WorkWingman.ScraperLab.Icims.Tests/
  IframeSwitchLogicTests.cs          # the iframe-switch decision (pure, browser-free)
  IcimsSelectorAndJudgementTests.cs  # selector philosophy + degree match + judgement gate

Run:

# short verification run
dotnet run --project tools/WorkWingman.ScraperLab.Icims -- --iterations 20

# long local hardening run (headless), reproducible seed
dotnet run --project tools/WorkWingman.ScraperLab.Icims -- --minutes 60 --seed 7

Why iCIMS is the hardest base

Workday hands you a clean single-document React app where every meaningful control carries a stable data-automation-id. iCIMS gives you the opposite on four axes at once:

Dimension Workday iCIMS
Document model single top document form is inside #icims_content_iframe, sometimes a second nested iframe
Element ids stable data-automation-id long generated ASP.NET ids (ctl00_ctl00_..._txtFirstName_field) that churn per control-tree
Layout semantic form groups <table> cells (<td>label</td><td>input</td>) — structural selectors are worthless
Account gate optional sign-in forced candidate-account/social-login before the form
Flow fixed step order multi-page, per-tenant reorder/skip

The single biggest difference is the iframe: page.Locator(sel) resolves against the top document and finds nothing, because the form lives one (or two) frames down. Everything else compounds from there.

The iframe-switching strategy (the core capability)

The engine treats frame resolution as a first-class, logged, fallback-driven step — exactly like field resolution:

  1. Resolve the outer content frame. Walk IcimsSelectors.ContentIframe (#icims_content_iframe → iframe[src*='icims.com'] → iframe[title*='iCIMS'] → …). The first selector whose <iframe> element exists wins. Each attempt is recorded as an IframeSwitch (hit/miss, level 1).
  2. Wait for the frame to actually load. A resolved FrameLocator is lazy; the <iframe> tag can exist while its inner document is still in flight. Probing immediately is the #1 false-negative. WaitForFrameReadyAsync waits for the frame's <body> to attach first.
  3. Probe, then descend only if needed. Try a known field (account email) in the outer frame. If it's missing, descend into a nested inner iframe (IcimsSelectors.InnerIframe: iframe[id*='iApply'], iframe[src*='inner'], …) and re-probe. This is the doubly-nested iApply/portal tenant.
  4. Fall back to the top document. If no content iframe is present at all (some tenants render inline), the engine resolves against the top document (frame == null).

The decision itself is factored into a pure, browser-free method so it is unit-tested without Playwright:

public static FrameContext DecideFrameContext(
    bool hasOuterIframe, bool fieldFoundInOuter,
    bool hasInnerIframe, bool fieldFoundInInner)
=> !hasOuterIframe                        ? FrameContext.TopDocument   // inline tenant
 : fieldFoundInOuter                      ? FrameContext.OuterFrame    // typical iCIMS
 : hasInnerIframe && fieldFoundInInner    ? FrameContext.InnerFrame    // doubly-nested
 : FrameContext.NotFound;                                              // switched in, still missing

Deliberate non-fallback: the inner-iframe chain has no bare iframe selector. In a single-frame tenant the outer content frame is the form; a bare iframe match would "descend" into a phantom and lose the real fields. Descent must only fire for a genuinely nested portal frame.

Over a 20-iteration run the lab confirmed correct resolution across all three topologies: OuterFrame 13, TopDocument 5, InnerFrame 2 — 100% frame-resolution success, with the #icims_content_iframe primary hitting when present and the src/title fallbacks covering the rest.

Selector philosophy — inverted from Workday

Because iCIMS ids are generated, the selector chains lead with a suffix/substring token match, not an exact id:

public static readonly string[] FirstName =
[
    "input[id$='FirstName']",         // primary: stable SUFFIX token
    "input[id*='first_name' i]",      // legacy generated id contains the token
    "input[id*='firstName' i]",
    "input[name*='first' i]",         // table layout → fall back to name/aria
    "input[aria-label*='First Name' i]"
];

A test asserts no field chain leads with a whole-id equality ([id='...']) selector — that would be Workday thinking applied to the wrong ATS. The hit/miss tallies bear this out: exact-suffix "primary" hits and substring "fallback" hits are roughly balanced (e.g. First name 37.7% match with 6 primary / 14 fallback hits), which is exactly why the fallback chain, not the primary alone, is load-bearing on iCIMS.

Two hardening fixes the loop surfaced

The loop earns its keep by finding real bugs an adapter would hit on live iCIMS:

  1. Async frame load. Resolving a FrameLocator and probing it in the same tick fails because the frame document hasn't arrived. Fix: wait for the frame <body> to attach before trusting the frame (WaitForFrameReadyAsync). Tolerant — a timeout re-probes and falls back.
  2. Duplicate ids across pages. iCIMS renders every page's Next control with the same ASP.NET id, and prior pages stay in the DOM hidden via CSS. Naively taking .First grabs a hidden earlier match and the flow stalls. Fix: resolve the first visible element (Locator(sel).Locator("visible=true").First) instead of blindly .First.

The judgement call — same as Workday, different control

The degree control on iCIMS is a plain native <select> (vs Workday's button-combobox), but the ambiguity is identical: the tenant's curated option list frequently has no exact match for "B.S. Game Development". The engine reads the options, scores each with a token-overlap + abbreviation-aware similarity (IcimsSelectors.SimilarityScore, with a B.S. ↔ BS bridge so dotted abbreviations match), and if nothing clears the confidence threshold it raises a JudgementCall instead of guessing. In the lab the loop auto-resolves to the top-ranked option and records the decision; in production the run genuinely pauses for the user. 13 of 20 iterations raised the degree judgement call — the expected rate given the randomized option lists.

Failure modes observed

  • judgement-call:degree-ambiguity — expected, the whole point; auto-resolved and logged.
  • frame-unresolved:<mode> — early runs, before the async-load + no-phantom-descend fixes.
  • exception:TimeoutException — early runs, from the duplicate-id .First stall (fixed with visible-locator resolution) and from relative iframe src resolving to the wrong route (fixed by using absolute src).

After the fixes: 20/20 complete, 0 faults, every iteration logging would submit (suppressed).

Safety (by construction)

  • The only host ever contacted is http://127.0.0.1:<port> served in-process by FakeIcimsSite.
  • No real network egress, no *.icims.com, no DNS to anything real.
  • The forced candidate-account step is SIMULATED locally — SimulateCreateAccountAsync fills a loopback gate whose "create account" link is a client-side no-op; no real account is created.
  • The final submit is never clicked. The loop locates the submit control (to prove it exists), then logs would submit (suppressed). Even the fixture's submit button is a client-side return false no-op with no endpoint.
  • AtsDetector (reused from production, unchanged) classifies the fixture as AtsKind.Icims via the icims.com marker, confirming the detection path without touching a real site.

Reusing this against production iCIMS

IcimsSelectors and the iframe-switch strategy in IcimsAutomationEngine are the transferable artifacts. To promote them: move the engine into WorkWingman.Infrastructure.Automation alongside WorkdayAutomationEngine, keep the IFrameLocator?-threading resolution model (null = top document), and keep the submit control locked to user review exactly as the lab enforces.