Skip to content

Scraper Automation Learnings — Greenhouse (offline lab)

This document captures what the offline Greenhouse learning harness (tools/WorkWingman.ScraperLab.Greenhouse) taught us about automating Greenhouse-powered application forms, and how a production adapter should target Greenhouse markup.

Everything here was learned locally. The harness never touches boards.greenhouse.io, never opens a real Greenhouse tenant, never creates an account, and never submits. The only site involved is a loopback HttpListener on 127.0.0.1 (FakeGreenhouseSite) that reproduces Greenhouse's public markup patterns. The submit button (#submit_app) is located but never clicked; even if clicked it is a client-side no-op with no network call.

1. What the lab is

Piece File Role
Fake site FakeGreenhouseSite.cs Loopback HTTP server rendering a realistic single-page Greenhouse apply form; parameterized per iteration.
Variation SiteVariation.cs Randomizes field ids / labels / required-ness / iframe embed / hydration delay / which custom questions appear.
Selectors GreenhouseSelectors.cs Ordered fallback selector chains (most-specific first) + the SelectorAttempt record.
Engine GreenhouseAutomationEngine.cs Drives the fixture via Playwright: resolves the form frame, fills core fields, attaches resume, answers custom questions, raises judgement calls, locates but never clicks submit.
Metrics LabMetrics.cs Rolling JSONL log + selector hit/miss tallies + failure-mode counts.
Loop LoopRunner.cs / Program.cs Iterates against the fixture, log-and-continue, --iterations N / --minutes M.
Profile SampleProfile.cs Local IntakeProfile (Andrew Jones) reusing the real WorkWingman.Core shapes.

Run:

dotnet run --project tools/WorkWingman.ScraperLab.Greenhouse -- --iterations 20
# long local soak:
dotnet run --project tools/WorkWingman.ScraperLab.Greenhouse -- --minutes 60 --seed 7

2. Greenhouse markup patterns to target (and WHY)

Greenhouse's application form is far simpler and more semantic than Workday's. From public knowledge, a Greenhouse apply page has these stable properties, which the adapter targets:

  • Single page. All fields render at once inside one <form id="application_form">. There is no multi-step wizard, no "Save and Continue", no per-step navigation. Why it matters: the adapter does not need step orchestration or "wait for next step" logic — it fills everything, then locates submit.

  • Core fields use plain semantic ids. #first_name, #last_name, #email, #phone, and the resume file input #resume. Names follow job_application[first_name], etc. Why it matters: the primary selector is a direct #id — cheap and reliable when present. This is the opposite of Workday, which uses opaque data-automation-id attributes.

  • Resume is a file <input type="file" id="resume">, often alongside a "paste" <textarea>. Why it matters: attach with SetInputFiles on the file input specifically — see the wrapper-div hazard in §4.

  • Custom questions are job_application[answers_attributes][N][...] inputs. Each has a hidden [question_id], and a value field that is one of [text_value] (free text), [answer_selected] (select), or a boolean. The only reliable anchor is the visible <label> text — the numeric index N shifts from posting to posting and the question_id is opaque. Why it matters: custom questions must be resolved by label (page.GetByLabel(...)), never by id or index. This is the single most important Greenhouse-specific rule.

  • Iframe embed. A company careers page frequently embeds the board via <iframe id="grnhse_app" src="https://boards.greenhouse.io/embed/...">. The form lives inside the iframe, not the top document. Why it matters: the adapter must resolve the correct frame first (ResolveFormFrame) and run all locators against that frame. A top-frame-only adapter silently finds nothing on embedded postings.

  • Submit is #submit_app. Why it matters: a single canonical id, easy to locate — but in the lab it is never clicked.

3. How Greenhouse differs from Workday (the core lesson)

Dimension Workday Greenhouse
Flow Multi-step wizard (Sign In → My Info → Experience → Education → EEO → Review) Single page — all fields at once
Field anchor data-automation-id (opaque, React-emitted) Plain semantic id (#first_name) + name
Account Usually requires sign-in / account creation Usually no account — apply directly
Custom questions Structured form fields per step answers_attributes[N] arrays, label-anchored only
Dropdowns Custom button + listbox (promptOption) Native <select> (SelectOptionAsync works directly)
Embedding Own domain (*.myworkdayjobs.com) Often an #grnhse_app iframe embed
Degree/education Ambiguous listbox → judgement call Sometimes a native <select>, sometimes absent
Adapter complexity Step orchestration + wait-for-step Frame resolution + label-driven custom questions

Takeaway: the Workday adapter's hard part is navigation and opaque ids; the Greenhouse adapter's hard part is frame resolution and label-anchored custom questions. The selector strategy inverts: Workday starts from data-automation-id, Greenhouse starts from #id and falls back to label.

4. Selector findings (from the loop)

The engine tries an ordered fallback chain per field, most-specific first, and records every attempt (SelectorAttempt { Label, ChainIndex, Selector, Matched }). ChainIndex == 0 is a primary (canonical-id) hit; anything higher is a fallback.

Core-field chain (GreenhouseSelectors.CoreCssChain):

0: #first_name                       canonical id
1: [id*='first_name']                renamed/suffixed id (#first_name_input)
2: [name*='[first_name]']            name attribute (job_application[first_name])
3: [name*='first_name']              looser name-substring
+  getByLabel('First Name')          last-resort label fallback

Findings from a 20-iteration run (seed 1337, all 20 completed, 0 faulted):

  • Fallbacks fire often and are necessary. Roughly a third of core-field resolutions came from a fallback link rather than the canonical #id, because the variation renames or drops ids. A single hard-coded #first_name would have missed those. The label fallback is the safety net that keeps the match rate at 100% of present fields.

  • The [id*=] substring hazard (a real bug the lab caught). The first version of the resume chain used [id*='resume']. On the fixture that matched the wrapper <div id="resume_field"> before the file input, and SetInputFiles then threw Node is not an HTMLInputElement — faulting half the iterations. Fix: scope substring id-matches by element type, e.g. input[type='file'][id*='resume'], never a bare [id*='resume']. Lesson: a loose *= id match will happily grab a container. Always constrain the tag.

  • Custom questions matched 100% via label (getByLabel), confirming the label-anchored approach. Index/id-based matching was never needed and would have been brittle.

  • Native <select> works with SelectOptionAsync. No custom-listbox dance is needed (unlike Workday's promptOption).

5. Failure modes observed

  • judgement:unmapped-custom-question — a custom question whose label the heuristic couldn't confidently map to a profile answer (e.g. "Why do you want to work here?", or an unfamiliar yes/no). In the lab this is recorded and auto-resolved (top option / blank); in production it becomes a genuine pause for the user. This is the expected, healthy failure mode — the adapter should never guess an answer to an unmapped question.
  • judgement:option-not-offered — a dropdown whose desired label the posting doesn't offer (e.g. education level phrased differently). Raised, not forced.
  • judgement:resume-input-not-found / field-not-found — the fallback chain exhausted without a hit. Optional fields (phone) log-and-skip; required fields raise.
  • exception:* (Playwright) — caught by the log-and-continue loop; one iteration's fault never stops the run. (The Node is not an HTMLInputElement case in §4 was one of these until fixed.)

6. Hardening approach

  1. Resolve the frame first. Always call ResolveFormFrame and run every locator against the resolved frame, so embedded (#grnhse_app) and direct-board postings behave identically.
  2. Chain, don't hard-code. Primary #id → substring id (tag-scoped!) → name → label. Record every attempt so hit/miss telemetry shows which anchor is actually load-bearing per tenant.
  3. Custom questions by label, always. Never trust the answers_attributes[N] index or the opaque question_id.
  4. Tag-scope every *= selector. A bare [id*='x'] can match a wrapper; scope it to the expected element (input[...], button[...]).
  5. Judgement over guessing. Unmapped custom questions and unoffered dropdown options raise a judgement call rather than submitting a wrong answer.
  6. Log-and-continue. Every iteration is independent; failures are recorded and the loop proceeds.
  7. Submit is located, never clicked. The lab proves the adapter can find #submit_app without ever transmitting anything.

7. Safety statement

  • Loopback only: http://127.0.0.1:<random-port>, bound via HttpListener to IPAddress.Loopback.
  • No real Greenhouse, no boards.greenhouse.io, no account creation, no network egress of any kind. The engine performs local fill-input only.
  • The /submit fixture endpoint is a no-op that returns a "nothing was submitted" page and is never requested by the loop. The #submit_app button is a client-side no-op.
  • The profile is local fixture data (Andrew Jones) and is never transmitted.