Skip to content

ATS traversal strategy (WING-333)

Authored by Jenny (Gemini 3.1 Pro High) on a brief built from measured findings, 2026-08-05. Posting-ID regexes VERIFIED against live producer APIs where possible: Greenhouse (7995199 via careers.airbnb.com vanity + gh_jid), Ashby (uuid), iCIMS (77665) all MATCH the id the vendor's own API reports. Lever UNTESTED — its public API returned an HTTP error during verification. Phenom deliberately left UNKNOWN rather than guessed.

1. Anchor-Scan Rules

When analyzing a page's HTML without a browser, extract all <a> tags with an href attribute. Exclude mailto:, tel:, and javascript: schemes. Resolve relative URLs using the page's base URI.

Rank the extracted anchors using the following precedence rules (Highest to Lowest). The highest-scoring anchor that meets the criteria is chosen as the apply URL.

  • Priority 1: Known ATS Shape + "Apply" Intent
    • Condition: The href matches a known ATS URL pattern (e.g., *.icims.com/jobs/*, boards.greenhouse.io/*) AND the anchor's inner text, aria-label, or title contains apply, submit, or application (case-insensitive).
    • Action: Immediately select.
  • Priority 2: Known ATS Shape Only
    • Condition: The href matches a known ATS URL pattern, but lacks explicit "apply" text (e.g., an icon link or a generically named button).
    • Action: Select, provided no Priority 1 links exist. If multiple different known ATS URLs exist (e.g., one to iCIMS, one to Lever), return Unknown (Ambiguous).
  • Priority 3: Aggregator / Tracking Handoff
    • Condition: The href does not match an ATS, but the anchor text is exactly Apply Now or Apply (ignoring whitespace/case).
    • Action: Select for next-hop traversal. (Must verify it's not a generic /how-to-apply page by ensuring the URL has query parameters or dynamic path segments).
  • Exclude/Discard:
    • Anchor text contains: how to apply, application process, learn more, contact us, login.
    • href contains: /search-jobs, /subscribe, /saved-jobs, /alerts.

2. Posting ID Extraction

Use the following URL shapes and regexes to extract the stable posting ID. Note: Always run the regex against the path and query string, not the entire URL string, to avoid matching tenant IDs or hostnames.

ATS URL Shape Regex (Group 1 = ID) Notes
Greenhouse boards.greenhouse.io/{tenant}/jobs/{id}
.../embed/job_app?gh_jid={id}		 (?:/jobs/\|\?gh_jid=\|token=)([0-9]+) IDs are numeric.
Lever jobs.lever.co/{tenant}/{id} /([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) UUID format.
Ashby jobs.ashbyhq.com/{tenant}/{id} /([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}) UUID format.
iCIMS careers-{tenant}.icims.com/jobs/{id}/... /jobs/(\d+)/ IDs are numeric.
Workday {tenant}.myworkdayjobs.com/.../job/{loc}/{slug}_{id} (?:/job/[^/]+/[^/]+_)(JR\d+\|[A-Z0-9-]+)$ Usually prefixed with JR (e.g., JR12345).
Workable apply.workable.com/{tenant}/j/{id} /j/([A-Z0-9]+) Alphanumeric (e.g., 0123456789A).
Jobvite jobs.jobvite.com/{tenant}/job/{id} /job/([a-zA-Z0-9]+) Alphanumeric.
SmartRecruiters jobs.smartrecruiters.com/{tenant}/{id}-{slug} /([0-9]{15,})- IDs are typically 15+ digit numeric strings.
Phenom careers.{tenant}.com/.../job/{id} UNKNOWN Highly variable per tenant. Cannot reliably extract without tenant-specific rules.

3. HTTP vs. Browser Decision Rule

Evaluate the plain HTTP GET response to decide if Playwright is necessary.

  • HTTP is Sufficient (Stop and Parse):
    • Status is 200 OK.
    • Content-Length > 10 KB (or body length > 10,000 characters).
    • The DOM contains identifiable <a> tags with href attributes.
    • The body contains plain text matching job description content (e.g., "responsibilities", "requirements").
  • Browser Required (Escalate to Playwright):
    • Response is a JS-only SPA shell: <div id="root"></div> or <app-root></app-root> with little to no static body text.
    • The only "Apply" buttons found are <button> elements with onclick handlers and no associated <a> tags or href attributes.
    • The target ATS is known to rely on heavily dynamic rendering (e.g., Workday forms).

4. Stop Conditions and Fail-Closed

Halt traversal immediately and return Unknown if any of the following occur:

  • Bot Protection / WAF Detected: HTTP 403, or HTTP 200 with <title>Just a moment...</title>, window._cf_chl_opt (Cloudflare), Incapsula, or datadome in the response body. Do not escalate to browser.
  • Login Walls: The resolved URL path contains /login, /signin, or auth., OR the page title contains "Log In" or "Sign In".
  • Search/Listing Page Detected: The page contains > 3 distinct known ATS Apply URLs pointing to different posting IDs. This indicates a search index, not a single job posting.
  • Consent Interstitials (Blocking): The URL path explicitly points to a /cookie-consent or /privacy-policy interstitial.
  • HTTP Errors: HTTP 400-499 (excluding WAFs) or 500-599.
  • Ambiguity: Multiple known ATS URLs are found for different vendors on the same page, and priority rules cannot safely break the tie.

5. Loop and Trap Avoidance

  • Hop Caps: Hard limit of 5 HTTP 3xx redirects per request, and 3 logical page traversals (e.g., Aggregator $\rightarrow$ Vanity $\rightarrow$ ATS).
  • Cycle Detection: Maintain a HashSet<string> of normalized absolute URLs (stripped of fragments and utm parameters). If a URL is encountered twice, abort and return Unknown.
  • Aggregator Trap: If an employer's vanity page contains outbound links to known other aggregators (e.g., ZipRecruiter, Indeed) instead of ATS vendors, treat the page as a secondary aggregator. Do not traverse further.

6. Playwright Specifics

When escalating to a browser, use the following minimal sequence to avoid timeouts and fragile selectors.

  1. Launch: Use standard user agent and viewport. Do not inject stealth scripts (violates constraint).
  2. Navigation: await page.GotoAsync(url, new PageGotoOptions { WaitUntil = WaitUntilState.DOMContentLoaded }); (Never wait for NetworkIdle).
  3. Target Acquisition: Wait for the job description or apply button to appear: await page.WaitForSelectorAsync("text=/Apply/i", new PageWaitForSelectorOptions { Timeout = 5000 });
  4. Interaction: 
    // Prepare to catch either a new tab or a cross-origin top-level navigation
var newPageTask = context.WaitForPageAsync();
await page.ClickAsync("button:has-text('Apply'), a:has-text('Apply')");
  5. Handoff Detection:
    • If a new tab opens: Check the URL of the newPageTask result.
    • If no tab opens: Wait up to 3000ms for page.Url to change. If page.Url changes to a cross-origin domain (e.g., jobs.spectrum.com $\rightarrow$ careers-charter.icims.com), handoff is successful. Extract the URL.
    • If URL stays the same: Check for injected iframes (e.g., Greenhouse embedded): page.Frames.FirstOrDefault(f => f.Url.Contains("boards.greenhouse.io")). Extract iframe src if found.