TRAVERSAL LAB — Indeed (`WorkWingman.ScraperLab.Indeed`)¶

Scope: Indeed as ONE HOP in the job-link resolution chain (matches the Adzuna→ZipRecruiter→ Spectrum→iCIMS pattern in the brief). This lab's only job is: given an Indeed URL or a bare jk, decide (a) does this posting stay on Indeed ("Easily apply" / "Apply with Indeed"), in which case Indeed is terminal, or (b) does it leave to the employer's real ATS ("Apply on company site"), in which case follow exactly one more hop and hand the landed host off to the vendor-specific lab (Icims/Google/Meta/Microsoft/etc. already in this repo) for posting-id extraction and form-filling. This lab does NOT fill forms and does NOT own vendor posting-id regexes beyond a first-pass guess.

Evidence discipline¶

Everything under OBSERVED was captured this session via an ordinary headful Chromium browser pane (Claude_Browser preview tab), no stealth, no CAPTCHA interaction, no header/UA spoofing — i.e. exactly the posture the constraints require ("a real browser as itself"). Real jk values, hrefs, and landed URLs are reproduced verbatim below. Everything under INFERRED is prior/general knowledge and is flagged as needing a dedicated measurement pass before being trusted as a hard rule.

1. TRAVERSAL PHILOSOPHY¶

Indeed is not a single host class — it is two host classes wearing one domain, and the button the job actually renders is the only trustworthy discriminator between them. Every other signal (page text, "Easily apply" as a concept, company name, presence of an apply-shaped element at all) is either absent, unstable, or actively misleading. Structurally: Indeed Apply ("Easily apply") postings terminate inside Indeed's own hosted apply pipeline and never navigate the browser away from indeed.com; "Apply on company site" postings route through an Indeed-owned redirector (/applystart?jk=...) before landing on the employer's real ATS or a vanity domain fronting one. Guessing this from anything other than the rendered apply control (e.g. from the presence of the "Easily apply" text badge in search results, which we measured has no stable attribute) risks the exact failure mode the brief warns about: claiming an ATS that doesn't exist, or missing the one that does.

Layered on top of that: Indeed itself sits behind Cloudflare bot-management that is measured to be intermittent and session-state-dependent, not a hard permanent wall like Adzuna's CloudFront 403. That changes the traversal posture from Adzuna's "assume blocked, use the app's real headful browser at human pace, or fail closed" to "assume reachable, detect the challenge screen explicitly every hop, and fail closed the instant it appears" — never treat one successful load as proof the next one will succeed too.

2. ORDERED RULE CHAIN¶

Entry point is either (A) a bare jk WorkWingman already holds (e.g. a saved JobSource.Indeed listing, or a jk surfaced by an upstream aggregator hop) or (B) an arbitrary Indeed URL landed on from another host. Normalize to (A) first, then walk the chain below.

Step 0 — normalize to a jk. Most specific to least specific, because sponsored links hide the id in an opaque payload that must never be parsed:

// Playwright-compatible, in priority order. STOP at the first one that yields a value.
"a[data-jk]"                    // read the data-jk ATTRIBUTE, never the anchor's href text
"a[href*='/rc/clk?jk=']"        // fallback: jk IS present in the querystring here
"a[href^='/viewjob?jk=']"       // fallback: canonical detail-page link shape
// NEVER attempt to parse jk out of "/pagead/clk?..." — sponsored redirect hrefs carry an opaque
// 'ad=' blob with no jk in cleartext. Measured: zero of the sampled /pagead/clk hrefs contained
// "jk=" anywhere in the querystring, even though the SAME anchor's data-jk attribute had it.

Step 1 — reach the detail page and read the apply control. Navigate to https://www.indeed.com/viewjob?jk={jk}, then in strict order:

"[data-testid='indeedApplyButton-test']"   // FIRST: presence alone means TERMINAL AT INDEED.
                                            // Measured text "Apply with Indeed", href=null (JS-driven,
                                            // no navigation). Do not go further — there is no
                                            // employer ATS to resolve for this posting.
"button:has-text('Apply on company site')" // SECOND, only checked if the above is absent. Read this
                                            // element's own href ATTRIBUTE (unusual: it is a <button>,
                                            // not an <a>, but it carries href="https://www.indeed.com
                                            // /applystart?jk=...&from=vj&...").

Reason for the ordering: data-testid is a stable, framework-emitted hook; button text is locale-fragile prose that a copy change or i18n build can silently break, so it is only the fallback discriminator, never the primary one, and it is only trusted once the stable hook has already been checked absent.

Step 2 — follow the Indeed-owned redirector exactly once. Navigate to the /applystart?jk=... URL captured above. Do not treat this as "the answer" — it is still Indeed's own domain. Read the post-navigation location.href.

// Measured, verbatim, both same session:
// jk=cba7a72b4aa6c5b3 -> https://jobs.lever.co/thinkahead/b044c4f0-4509-425f-8581-58070ded2379?lever-source=Indeed
// jk=8f4942de0502d08e -> https://careers.vituity.com/job/26000182/Software-Engineer-I-Remote-Nationwide?biD=1702

Step 3 — classify the landed host. This is where the Adzuna-chain host classes apply, exactly as in the brief:

Host is a known ATS vendor domain directly (*.lever.co, *.myworkday.com, *.icims.com, boards.greenhouse.io, etc.) → class ats, hand off to that vendor's lab now.
Host is neither Indeed nor a known ATS vendor domain (e.g. careers.vituity.com) → class employer (vanity domain). Do NOT guess the vendor from page text (see negative rules). Look for the SAME anchor-href-host signal the brief's Spectrum trap prescribes:

// Scoped to the primary "Apply"/"Apply Now" control on the LANDED page, not the whole document:
"a[href*='.myworkday.com/'], a[href*='.icims.com/'], a[href*='.greenhouse.io/'], " +
"a[href*='smartrecruiters.com/'], a[href*='.ashbyhq.com/'], a[href*='taleo.net/']"
// If found: that anchor's HOST is the vendor. Hand off to that vendor's lab.
// If not found (as measured on careers.vituity.com — no outbound anchor to any ATS host at all):
// STOP. Emit host class 'employer', vendor Unknown. Do not scan page TEXT for vendor names as a
// substitute — that is exactly the first-marker-wins failure mode the brief measured on Spectrum.

3. NEGATIVE RULES¶

Never resolve a vendor from page TEXT. Measured directly on careers.vituity.com: the rendered HTML contains the literal string "taleo" (case-insensitive substring hit), but a scan of every outbound <a href> on the page found zero anchors to any taleo.net (or any other ATS vendor) host. Text-content scanning would have manufactured a Taleo answer with no evidence backing it — the same failure class as the brief's Lever/iCIMS/Radancy/TalentBrew four-marker trap on jobs.spectrum.com. Only an anchor HOST counts as evidence.
Never parse jk out of a /pagead/clk?... href. It is not there in cleartext; the id lives in an opaque ad= payload meant for Indeed's own ad server. Use data-jk instead (Step 0).
Never treat "Easily apply" (the search-results-list badge) as the termination signal. Measured: it renders as a bare <div> with no data-testid, no stable class, text-match only, and it is a DIFFERENT string ("Easily apply") from the detail-page ground truth ("Apply with Indeed" / data-testid="indeedApplyButton-test"). Use it at most as a cheap pre-filter hint on the results list, never as the final answer — always confirm on the detail page.
Never click, dismiss, or programmatically satisfy the Cloudflare "Additional Verification Required" checkbox. Measured directly this session on a cold GET /jobs search: Indeed served an interactive Verify you are human challenge (Ray ID captured, checkbox present). Per the hard line, do not click it. Treat its appearance as an unconditional STOP for that navigation — retry later or fail closed to Unknown, never automate past it.
Never assume the first successful Indeed load guarantees the next one succeeds. The challenge above was intermittent within the SAME session — a /jobs search was challenged, a later /jobs search on the same tab was not. Check for the challenge marker on every hop, not just the first.
Never treat /applystart?jk=... itself as the terminal URL. It is still indeed.com — it is a redirector, not the answer. The answer is the host the browser is on after that navigation resolves.
Never invent a posting id for the landed employer page from a guessed regex pattern shared with a different vendor. See section 6 — mark Unknown rather than force a pattern that hasn't been confirmed for that specific vendor via an Indeed-originated hop.

4. TERMINATION PREDICATE¶

IF Cloudflare challenge marker present at any hop         -> FAIL CLOSED (Unknown, class=blocked)
ELSE IF jk cannot be resolved via Step 0 chain             -> FAIL CLOSED (Unknown)
ELSE IF [data-testid='indeedApplyButton-test'] present     -> TERMINAL, class=portal-like,
                                                               vendor=Indeed, no further hop
ELSE IF button text == "Apply on company site" (href found)-> HOP via /applystart, then:
    IF landed host == indeed.com (still)                   -> FAIL CLOSED (redirector didn't resolve)
    ELSE IF landed host matches a known ATS vendor pattern  -> TERMINAL, class=ats, hand off to
                                                               that vendor's lab
    ELSE IF an outbound anchor on landed page resolves to a
         known ATS vendor host (Spectrum-trap check)        -> TERMINAL, class=employer→ats,
                                                               hand off using the ANCHOR's host,
                                                               never page text
    ELSE                                                     -> TERMINAL, class=employer,
                                                               vendor=Unknown (correct, not a bug —
                                                               e.g. careers.vituity.com as measured)
ELSE (neither apply control found at all)                  -> FAIL CLOSED (Unknown) — page shape
                                                               changed under us, do not guess

5. TEST ASSERTIONS (house style, xunit, real measured URLs)¶

using WorkWingman.Core.Models;
using WorkWingman.ScraperLab.Indeed;

namespace WorkWingman.ScraperLab.Indeed.Tests;

/// <summary>
/// Covers the Indeed traversal philosophy: the apply CONTROL (not the "Easily apply" list badge,
/// not page text) is the only trustworthy signal for whether Indeed is terminal or a one-hop
/// redirector to the employer's real ATS. Real jk values and landed hosts were captured live this
/// session via an ordinary headful browser (see lab-indeed.md for provenance).
/// </summary>
public class IndeedTraversalSelectorTests
{
    [Fact]
    public void Jk_extraction_reads_the_data_jk_attribute_first_never_the_pagead_href()
    {
        // Sponsored cards route through /pagead/clk?... whose querystring never contains "jk=" —
        // the id only exists in the anchor's data-jk attribute.
        Assert.StartsWith("a[data-jk]", IndeedSelectors.JkChain[0]);
        foreach (var chain in new[] { IndeedSelectors.JkChain })
            Assert.DoesNotContain(chain, s => s.Contains("pagead"));
    }

    [Fact]
    public void Terminal_signal_is_the_stable_testid_not_the_locale_fragile_button_text()
    {
        Assert.Equal("[data-testid='indeedApplyButton-test']", IndeedSelectors.EasilyApplyChain[0]);
        // Text-based "Apply on company site" match must be checked SECOND, never first.
        Assert.True(
            Array.IndexOf(IndeedSelectors.ApplyOnCompanySiteChain, "button:has-text('Apply on company site')")
            >= 0);
    }

    [Theory]
    [InlineData("indeedApplyButton-test", true)]   // measured: TherapyNotes.com posting (jk=7b993d015f4b257a)
    [InlineData(null, false)]                       // measured: AHEAD posting (jk=cba7a72b4aa6c5b3)
    public void Presence_of_indeed_apply_testid_alone_decides_terminal_vs_hop(string? testId, bool expectTerminal)
        => Assert.Equal(expectTerminal, IndeedTraversal.IsTerminalAtIndeed(testId));

    [Theory]
    [InlineData("https://jobs.lever.co/thinkahead/b044c4f0-4509-425f-8581-58070ded2379?lever-source=Indeed", "lever.co", true)]
    [InlineData("https://careers.vituity.com/job/26000182/Software-Engineer-I-Remote-Nationwide?biD=1702", "vituity.com", true)]
    [InlineData("https://www.indeed.com/applystart?jk=x", "indeed.com", false)] // redirector didn't resolve -> not a valid landing
    public void Landed_host_after_applystart_is_never_indeed_itself(string landedUrl, string expectedHostSuffix, bool shouldBeValidLanding)
    {
        var host = new Uri(landedUrl).Host;
        Assert.EndsWith(expectedHostSuffix, host);
        Assert.Equal(shouldBeValidLanding, host != "www.indeed.com" && host != "indeed.com");
    }

    [Fact]
    public void Vituity_landing_must_NOT_resolve_to_taleo_from_page_text_alone()
    {
        // NEGATIVE TEST — the core trap this lab exists to prevent.
        // Measured: the rendered HTML of careers.vituity.com contains the literal substring
        // "taleo", but zero outbound <a href> anchors point at any taleo.net host. A resolver that
        // scans page text would wrongly claim Taleo. The correct, evidence-backed answer is
        // Unknown, exactly like the brief's Spectrum/Lever four-marker trap.
        var landedHtmlContainsTaleoText = true;   // measured fact, reproduced from the captured DOM
        var outboundAnchorsToTaleoHost = Array.Empty<string>(); // measured: none found
        var resolved = IndeedTraversal.ResolveVendorFromAnchorsOnly(
            outboundAnchorsToTaleoHost, textMentionsVendor: landedHtmlContainsTaleoText);

        Assert.Null(resolved); // NOT "Taleo" — text mentions must never be load-bearing
    }

    [Fact]
    public void Cloudflare_challenge_marker_fails_closed_and_never_auto_clicks_the_checkbox()
    {
        // Measured: a cold GET to /jobs served an "Additional Verification Required" interstitial
        // with a "Verify you are human" checkbox and a Ray ID. The lab must stop, not solve it.
        var pageTitle = "Just a moment...";
        var bodyText = "Additional Verification Required";

        Assert.True(IndeedTraversal.IsBotChallenge(pageTitle, bodyText));
        // No selector in this lab may target the challenge checkbox for interaction.
        Assert.DoesNotContain(IndeedSelectors.EasilyApplyChain, s => s.Contains("Verify"));
        Assert.DoesNotContain(IndeedSelectors.ApplyOnCompanySiteChain, s => s.Contains("Verify"));
    }
}

Minimal production-side skeleton the tests above assume (deterministic, no LLM calls):

namespace WorkWingman.ScraperLab.Indeed;

public static class IndeedSelectors
{
    public static readonly string[] JkChain =
    [
        "a[data-jk]",
        "a[href*='/rc/clk?jk=']",
        "a[href^='/viewjob?jk=']",
    ];

    public static readonly string[] EasilyApplyChain =
    [
        "[data-testid='indeedApplyButton-test']",
    ];

    public static readonly string[] ApplyOnCompanySiteChain =
    [
        "button:has-text('Apply on company site')",
    ];
}

public static class IndeedTraversal
{
    public static bool IsTerminalAtIndeed(string? indeedApplyTestId) => indeedApplyTestId is not null;

    public static bool IsBotChallenge(string pageTitle, string bodyText) =>
        pageTitle.Contains("Just a moment", StringComparison.OrdinalIgnoreCase) ||
        bodyText.Contains("Additional Verification Required", StringComparison.OrdinalIgnoreCase);

    /// Vendor is resolved ONLY from anchor hosts. A text mention is logged for telemetry but is
    /// never sufficient on its own — this is the direct fix for the Spectrum/Vituity trap class.
    public static string? ResolveVendorFromAnchorsOnly(string[] outboundAnchorHostsMatchingKnownVendors, bool textMentionsVendor)
        => outboundAnchorHostsMatchingKnownVendors.Length > 0 ? outboundAnchorHostsMatchingKnownVendors[0] : null;
}

6. POSTING ID EXTRACTION¶

Source	Pattern	Confidence	Basis
Indeed itself	`jk` value from `data-jk` / `?jk=` / `<link rel="canonical" href="https://www.indeed.com/viewjob?jk=...">`	High	OBSERVED — all three sources agreed on the same value in every sample this session; canonical tag independently confirms Indeed treats `jk` as its own stable id.
Lever (via `/applystart` hop)	UUID path segment: `jobs.lever.co/{tenant}/{uuid}`	High	OBSERVED once (`jobs.lever.co/thinkahead/b044c4f0-4509-425f-8581-58070ded2379`), and matches Lever's well-known posting-id shape used elsewhere in this repo's labs. Still only one live sample from this hop — treat as high confidence on shape, not yet volume-tested from Indeed specifically.
Taleo-style vanity career site (e.g. `careers.vituity.com`)	numeric id in `/job/{id}/...`, e.g. `/job/26000182/...?biD=1702`	UNKNOWN / provisional	OBSERVED exactly once, on one tenant, with one URL shape. Do not generalize to "all Taleo-branded pages use `/job/{id}`" — the `taleo` marker was TEXT-ONLY in this sample (see negative rule 1), so the vendor itself is not even confirmed, only the domain and a plausible requisition id. Needs a second independent sample before this graduates past Unknown.
Any other vendor reached via `/applystart` (Workday, iCIMS, Greenhouse, SmartRecruiters, Ashby, BambooHR, ...)	—	UNKNOWN — out of scope for this lab	Not observed via an Indeed hop this session. Once Step 3 resolves the landed host to one of these vendors, hand off to that vendor's own lab in this repo (`WorkWingman.ScraperLab.{Vendor}`), which already owns posting-id extraction and form-filling for it. This lab's job stops at "here is the vendor host, here is the evidence anchor."

OBSERVED vs INFERRED — explicit split¶

OBSERVED (measured this session, ordinary headful browser, no bypass, verbatim values captured): - Cloudflare "Additional Verification Required" challenge occurred once on a cold /jobs search; a later /jobs search in the same tab succeeded without it (intermittent, session-state-linked). - a[data-jk] is present and correct on every sampled search-result card; href shape varies (/pagead/clk?... with no jk in cleartext, /rc/clk?jk=..., /viewjob?jk=...). - "Easily apply" search-list badge is a bare <div>, text-match only, no stable attribute. - Detail-page discriminator: [data-testid='indeedApplyButton-test'] (stays on Indeed) vs a <button> reading "Apply on company site" carrying href="https://www.indeed.com/applystart?jk=...". - /applystart?jk=... resolved in one hop to jobs.lever.co/thinkahead/{uuid}?lever-source=Indeed and separately to careers.vituity.com/job/26000182/...?biD=1702. - careers.vituity.com page HTML contains the text "taleo" but zero outbound anchors to any taleo.net (or other ATS) host. - Indeed detail page carries a schema.org JobPosting JSON-LD block and a canonical <link> pinned to viewjob?jk=.... - /viewjob?jk=<invalid> returns Indeed's own branded 404, not a Cloudflare challenge.

INFERRED (prior/general knowledge, NOT measured today — needs its own verification pass): - The Cloudflare challenge's exact trigger conditions (IP/fleet reputation, request rate, geo) are unknown from a single session; treat every hop as independently challengeable. - The Taleo-style /job/{numericId} shape is a one-sample pattern, not a confirmed Taleo rule. - Posting-id shapes for Workday/iCIMS/Greenhouse/SmartRecruiters/Ashby/BambooHR reached via an Indeed hop specifically are unverified here; this lab defers to each vendor's existing lab. - Whether "Apply on company site" landings ever require a warmed cookie/consent context (as the brief measured for Adzuna's aztt JWT) was only weakly suggested here (a cookie-settings banner was present on the Vituity landing, full apply CTA did not render in the captured snapshot) and was not independently confirmed as a hard block — flag for a follow-up measurement pass.

TRAVERSAL LAB — Indeed (WorkWingman.ScraperLab.Indeed)¶