TRAVERSAL LAB — resolving an Adzuna API result to a drivable employer ATS url¶

Companion to the per-ATS ScraperLabs (tools/WorkWingman.ScraperLab.{Icims,Meta,...} + tests/WorkWingman.ScraperLab.{Icims,Meta,...}.Tests). Those labs answer "how do I fill a form on a KNOWN ats host." This lab answers the question that has to be solved before that one: "which host is the ats, and how do I legitimately get there from an Adzuna search result." Same house shape — a static, ordered rule-chain class per surface, tests that assert the philosophy of the chain (so a bad reorder fails), fake offline fixtures, no LLM calls in the hot path.

Grounded directly in the measured facts already captured in src/WorkWingman.Infrastructure/Automation/host-taxonomy.seed.json (WING-333) and the detection rules in AtsDetector.cs (DetectFromUrl, DetectFromDescriptionPage, ApplyUrlMarkers, HostMatchesMarker). This lab does not re-implement ATS-host detection — it reuses AtsDetector as the terminal-state oracle and adds the layer above it: classifying every non-ats host encountered on the way (aggregator / employer / portal / blocked) and driving the walk between them.

File layout (mirrors the house pattern)¶

tools/WorkWingman.ScraperLab.Traversal/
  WorkWingman.ScraperLab.Traversal.csproj      # Exe, net10.0, IsPackable=false, Playwright ref,
                                                # ProjectReference -> WorkWingman.Core + Infrastructure
  HostClass.cs                                  # enum: Ats, Employer, Aggregator, Portal, Blocked, Unknown
  TraversalHop.cs                                # record of one visited url + its classification
  TraversalResult.cs                             # final outcome of a walk
  AdzunaJobResult.cs                             # DTO mirroring the API payload (Hop 0 input)
  HostTaxonomy.cs                                # loads host-taxonomy.seed.json, exposes lookups
  AggregatorSelectors.cs                         # ordered outbound-link rule chain, aggregator hosts
  EmployerSelectors.cs                           # ordered ats-anchor rule chain for vanity domains
  PortalHosts.cs / BlockedHosts.cs               # static allowlists + fail-closed rules
  PostingIdExtractors.cs                         # per-vendor regex, Confident vs Unknown
  AdzunaHop0Driver.cs                            # headful Playwright: redirect_url -> apply anchor
  TraversalEngine.cs                             # the hop loop / dispatcher
  FakeChainSite.cs                               # offline fixture: Adzuna->Zip->Spectrum->iCIMS
  LabMetrics.cs / Program.cs                     # harness boilerplate, copy from Lever lab

tests/WorkWingman.ScraperLab.Traversal.Tests/
  HostClassPhilosophyTests.cs
  EmployerAnchorOverMarkerTests.cs
  AggregatorNeverTerminalTests.cs
  PortalTerminalNotUnknownTests.cs
  BlockedFailsClosedTests.cs
  AdzunaHop0WarmContextTests.cs
  PostingIdExtractionTests.cs
  EndToEndChainWalkTests.cs
  WorkWingman.ScraperLab.Traversal.Tests.csproj

Add both projects to WorkWingman.slnx, same as every other lab.

Core types¶

namespace WorkWingman.ScraperLab.Traversal;

/// <summary>What KIND of place a host is, for traversal purposes. Mirrors the five classes in
/// host-taxonomy.seed.json exactly — do not add a sixth without updating that file's $comment.</summary>
public enum HostClass { Ats, Employer, Aggregator, Portal, Blocked, Unknown }

/// <summary>One visited url in a walk, plus what we learned there. Immutable — a walk is a list of
/// these, never mutated in place, so a test can assert on the full trail, not just the tail.</summary>
public sealed record TraversalHop(
    string Url,
    HostClass Class,
    string? AtsKindName,       // set only when Class == Ats (mirrors WorkWingman.Core.Models.AtsKind)
    string? PostingId,         // set only when Class == Ats and extraction succeeded
    string Reason);            // WHY this hop was classified this way — required, never empty

/// <summary>Final outcome of a walk. IsResolved is true ONLY for Ats (posting id may still be
/// null — a wrong regex is worse than no id) or Portal (terminal-by-design). Aggregator/Employer
/// can never be a terminal Class in a finished result; Blocked and hop-budget-exhausted both
/// collapse to Unknown so callers have exactly one "give up" shape to handle.</summary>
public sealed record TraversalResult(
    IReadOnlyList<TraversalHop> Hops,
    HostClass Terminal,
    bool IsResolved,
    string? ApplyUrl,
    string? AtsKindName,
    string? PostingId)
{
    public static TraversalResult Unknown(IReadOnlyList<TraversalHop> hops, string reason) =>
        new(hops, HostClass.Unknown, false, null, null, null) is var r
            ? r with { Hops = [.. hops, new TraversalHop(hops.Count > 0 ? hops[^1].Url : "", HostClass.Unknown, null, null, reason)] }
            : r;
}

/// <summary>Verbatim shape of the Adzuna /v1/api/jobs/.../search result object (Hop 0 input).
/// Field list pinned from the real payload — do NOT add fields the API doesn't send, and do NOT
/// assume redirect_url is a direct employer/ATS link; it is Adzuna's own tracked lander.</summary>
public sealed record AdzunaJobResult(
    string Id,
    string Title,
    string CompanyDisplayName,
    string RedirectUrl,   // e.g. https://www.adzuna.com/land/ad/{id} — Hop 1 input, NOTHING more
    string Adref,          // Adzuna's own tracking token; NOT the apply anchor's 'aztt' token (Hop 2)
    string Description);

HOP 0 — the Adzuna API result object, forward to a drivable ATS url¶

The walk starts at data, not a url. Map what each hop knows, needs, does next, and how it fails closed.

Hop	Input	What we know	What we still need	Next programmatic action	Fail-closed when
0	`AdzunaJobResult`	id, title, company name, `redirect_url` host is `www.adzuna.com`, `adref` (Adzuna's own tracking token)	`redirect_url` is a tracked lander, not a direct employer/ATS link — nothing in the payload points at the employer	Parse only. Validate `redirect_url` is absolute https on host `adzuna.com`/`www.adzuna.com`. No network yet.	`redirect_url` missing, empty, or not on an adzuna.com host → `Unknown` immediately. Never guess a company-name-derived career-site URL — that is fabrication, explicitly banned.
1	`redirect_url` (`/land/ad/{id}`)	Adzuna's own aggregator host; measured to 302 to `/details/{id}`	Whether we get the real page or a CloudFront challenge; whether the apply anchor is even served to us	Navigate with the headful Playwright browser WW already ships for applying, on a persistent/warm profile (see Hop-1 driver below). Never HttpClient (403, 180/180). Never headless (403, 36/40 — not viable).	Non-200, or DOM never reaches an expected anchor within timeout, or content matches a WAF-challenge shape → `HostClass.Blocked`, STOP, `Unknown`. No UA/TLS/proxy escalation.
2	`/details/{id}` DOM	Still `adzuna.com`; the apply control is an anchor carrying a freshly signed `aztt` JWT	The anchor is measured to be absent from a fresh context in 14/14 attempts and present for an ordinary (cookie-bearing) profile — this is a state requirement, not a selector problem	Locate `a[href='aztt=']` (see Employer/Aggregator hop below for the exact chain). Read its `href` — never click* (mirrors the house "Submit is located, never clicked" invariant, applied here to the outbound hop).	Anchor absent after the wait window even on a warm profile → `Unknown`. Do not fall back to a text-only `Apply` button that lacks the `aztt` param — that has not been measured and may be a different (e.g. Easy-Apply-in-place) control.
3	outbound `href` from Hop 2	New host, unclassified	Is it a known aggregator (ZipRecruiter, Jobcase), a known ats host, a portal host, or an unclassified employer vanity domain?	`HostTaxonomy.Classify(uri.Host)` — allowlist lookup first (cheap, no fetch), `AtsDetector.DetectFromUrl` second.	No classification and host fetch itself blocked → `Blocked` → `Unknown`.
4	ZipRecruiter (measured example)	`HostClass.Aggregator`; outbound link carried `ss=paid`, `utm_medium=sponsored-jobs`, `utm_source=ziprecruiter` — evidence the employer paid for placement, not evidence of where the outbound anchor lives in the DOM	Exact CSS anchor for "Apply on Company Site" — not measured, mark UNKNOWN selector, use host-based predicate instead (see Aggregator chain)	Same-page anchor scan for the first `http(s)` anchor whose host is not `ziprecruiter.com` and is not a `ziprecruiter.com` tracking/asset host	No qualifying outbound anchor found → keep-hopping budget exhausted → `Unknown`. Never treat ZipRecruiter's own `/job/{id}` canonical link as an answer (it has "job" in the path and would false-positive on a naive apply-shaped-path heuristic).
5	`jobs.spectrum.com` (measured)	`HostClass.Employer`; page HTML literally contains `icims`, `lever`, `radancy`, and `talentbrew` markers; also has a same-host `/application-process` informational link	The correct vendor is whichever one has a real anchor href, not whichever marker appears first in page text/scripts	`AtsDetector.DetectFromDescriptionPage(html)` — reuse verbatim, it already implements anchor-host-first + asset filtering	`/application-process` must never satisfy termination (same-host, informational) even though "application" reads apply-shaped. If `DetectFromDescriptionPage` returns `Unknown` and there is no qualifying external anchor → `Unknown`.
6 (terminal)	`careers-charter.icims.com/jobs/77665/.../job`	`HostClass.Ats`, `AtsKind.Icims`, posting id 77665 (regex, confident — see table below)	Nothing — this is the answer	Stop. Return `TraversalResult` with `IsResolved = true`.	N/A — this is the success exit.

This is the exact chain in the brief (Adzuna -> ZipRecruiter -> jobs.spectrum.com -> careers-charter.icims.com), expressed as a hop table so the engine below is a direct transliteration of it.

Per-host-class specs¶

Class: `Ats` — terminal, system of record¶

Traversal philosophy. An ats-class host is one AtsDetector already knows how to fill a form on — by construction, once we're here the walk is answered, not merely closer. The traversal lab's only job for this class is to recognize it fast (cheap host check before any page content) and to never let a downstream trap (a mentioned-but-not-linked vendor, a same-host informational page) impersonate it.

Ordered rule chain (most specific / cheapest first):

public static class AtsRecognition
{
    /// <summary>Rule 1 (cheapest, no fetch): does the CURRENT url's host already match a known
    /// ATS host marker? Reuses AtsDetector.DetectFromUrl verbatim — never re-implement the host
    /// matching, or the two tables will drift (this is exactly the failure HostMatchesMarker exists
    /// to prevent).</summary>
    public static AtsInfo TryFromUrl(string currentUrl) => AtsDetector.DetectFromUrl(currentUrl);

    /// <summary>Rule 2 (needs the fetched page): scan the CURRENT page's HTML for an anchor whose
    /// HREF host matches a known ATS marker. This is the rule that resolves the jobs.spectrum.com
    /// trap correctly — DetectFromDescriptionPage prefers an anchor href host over a marker found
    /// in page text/scripts, per its own doc comment and the markerTraps entry in
    /// host-taxonomy.seed.json ("prefer the vendor of an ANCHOR HREF host over a marker found in
    /// page text/scripts").</summary>
    public static AtsInfo TryFromPage(string html) => AtsDetector.DetectFromDescriptionPage(html);
}

Negative rules. 1. Never terminate on a vendor marker found only in page text/scripts without a corresponding anchor href on that host. jobs.spectrum.com mentions icims, lever, radancy, and talentbrew in one page; a first-marker-wins scan resolves to Lever (wrong). Only the anchor href host is authoritative. 2. Never terminate on a same-host informational path even if it looks apply-shaped — /application-process contains "application" and would pass a naive path.Contains("apply") check. Termination requires the URL to be on a known ats host, not merely apply-shaped text on the current host. 3. Never fabricate a TenantHost or AtsKind when DetectFromUrl/DetectFromDescriptionPage return Unknown — pass the Unknown straight through to the caller.

Termination predicate.

TryFromUrl(currentUrl).Kind != AtsKind.Unknown
  OR TryFromPage(html).Kind != AtsKind.Unknown
  => HostClass.Ats, terminal = true, IsResolved = true

Test assertions (house style — assert the philosophy).

public class EmployerAnchorOverMarkerTests
{
    private const string SpectrumHtml = /* fixture: icims+lever+radancy+talentbrew markers,
        an /application-process same-host anchor, and one real anchor to
        https://careers-charter.icims.com/jobs/77665/.../job — see FakeChainSite.cs */ "";

    [Fact]
    public void Multi_marker_employer_page_resolves_to_the_ANCHOR_hosts_vendor_not_the_first_marker()
    {
        var info = AtsDetector.DetectFromDescriptionPage(SpectrumHtml);
        Assert.Equal(AtsKind.Icims, info.Kind);
    }

    [Fact]
    public void Multi_marker_employer_page_NEVER_resolves_to_a_vendor_that_has_no_real_anchor()
    {
        // The whole point of the trap: lever/radancy/talentbrew are TEXT-ONLY mentions on this
        // fixture. A reordered/first-wins implementation would pick Lever. This must never pass.
        var info = AtsDetector.DetectFromDescriptionPage(SpectrumHtml);
        Assert.NotEqual(AtsKind.Lever, info.Kind);
    }

    [Fact]
    public void Same_host_informational_apply_shaped_path_never_terminates_the_walk()
    {
        const string html = "<a href='/application-process'>How to apply</a>" +
                             "<a href='https://careers-charter.icims.com/jobs/77665/x/job'>Apply Now</a>";
        var info = AtsDetector.DetectFromDescriptionPage(html);
        Assert.Equal(AtsKind.Icims, info.Kind);           // NOT /application-process
        Assert.DoesNotContain("application-process", info.ApplyUrl);
    }

    [Theory]
    [InlineData("https://careers-charter.icims.com/jobs/77665/call-center-operator/job", AtsKind.Icims)]
    [InlineData("https://boards.greenhouse.io/acme/jobs/4551123", AtsKind.Greenhouse)]
    [InlineData("https://careers.airbnb.com/positions/7995199?gh_jid=7995199", AtsKind.Unknown)] // host-only detect misses this — see EmployerSelectors below, this is a KNOWN gap, not a bug
    public void Url_only_recognition_matches_known_ats_hosts(string url, AtsKind expected)
    {
        Assert.Equal(expected, AtsDetector.DetectFromUrl(url).Kind);
    }
}

The third [Theory] case is deliberately a documented gap, not a passing assertion of success: careers.airbnb.com is not a Greenhouse host string, so DetectFromUrl alone returns Unknown — the gh_jid query param is the only signal, and that is a posting-id-extraction concern (see below), not a host-classification one. The test pins the gap so nobody "fixes" DetectFromUrl into a query-string scanner and breaks its safety property (host-based matching only, never a query-value substring match — that would reopen exactly the attacker-controlled-link hole ExtractUrl's doc comments warn about).

Class: `Employer` — vanity domain hiding the real ATS¶

Traversal philosophy. An employer's own career site is host-classified Unknown by AtsDetector (no known vendor host) but is not itself an answer — it is a pass-through whose job is to leak the real ATS host through its own outbound anchors. Structurally: if the current host doesn't match any Ats marker AND the page has external anchors, it's Employer, not Unknown — the two must be kept distinct because Unknown means "give up," Employer means "one more anchor scan."

Ordered rule chain.

public static class EmployerSelectors
{
    /// <summary>Rule 1: full delegate to AtsDetector.DetectFromDescriptionPage — it already does
    /// anchor-host-first scanning with asset filtering (see ATS class above). This IS the
    /// employer-vanity-domain resolver; there is no separate implementation to write.</summary>
    public static AtsInfo TryResolveAts(string html) => AtsDetector.DetectFromDescriptionPage(html);

    /// <summary>Rule 2 (fallback, ONE hop only): if rule 1 found no known-ats anchor, look for any
    /// external (different-host) anchor that reads apply-shaped, and re-classify from there. Capped
    /// at one extra hop — an employer page that needs two guesses to find its own apply link is a
    /// sign to fail closed, not to keep guessing.</summary>
    public static string? NextCandidateHref(string html, string currentHost)
    {
        // Playwright-side equivalent selector for the live-page version of this rule:
        //   page.Locator("a[href]:visible")
        //       .Filter(new() { HasText = new Regex("(?i)apply|careers|jobs") })
        // The offline/HTML-string version reuses AtsDetector's own href walker rather than a second
        // regex implementation — see FakeChainSite.cs for the parsing helper reused by both.
        return AtsDetector.DetectFromDescriptionPage(html).ApplyUrl is { Length: > 0 } url &&
               !new Uri(url).Host.Equals(currentHost, StringComparison.OrdinalIgnoreCase)
            ? url : null;
    }
}

Negative rules. 1. Never accept a same-host anchor as the resolution target — an employer page's own /application-process or /careers/faq is not progress. 2. Never accept the first marker found in raw HTML body text (widget/script noise from Radancy/TalentBrew, which is itself the front-end framework and will always appear in the page's own script tags) — anchor href only. 3. Cap employer-class hops at one re-classification attempt. An employer site that requires guessing twice is not a page this lab should be threading a needle through.

Termination predicate. Employer is never a terminal HostClass in a finished TraversalResult — either it resolves to Ats/Portal within the hop budget, or the walk ends in Unknown. This is enforced by construction: TraversalResult.Terminal only accepts Ats or Portal as IsResolved = true states.

Test assertions.

public class EmployerNeverTerminalTests
{
    [Fact]
    public void Employer_class_can_never_be_the_Terminal_value_of_a_resolved_result()
    {
        var result = new TraversalResult([], HostClass.Employer, IsResolved: true, null, null, null);
        // This is the contract test: an Employer-terminal, IsResolved-true result is a
        // CONSTRUCTION-LEVEL contradiction. Assert the invariant a reviewer must uphold, e.g. via
        // a factory that throws, or (if TraversalResult stays a plain record) a targeted analyzer
        // rule / smart constructor. Skeleton below shows the smart-constructor shape.
        Assert.Throws<InvalidOperationException>(() => TraversalResult.Resolved(HostClass.Employer, [], null, null, null));
    }
}

TraversalResult.Unknown(...) above needs a matching TraversalResult.Resolved(HostClass, ...) smart constructor that throws for anything other than Ats/Portal — that is the mechanism that turns "Employer must never terminate" from a comment into something a wrong future edit cannot compile past silently.

Class: `Aggregator` — resells listings, never final¶

Traversal philosophy. An aggregator's own page is never the answer regardless of what it contains — its entire commercial purpose is to host someone else's job and route the applicant onward. Unlike Employer (unknown host, discovered by elimination), Aggregator should be recognized by an explicit host allowlist, mirroring the same whole-host-match discipline AtsDetector.HostMatchesMarker uses — a heuristic ("many jobs from many companies on one page") is too easy to false-positive on a large careers portal. Only add a host here once it's been measured, per the seed file's own rule ("an entry here is a starting hypothesis, not a licence to skip verification").

Ordered rule chain.

public static class AggregatorSelectors
{
    /// <summary>Known aggregator hosts, in the SAME whole-host-or-subdomain sense as
    /// AtsDetector.HostMatchesMarker (never a bare substring match). www.adzuna.com is listed here
    /// too — Adzuna's OWN role, once past its access gate (see Hop-1 driver), is an aggregator like
    /// ZipRecruiter and Jobcase, not a special case.</summary>
    public static readonly string[] KnownHosts =
        ["ziprecruiter.com", "jobcase.com", "adzuna.com"];

    public static bool IsAggregatorHost(string host) =>
        KnownHosts.Any(marker => host.Equals(marker, StringComparison.OrdinalIgnoreCase) ||
                                  host.EndsWith("." + marker, StringComparison.OrdinalIgnoreCase));

    /// <summary>Rule 1: an outbound anchor whose host DIFFERS from the current aggregator host and
    /// is not one of the aggregator's own asset/tracking hosts. This is a HOST predicate, not a CSS
    /// selector — the exact "Apply on Company Site" button class/id was not measured for
    /// ZipRecruiter, so committing to a specific selector here would be exactly the kind of
    /// unverified guess this lab exists to avoid. Playwright shape:
    ///   var anchors = await page.Locator("a[href^='http']").AllAsync();
    ///   foreach anchor: href host != currentHost AND host not in KnownAssetHosts(currentHost) -> candidate
    /// </summary>
    public static string? FirstOutboundNonAggregatorHref(IEnumerable<string> anchorHrefs, string currentHost)
    {
        foreach (var href in anchorHrefs)
        {
            if (!Uri.TryCreate(href, UriKind.Absolute, out var uri)) continue;
            if (uri.Host.Equals(currentHost, StringComparison.OrdinalIgnoreCase)) continue;   // same-host: never a hop
            if (IsAggregatorHost(uri.Host)) continue;                                          // aggregator-to-aggregator: keep scanning, don't stop here
            return href;
        }
        return null;
    }
}

The exact CSS anchor for ZipRecruiter's outbound "Apply on Company Site" control is UNKNOWN — not measured in this pass. The host-difference predicate above is the deliberately conservative substitute: it will find the right anchor on any page shape without needing a vendor-specific selector, at the cost of also matching e.g. a "Share" button that happens to link off-site first. That tradeoff is acceptable for a keep-hopping class (a wrong candidate just gets reclassified and discarded one hop later); it would not be acceptable for the Ats termination rule, which is why that class does not use this predicate.

Negative rules. 1. Never treat the aggregator's own canonical job url (e.g. ziprecruiter.com/job/{id}, which literally contains "job") as a resolution — same-host, so rule 1 above already excludes it, but it is worth stating as its own negative rule because a naive path.Contains("job") heuristic (as used for LooksLikeApplyLink in AtsDetector, correctly, in a different context) would wrongly select it here. 2. Never terminate on query-param evidence alone (ss=paid, utm_source=ziprecruiter) — those params prove the employer paid for placement, not that a given anchor is the outbound one. They are provenance metadata to log, not a selection signal. 3. If the aggregator page's "apply" flow is entirely in-page (no outbound anchor at all — e.g. an Indeed-style Easy Apply embedded form) treat as a dead end, not a Portal. Portal is reserved for hosts where the public page itself declares the completion happens elsewhere behind login (measured: usajobs.gov). An in-page apply-without-login flow that this lab hasn't measured a host for is Unknown, never guessed into either bucket.

Termination predicate. Aggregator is never a terminal HostClass, same construction-level guarantee as Employer.

Test assertions.

public class AggregatorNeverTerminalTests
{
    [Theory]
    [InlineData("https://www.ziprecruiter.com/job/abc123")]
    [InlineData("https://www.jobcase.com/job/xyz789")]
    [InlineData("https://www.adzuna.com/details/5684633191")]
    public void Known_aggregator_hosts_are_classified_Aggregator_not_Ats_or_Employer(string url)
    {
        var host = new Uri(url).Host;
        Assert.True(AggregatorSelectors.IsAggregatorHost(host));
    }

    [Fact]
    public void Outbound_selection_never_returns_a_same_host_or_other_aggregator_href()
    {
        string[] hrefs =
        [
            "https://www.ziprecruiter.com/job/abc123",                         // same host — excluded
            "https://www.jobcase.com/job/xyz789",                              // aggregator-to-aggregator — excluded
            "https://jobs.spectrum.com/us/en/job/RC1234/Call-Center-Operator", // real outbound — expected
        ];
        var picked = AggregatorSelectors.FirstOutboundNonAggregatorHref(hrefs, "www.ziprecruiter.com");
        Assert.Equal("https://jobs.spectrum.com/us/en/job/RC1234/Call-Center-Operator", picked);
    }

    [Fact]
    public void A_paid_placement_query_param_alone_never_selects_an_anchor()
    {
        // ss=paid on a SAME-HOST link must not be mistaken for the outbound hop.
        string[] hrefs = ["https://www.ziprecruiter.com/job/abc123?ss=paid&utm_source=ziprecruiter"];
        Assert.Null(AggregatorSelectors.FirstOutboundNonAggregatorHref(hrefs, "www.ziprecruiter.com"));
    }
}

Class: `Portal` — terminal by design, apply completes behind login¶

Traversal philosophy. usajobs.gov is the measured example: the application hand-off happens after login, and attempting to resolve further always burns a fetch and returns Unknown. This must be represented as a distinct success-shaped terminal state, not a failure — the seed file is explicit: "Mark terminal, never 'unresolved'." AtsKind currently has no federal/portal member (confirmed by reading JobPosting.cs's enum), so Portal is a HostClass-level outcome independent of AtsKind — a TraversalResult can be Terminal = HostClass.Portal, IsResolved = true, AtsKindName = null. Do not wait on an AtsKind enum change to model this correctly.

Ordered rule chain.

public static class PortalHosts
{
    // Whole-host match, same discipline as everywhere else in this lab.
    public static readonly string[] KnownHosts = ["usajobs.gov"];

    public static bool IsPortalHost(string host) =>
        KnownHosts.Any(marker => host.Equals(marker, StringComparison.OrdinalIgnoreCase) ||
                                  host.EndsWith("." + marker, StringComparison.OrdinalIgnoreCase));
}

There is no second rule: a portal host is recognized by host alone, before any page fetch — the whole point is to spend zero further requests on it.

Negative rules. 1. Never keep hopping past a portal host looking for a downstream ATS — the seed file measured this to always return Unknown while burning a fetch. Treat the host check as sufficient and stop immediately. 2. Never represent a portal resolution as Unknown or as an error — a caller (dedupe / UI status) that can't tell "we resolved to a login-gated portal" apart from "we gave up" will mis-surface this to the user as a failure when it is actually a correct, complete answer. 3. Never attempt to extract a posting id past the login wall — public-pages-only means whatever control-number-like token is visible pre-login is the most that can ever be captured here, and only if it is unambiguous (see posting-id table: marked UNKNOWN for USAJOBS pending a measured example).

Termination predicate. IsPortalHost(uri.Host) => HostClass.Portal, terminal = true, IsResolved = true — unconditional, no further hop attempted.

Test assertions.

public class PortalTerminalNotUnknownTests
{
    [Theory]
    [InlineData("https://www.usajobs.gov/job/123456789")]
    [InlineData("https://www.usajobs.gov/help/faq/application")]  // even an off-topic path on the host
    public void Portal_host_terminates_immediately_without_a_further_hop(string url)
    {
        Assert.True(PortalHosts.IsPortalHost(new Uri(url).Host));
    }

    [Fact]
    public void Portal_result_is_marked_resolved_true_never_collapsed_into_Unknown()
    {
        var result = TraversalResult.Resolved(HostClass.Portal, [], applyUrl: "https://www.usajobs.gov/job/123456789", null, null);
        Assert.True(result.IsResolved);
        Assert.Equal(HostClass.Portal, result.Terminal);
        // The distinguishing assertion: a caller must be able to tell this apart from a real Unknown.
        Assert.NotEqual(HostClass.Unknown, result.Terminal);
    }
}

Class: `Blocked` — refuses automation; STOP, never circumvent¶

Traversal philosophy. "Blocked" is method-dependent, and that nuance is the whole point of this class: the measured facts show adzuna.com returning 403 to a plain HttpClient (180/180) and to a headless browser (36/40 — not a viable rate either), but succeeding with the exact headful browser WW already ships for applying (0/15 blocked). That is not evasion — it is literally the same access method a human user's already-consented WW session uses, run at human pace, with no fingerprint modification. So: a host does not get the Blocked classification just because the cheap/unattended method failed. It earns Blocked only when the legitimate headful method, with warm state, also fails — a real CloudFront/WAF challenge page, or a non-200 that survives a reasonable wait. At that point the rule is absolute: stop, return Unknown, do not escalate to UA spoofing, TLS fingerprint changes, proxy rotation, or CAPTCHA solving. Those are explicitly out of scope regardless of how close the block feels to bypassable.

Ordered rule chain.

public static class BlockedDetection
{
    /// <summary>Rule 1: does the page content match a known WAF/CDN challenge shape rather than
    /// the expected job-page DOM? A 200 status is NOT evidence of a real document — the seed file
    /// records this exact trap for Adzuna's CloudFront challenge.</summary>
    public static bool LooksLikeChallengePage(string html) =>
        html.Contains("cf-error-details", StringComparison.OrdinalIgnoreCase) ||
        html.Contains("Attention Required", StringComparison.OrdinalIgnoreCase) ||
        html.Contains("cloudfront", StringComparison.OrdinalIgnoreCase) && html.Length < 4_000; // a real job page is never this short

    /// <summary>Rule 2: did the response itself fail transport-level, even on the headful method?</summary>
    public static bool IsHardBlocked(int httpStatus, string html) =>
        httpStatus is >= 400 or 0 || LooksLikeChallengePage(html);
}

Negative rules. 1. Never conclude Blocked from an HttpClient or headless failure alone — those are documented to have a nonzero false-block rate against this exact target; only the headful method's own failure counts. 2. Never escalate past a genuine headful block with any of: a stealth/automation-hiding plugin, navigator.webdriver patching, a spoofed User-Agent or TLS fingerprint, CAPTCHA solving, or proxy rotation. This is a hard line from the brief, not a tuning knob. 3. Never retry a blocked host in the same run — one failed headful attempt is definitive for that hop; retrying just spends another request against a host that has already told us no.

Termination predicate.

attempted with plain HttpClient or headless => failure is NOT evidence, do not classify yet, escalate to headful
attempted with headful, warm profile, human pace =>
    IsHardBlocked(status, html) == true  => HostClass.Blocked, STOP, Unknown
    otherwise                            => proceed with normal classification (Ats/Employer/Aggregator/Portal)

Test assertions.

public class BlockedFailsClosedTests
{
    [Fact]
    public void A_403_from_HttpClient_alone_does_not_classify_the_host_as_Blocked()
    {
        // The philosophy under test: cheap-method failure escalates to headful, it does not
        // terminate the walk. This test exists to fail if someone later "optimizes" the engine to
        // short-circuit on the first 403 instead of trying the headful method.
        var outcome = TraversalEngine.ClassifyTransportFailure(method: "HttpClient", status: 403);
        Assert.Equal(TransportOutcome.EscalateToHeadful, outcome);
    }

    [Fact]
    public void A_403_from_headless_alone_also_escalates_rather_than_blocking()
    {
        var outcome = TraversalEngine.ClassifyTransportFailure(method: "Headless", status: 403);
        Assert.Equal(TransportOutcome.EscalateToHeadful, outcome);
    }

    [Fact]
    public void A_403_from_the_headful_method_is_definitive_and_STOPS_the_walk()
    {
        var outcome = TraversalEngine.ClassifyTransportFailure(method: "Headful", status: 403);
        Assert.Equal(TransportOutcome.Blocked, outcome);
    }

    [Fact]
    public void A_200_status_with_challenge_page_content_is_still_Blocked_not_a_resolved_document()
    {
        const string challengeHtml = "<html><body>Attention Required! | Cloudflare</body></html>";
        Assert.True(BlockedDetection.IsHardBlocked(httpStatus: 200, challengeHtml));
    }

    [Fact]
    public void A_real_job_document_at_200_is_never_misclassified_as_a_challenge()
    {
        var realHtml = "<html>" + new string('x', 5_000) + "<a href='https://careers-charter.icims.com/jobs/77665/x/job'>Apply</a></html>";
        Assert.False(BlockedDetection.IsHardBlocked(httpStatus: 200, realHtml));
    }
}

The Hop-0/Hop-1 Adzuna driver — headful, warm context, no stealth¶

namespace WorkWingman.ScraperLab.Traversal;

using Microsoft.Playwright;

/// <summary>Drives Hop 1-2 ONLY: from AdzunaJobResult.RedirectUrl to the outbound apply anchor's
/// href. Uses the SAME real Chromium the app ships for applying, headful, on a PERSISTENT profile
/// directory so the browsing context is a genuinely warm/returning one — not a stealth trick, just
/// reuse of an ordinary already-consented profile. No UserAgent override. No extra headers. No
/// navigator.webdriver patch. If the site still refuses us as ourselves, we stop.</summary>
public sealed class AdzunaHop0Driver
{
    private readonly string _profileDir;

    public AdzunaHop0Driver(string profileDir) => _profileDir = profileDir;

    public async Task<TraversalHop> ResolveOutboundHopAsync(AdzunaJobResult job, CancellationToken ct)
    {
        if (!Uri.TryCreate(job.RedirectUrl, UriKind.Absolute, out var landerUri) ||
            (landerUri.Scheme != Uri.UriSchemeHttps) ||
            !(landerUri.Host.Equals("www.adzuna.com", StringComparison.OrdinalIgnoreCase) ||
              landerUri.Host.Equals("adzuna.com", StringComparison.OrdinalIgnoreCase)))
        {
            // Fail closed per Hop 0's own rule: never guess past a malformed/foreign redirect_url.
            return new TraversalHop(job.RedirectUrl, HostClass.Unknown, null, null,
                "redirect_url missing/foreign — refused without a fetch");
        }

        using var playwright = await Playwright.CreateAsync();
        await using var context = await playwright.Chromium.LaunchPersistentContextAsync(
            userDataDir: _profileDir,
            new BrowserTypeLaunchPersistentContextOptions
            {
                Headless = false,   // NON-NEGOTIABLE: measured 0/15 blocked headful vs 36/40 headless
                // Deliberately NOT set: UserAgent, ExtraHTTPHeaders, Proxy, ViewportSize spoofing.
                // This context is a real Chromium browser presenting as exactly what it is.
            });

        var page = context.Pages.Count > 0 ? context.Pages[0] : await context.NewPageAsync();
        var response = await page.GotoAsync(job.RedirectUrl, new PageGotoOptions { WaitUntil = WaitUntilState.NetworkIdle });

        // Best-effort consent dismissal — a warm profile that has visited before often doesn't show
        // this at all; when it does, this is ordinary user-facing UI interaction, not evasion.
        var consent = page.Locator(
            "#onetrust-accept-btn-handler, button:has-text('Accept'), [aria-label='Accept cookies']");
        if (await consent.CountAsync() > 0)
            await consent.First.ClickAsync(new LocatorClickOptions { Timeout = 3_000 }).ContinueWith(_ => { }, ct);

        var html = await page.ContentAsync();
        var status = response?.Status ?? 0;
        var outcome = TraversalEngine.ClassifyTransportFailure("Headful", status);
        if (outcome == TransportOutcome.Blocked || BlockedDetection.IsHardBlocked(status, html))
            return new TraversalHop(page.Url, HostClass.Blocked, null, null,
                $"headful status={status}, challenge-shape={BlockedDetection.LooksLikeChallengePage(html)}");

        // Hop 2: the apply anchor carries a freshly signed 'aztt' JWT — measured to require this
        // warm, non-fresh context (14/14 fresh-context attempts did NOT see it).
        var applyAnchor = page.Locator("a[href*='aztt=']");
        try
        {
            await applyAnchor.First.WaitForAsync(new LocatorWaitForOptions { Timeout = 8_000 });
        }
        catch (TimeoutException)
        {
            return new TraversalHop(page.Url, HostClass.Blocked, null, null,
                "apply anchor (aztt) never attached even on a warm headful context");
        }

        var href = await applyAnchor.First.GetAttributeAsync("href");
        if (string.IsNullOrEmpty(href))
            return new TraversalHop(page.Url, HostClass.Unknown, null, null, "aztt anchor present but href empty");

        return new TraversalHop(href, HostClass.Unknown /* classified by caller on next hop */, null, null,
            "outbound apply anchor located (never clicked — href read and followed as navigation)");
    }
}

public enum TransportOutcome { Ok, EscalateToHeadful, Blocked }

TraversalEngine.ClassifyTransportFailure is the pure function under test above:

public static partial class TraversalEngine
{
    public static TransportOutcome ClassifyTransportFailure(string method, int status)
    {
        if (status is >= 200 and < 400) return TransportOutcome.Ok;
        return method switch
        {
            "HttpClient" or "Headless" => TransportOutcome.EscalateToHeadful, // cheap methods lie; not evidence
            "Headful" => TransportOutcome.Blocked,                             // the real method failed — definitive
            _ => TransportOutcome.Blocked
        };
    }
}

Posting-ID extraction per ATS vendor¶

Per the brief: a wrong regex silently corrupts a dedupe key, which is worse than no key at all. Only vendors with a measured or extremely stable, well-documented public shape get a regex here. Everything else is UNKNOWN on purpose.

namespace WorkWingman.ScraperLab.Traversal;

using System.Text.RegularExpressions;

public static partial class PostingIdExtractors
{
    /// <summary>Confidence-tagged extractor. UNKNOWN vendors return null on purpose — a caller must
    /// treat null as "no dedupe key available," never synthesize one.</summary>
    public delegate string? Extractor(string url);

    // --- CONFIDENT (measured or stable-by-vendor-design) ---------------------------------------

    /// <summary>iCIMS: MEASURED directly (careers-charter.icims.com/jobs/77665/.../job).
    /// Path shape /jobs/{numeric-id}/{slug}/... is iCIMS's standard job-detail route.</summary>
    [GeneratedRegex(@"icims\.com/jobs/(\d+)/", RegexOptions.IgnoreCase)]
    private static partial Regex IcimsIdRegex();
    public static string? Icims(string url) => IcimsIdRegex().Match(url) is { Success: true } m ? m.Groups[1].Value : null;

    /// <summary>Greenhouse: two independent stable shapes — the gh_jid query param (works even on a
    /// vanity host that HIDES the vendor, per the brief's careers.airbnb.com example) and the
    /// boards.greenhouse.io/{co}/jobs/{id} path. Both are long-standing, documented Greenhouse
    /// conventions, not a one-off observation.</summary>
    [GeneratedRegex(@"[?&]gh_jid=(\d+)", RegexOptions.IgnoreCase)]
    private static partial Regex GreenhouseQueryRegex();
    [GeneratedRegex(@"greenhouse\.io/[^/]+/jobs/(\d+)", RegexOptions.IgnoreCase)]
    private static partial Regex GreenhousePathRegex();
    public static string? Greenhouse(string url) =>
        GreenhouseQueryRegex().Match(url) is { Success: true } q ? q.Groups[1].Value :
        GreenhousePathRegex().Match(url) is { Success: true } p ? p.Groups[1].Value : null;

    /// <summary>Lever: job ids are UUIDs in the path (jobs.lever.co/{company}/{uuid}) — a
    /// long-stable, documented Lever convention.</summary>
    [GeneratedRegex(@"lever\.co/[^/]+/([0-9a-fA-F-]{36})", RegexOptions.IgnoreCase)]
    private static partial Regex LeverIdRegex();
    public static string? Lever(string url) => LeverIdRegex().Match(url) is { Success: true } m ? m.Groups[1].Value : null;

    /// <summary>Workable: the {TOKEN} path segment (apply.workable.com/{co}/j/{TOKEN}/apply/) is
    /// opaque but positionally stable per Workable's own routing — safe as a dedupe key even though
    /// its internal structure is unknown.</summary>
    [GeneratedRegex(@"workable\.com/[^/]+/j/([A-Za-z0-9]+)/", RegexOptions.IgnoreCase)]
    private static partial Regex WorkableTokenRegex();
    public static string? Workable(string url) => WorkableTokenRegex().Match(url) is { Success: true } m ? m.Groups[1].Value : null;

    // --- UNKNOWN (do not extract — format varies per tenant / not measured) --------------------
    // Workday      — req id format varies WIDELY per tenant config (numeric, "R-12345", GUID-like
    //                job-posting-site ids all seen in the wild). No single regex is safe.
    // SmartRecruiters, Jobvite, Eightfold, Ashby — plausible path shapes exist but were not
    //                measured in this pass; committing a regex here without a captured real payload
    //                is exactly the "corrupts the dedupe key" failure the brief warns against.
    // Adp, Paycom, Paycor, Oracle, Dayforce, SuccessFactors, Ukg, Avature, Amazon, Google,
    // RippleHire, SalesforceCareers, Phenom, Microsoft, USAJOBS/portal — UNKNOWN, no measured or
    //                sufficiently standardized public id shape. Return null; never a best-effort guess.

    public static readonly IReadOnlyDictionary<string, Extractor> ConfidentExtractors =
        new Dictionary<string, Extractor>(StringComparer.OrdinalIgnoreCase)
        {
            ["Icims"] = Icims,
            ["Greenhouse"] = Greenhouse,
            ["Lever"] = Lever,
            ["Workable"] = Workable,
        };

    /// <summary>The single entry point callers use. Returns null for any vendor not in
    /// ConfidentExtractors — this IS the "mark UNKNOWN" behavior, enforced by construction rather
    /// than left to caller discipline.</summary>
    public static string? TryExtract(string atsKindName, string url) =>
        ConfidentExtractors.TryGetValue(atsKindName, out var extractor) ? extractor(url) : null;
}

Test assertions.

public class PostingIdExtractionTests
{
    [Theory]
    [InlineData("Icims", "https://careers-charter.icims.com/jobs/77665/call-center-operator/job", "77665")]
    [InlineData("Greenhouse", "https://boards.greenhouse.io/acme/jobs/4551123", "4551123")]
    [InlineData("Greenhouse", "https://careers.airbnb.com/positions/7995199?gh_jid=7995199", "7995199")] // vanity host, query-param path
    [InlineData("Lever", "https://jobs.lever.co/acme/3fa85f64-5717-4562-b3fc-2c963f66afa6", "3fa85f64-5717-4562-b3fc-2c963f66afa6")]
    [InlineData("Workable", "https://apply.workable.com/acme/j/AB12CD34EF/apply/", "AB12CD34EF")]
    public void Confident_vendors_extract_the_measured_id_shape(string kind, string url, string expected)
    {
        Assert.Equal(expected, PostingIdExtractors.TryExtract(kind, url));
    }

    [Theory]
    [InlineData("Workday", "https://acme.wd1.myworkdayjobs.com/en-US/External/job/Remote/Call-Center-Operator_R-12345")]
    [InlineData("SmartRecruiters", "https://jobs.smartrecruiters.com/Acme/743999812345678")]
    [InlineData("Eightfold", "https://acme.eightfold.ai/careers/job/12345678")]
    public void Unmeasured_vendors_return_null_rather_than_a_guessed_id(string kind, string url)
    {
        // The point of this test: it must FAIL the moment someone adds a plausible-looking regex
        // for one of these vendors without a measured/verified payload backing it.
        Assert.Null(PostingIdExtractors.TryExtract(kind, url));
    }

    [Fact]
    public void An_unknown_ats_kind_name_returns_null_not_an_exception()
    {
        Assert.Null(PostingIdExtractors.TryExtract("SomeFutureVendor", "https://example.com/job/1"));
    }
}

The end-to-end walk (fake fixtures, house pattern)¶

FakeChainSite.cs mirrors FakeIcimsSite.cs: an offline, in-memory HTML fixture per hop so the walk can be exercised deterministically without a network call, exactly the jobs.spectrum.com trap page described above (icims + lever + radancy + talentbrew markers, an /application-process decoy, one real iCIMS anchor with id 77665).

public static class FakeChainSite
{
    public const string ZipRecruiterOutboundPage = """
        <html><body>
          <a href="/job/abc123">Call Center Operator</a>
          <a href="https://jobs.spectrum.com/us/en/job/RC1234/Call-Center-Operator?ss=paid&utm_source=ziprecruiter&utm_medium=sponsored-jobs">
            Apply on Company Site
          </a>
        </body></html>
        """;

    public const string SpectrumEmployerPage = """
        <html><body>
          <script>window.__RADANCY_CONFIG__ = { vendor: "talentbrew" };</script>
          <p>Powered by lever.co widgets and icims.com integrations.</p>
          <a href="/application-process">How to apply</a>
          <a href="https://careers-charter.icims.com/jobs/77665/call-center-operator/job">Apply Now</a>
        </body></html>
        """;
}

public class EndToEndChainWalkTests
{
    [Fact]
    public void Full_walk_from_zip_outbound_through_spectrum_resolves_to_icims_77665()
    {
        var outboundHref = AggregatorSelectors.FirstOutboundNonAggregatorHref(
            ExtractHrefs(FakeChainSite.ZipRecruiterOutboundPage), "www.ziprecruiter.com");
        Assert.NotNull(outboundHref);
        Assert.Contains("jobs.spectrum.com", outboundHref);

        var atsInfo = AtsDetector.DetectFromDescriptionPage(FakeChainSite.SpectrumEmployerPage);
        Assert.Equal(AtsKind.Icims, atsInfo.Kind);

        var postingId = PostingIdExtractors.TryExtract("Icims", atsInfo.ApplyUrl!);
        Assert.Equal("77665", postingId);
    }

    private static IEnumerable<string> ExtractHrefs(string html) =>
        System.Text.RegularExpressions.Regex.Matches(html, "href=\"([^\"]+)\"").Select(m => m.Groups[1].Value);
}

Hard-line restatement (every hop, every class)¶

No stealth plugins. No navigator.webdriver patching. No User-Agent or TLS fingerprint spoofing. No CAPTCHA solving. No proxy rotation. A real browser, headful, as itself.
"Warm context" means an ordinary persistent profile directory the app already owns — the same mechanism a returning human user's browser has — not a technique to defeat detection. If a page still refuses a real headful browser being honest about what it is, that is the site's answer: STOP, return Unknown.
Unknown is not a bug in this design — it is the majority-safe outcome for anything unmeasured. Every extractor, every host list, every selector above either cites the measured evidence backing it or is explicitly marked UNKNOWN. Nothing here fabricates a vendor, tenant, or posting id.
Employer and Aggregator can never be a TraversalResult.Terminal value in a resolved result — enforced by a smart constructor, not just a comment, so a future edit that tries to short-circuit the walk on a pass-through class fails to compile/throws rather than silently shipping a wrong answer.

Open items (explicitly out of scope for this pass, do not guess)¶

Exact CSS/ARIA selector for ZipRecruiter's outbound "Apply on Company Site" control — not measured; the host-difference predicate in AggregatorSelectors is the deliberate substitute.
Jobcase's outbound-hop shape — host is known (aggregator allowlist), anchor shape is not; same treatment as ZipRecruiter.
USAJOBS posting-id shape — Portal terminates on host alone; no id extraction attempted pending a measured control-number example.
Workday/SmartRecruiters/Ashby/Jobvite/Eightfold and the rest of the ATS-LABS-TODO roster — posting-id extractors intentionally left UNKNOWN; wire them into ConfidentExtractors only after a real captured payload backs the regex, per the "verify at the producer" standard.

TRAVERSAL LAB — resolving an Adzuna API result to a drivable employer ATS url¶

File layout (mirrors the house pattern)¶

Core types¶

HOP 0 — the Adzuna API result object, forward to a drivable ATS url¶

Per-host-class specs¶

Class: Ats — terminal, system of record¶

Class: Employer — vanity domain hiding the real ATS¶

Class: Aggregator — resells listings, never final¶

Class: Portal — terminal by design, apply completes behind login¶

Class: Blocked — refuses automation; STOP, never circumvent¶

The Hop-0/Hop-1 Adzuna driver — headful, warm context, no stealth¶

Posting-ID extraction per ATS vendor¶

The end-to-end walk (fake fixtures, house pattern)¶

Hard-line restatement (every hop, every class)¶

Open items (explicitly out of scope for this pass, do not guess)¶

Class: `Ats` — terminal, system of record¶

Class: `Employer` — vanity domain hiding the real ATS¶

Class: `Aggregator` — resells listings, never final¶

Class: `Portal` — terminal by design, apply completes behind login¶

Class: `Blocked` — refuses automation; STOP, never circumvent¶