Adzuna Path Gronktayvius

I'll mirror the existing ScraperLab pattern first, then write a traversal lab spec that encodes host-class philosophy, ordered rules, negatives, and testable assertions from the measured hops.Next I'll pull the iCIMS philosophy tests and any existing hop/ATS-detection code so the traversal lab matches house style.# ScraperLab house pattern (read-only survey)

Inventory¶

Test projects (`tests/WorkWingman.ScraperLab.*.Tests/`)¶

Project	Files
`WorkWingman.ScraperLab.Icims.Tests`	`IcimsSelectorAndJudgementTests.cs`, `IframeSwitchLogicTests.cs`, `.csproj`
`WorkWingman.ScraperLab.Meta.Tests`	`MetaSelectorsTests.cs`, `PickIdStyleBoundaryTests.cs`, `SiteVariationRandomizerTests.cs`, `JudgementPauseTests.cs`, `LabMetricsTests.cs`, `SubmitGuardAndEngineTests.cs`
`WorkWingman.ScraperLab.Google.Tests`	`GoogleLabTests.cs`, `FixtureHydrationTests.cs`
`WorkWingman.ScraperLab.Microsoft.Tests`	`MicrosoftSelectorTests.cs`, `MicrosoftEngineDecisionTests.cs`, `stryker-config.json`

Also scattered: tools/WorkWingman.ScraperLab.{Adp,Amazon,Dayforce,Greenhouse,Lever,Oracle,Paycom,SuccessFactors}.Tests/ (tool-local tests).

Lab tools (`tools/WorkWingman.ScraperLab.<Ats>/`)¶

Template = Lever. ~24 labs: Workday (base WorkWingman.ScraperLab), Icims, Meta, Google, Microsoft, Greenhouse, Lever, Amazon, Adp, Dayforce, Oracle, Paycom, Paycor, Ukg, SuccessFactors, Ashby, SmartRecruiters, Workable, Jobvite, Eightfold, RippleHire, SalesforceCareers, Phenom, …

Canonical lab skeleton¶

tools/WorkWingman.ScraperLab.<Ats>/
  Program.cs                 # CLI: --iterations / --minutes / --seed / --headed / --output
  LoopRunner.cs | LearningLoop.cs
  <Ats>AutomationEngine.cs   # Playwright fill; submit LOCATE only
  <Ats>Selectors.cs          # pure string[] chains (no Playwright dep when possible)
  Fake<Ats>Site.cs           # HttpListener loopback 127.0.0.1
  <Ats>Variation.cs | SiteVariation.cs | FixtureVariation.cs
  LabMetrics.cs | LabProfile.cs | LabVault.cs | SampleProfile.cs
  stryker-config.json        # optional, scoped mutation

tests/WorkWingman.ScraperLab.<Ats>.Tests/
  *Selectors*Tests.cs        # philosophy shape + negative asserts
  *Judgement* / *Engine*     # pure decision logic
  *.csproj → xunit + ProjectRef lab + Core + Infrastructure

Promotion path: lab → src/WorkWingman.Infrastructure/Automation/{Ats}ApplyEngine.cs + {Ats}Selectors.cs.

Icims special case: selectors live in Infrastructure (IcimsSelectors.cs), not the lab tool. Lab engine consumes them.

Project / naming conventions¶

Item	Pattern
Namespace	`WorkWingman.ScraperLab.<Ats>` / `.Tests`
TFM	`net10.0`, `IsPackable=false`
Lab OutputType	`Exe`
Packages	lab: `Microsoft.Playwright` 1.61; tests: xunit 2.9.3, Microsoft.NET.Test.Sdk, optional Bogus
Global usings	tests: `<Using Include="Xunit" />`
Internals	Meta: `InternalsVisibleTo` test assembly
Safety banner	Program.cs: loopback-only, submit suppressed, no real network
Submit invariant	locate + log `"would submit (suppressed)"`; never click
Judgement	ambiguous → `RunStatus.AwaitingJudgement` + `PendingCall`; never guess

Template doc: docs/ATS-LABS-TODO.md
Learnings: docs/technical/scraper-automation-learnings-<ats>.md

Selector class shapes (philosophy = XML docs + ordered chains)¶

1. Icims — suffix/substring first (inverted Workday)¶

Key file: C:\Users\fives\source\repos\WW-ats-labs\src\WorkWingman.Infrastructure\Automation\IcimsSelectors.cs

Static readonly string[] per field
Lead: input[id$='FirstName'] then id*= / name*= / aria-label*=
Also: ContentIframe[], InnerIframe[] (no bare iframe fallback)
Philosophy in class XML: Workday exact-id-first; iCIMS ends-with-first

// FirstName chain lead:
"input[id$='FirstName']",
"input[id*='first_name' i]",
"input[id*='firstName' i]",
"input[name*='first' i]",
"input[aria-label*='First Name' i]"

2. Meta — semantic `#id` first; NEVER class¶

Key file: C:\Users\fives\source\repos\WW-ats-labs\tools\WorkWingman.ScraperLab.Meta\MetaSelectors.cs

Factory methods: CoreCssChain, AriaChain, ResumeCssChain, SubmitCssChain
Order: #id → [id*=] → [name=] → [name*=] → aria → label → placeholder
Explicit ban: bare .class (hashed CSS modules)

3. Google — `data-fieldid` first; 5–6 rungs; `SelectorChain` type¶

Key files: - ...\Google\GoogleSelectors.cs - ...\Google\SelectorChain.cs (resolve + SelectorTelemetry)

0 [data-fieldid='…']
1 [class*='gc-field--…']   // class OK here (fixture stamps stable fragment)
2 [name='…']
3 [aria-label='…']
4 label:has-text + input/select
5 [placeholder='…'] optional

4. Microsoft — exact semantic id first¶

Key file: ...\Infrastructure\Automation\MicrosoftSelectors.cs
Order: input[id='firstName'] → name → aria-label* → label:has-text

5. Workday (baseline contrast)¶

data-automation-id exact → substring → aria/label. Real-tenant correction: container-scoped [data-automation-id='formField-…'] input leads.

6. Lever (template)¶

name first → data-qa → name*= → aria/placeholder.

How PHILOSOPHY is asserted (not just string equality)¶

Lab	Positive	Negative (philosophy)
Icims	lead = `id$='FirstName'`	`Assert.DoesNotContain("[id='", chain[0])`
Meta	lead = `#firstName`	`Assert.DoesNotContain(".", sel)` over Faker tokens
Google	lead = `[data-fieldid=…]`, 6 rungs	`Assert.DoesNotContain(chain.Candidates[0], "#")`
Microsoft	lead id then name	submit chain has no `form[action`

Excerpt — Icims philosophy test¶

tests/WorkWingman.ScraperLab.Icims.Tests/IcimsSelectorAndJudgementTests.cs:13-24

[Fact]
public void Field_chains_lead_with_a_suffix_or_substring_match_not_an_exact_id()
{
    Assert.StartsWith("input[id$='FirstName']", IcimsSelectors.FirstName[0]);
    // No field chain should LEAD with an exact whole-id equality selector (id='...').
    foreach (var chain in new[] { IcimsSelectors.FirstName, IcimsSelectors.LastName, IcimsSelectors.School })
        Assert.DoesNotContain("[id='", chain[0]);
}

Excerpt — Meta philosophy test¶

tests/WorkWingman.ScraperLab.Meta.Tests/MetaSelectorsTests.cs:27-38

[Fact]
public void Core_chain_never_contains_a_bare_class_selector()
{
    for (var i = 0; i < 20; i++)
    {
        var token = Faker.Database.Column().Replace(" ", "");
        var chain = MetaSelectors.CoreCssChain(token, token);
        foreach (var sel in chain)
            Assert.DoesNotContain(".", sel);
    }
}

Excerpt — Google philosophy test¶

tests/WorkWingman.ScraperLab.Google.Tests/GoogleLabTests.cs:257-273

[Fact]
public void GoogleSelectors_NoChain_LeadsWithAnExactIdSelector()
{
    // never a bare #id (ids rotated per SiteVariation.RotateGeneratedIds)
    Assert.DoesNotContain(chain.Candidates[0], "#");
    Assert.StartsWith("[data-fieldid=", chain.Candidates[0]);
}

Theory / InlineData patterns¶

Degree similarity: [Theory][InlineData("Bachelor of Science (BS)", "B.S. Game Development", true)] + BS≥BA ranking (Icims, Microsoft)
Judgement triggers: work-auth yes / sponsorship no (Meta JudgementPauseTests)
Variation roll boundaries: exact int rolls map to IdStyle (Meta PickIdStyleBoundaryTests — kills threshold mutants)
Randomizer percentages: Google pins DropLeadAnchor etc. to documented % over seeded trials
Widget styles: SelfIdWidgetStyle.CustomRadioGroup / CustomListbox
Probe helpers: Microsoft ChooseFirstVisible(SelectorProbe[]) pure walk

Traversal / hop / apply-link / redirect (existing)¶

No dedicated TRAVERSAL lab. Closest existing pieces:

Concern	Where
Iframe hop	Icims `DecideFrameContext` + `ContentIframe`/`InnerIframe` chains; tests in `IframeSwitchLogicTests.cs`
Apply-link preference	`AtsDetector.DetectFromDescriptionPage` — prefer `/apply` over asset script
LinkedIn redirect unwrap	`AtsDetectorTests.UnwrapsLinkedInExternalApplyRedirect_*` — decode nested `url=`
Relative redirect	same tests, relative `/jobs/view/externalApply/...`
AtsKind from synthesized apply URL	Adp/Paycom/Dayforce lab tests (`AtsDetector_classifies_a_synthesized_*_apply_link`)
Client-side "sign-in" redirect	Adp/Paycom engines: fixture redirect, no external IdP

Key detector tests:
C:\Users\fives\source\repos\WW-ats-labs\tests\WorkWingman.Tests\AtsDetectorTests.cs:113-168

Pure-logic extraction pattern (house style for TRAVERSAL)¶

Icims model for browser-free decisions:

// tools/.../IcimsAutomationEngine.cs
public static FrameContext DecideFrameContext(
    bool hasOuterIframe, bool fieldFoundInOuter,
    bool hasInnerIframe, bool fieldFoundInInner)
=> !hasOuterIframe ? FrameContext.TopDocument
 : fieldFoundInOuter ? FrameContext.OuterFrame
 : hasInnerIframe && fieldFoundInInner ? FrameContext.InnerFrame
 : FrameContext.NotFound;

Tests assert each branch + "outer wins even if inner exists" + chain order (#icims_content_iframe first).

Fixture / Playwright usage¶

Layer	Pattern
Fixture	`Fake*Site` = in-process `HttpListener` on `127.0.0.1`
Variation	per-iteration RNG: drop anchors, rename ids, hash classes, skip pages, hydration delay
Engine	Playwright against fixture only
Metrics	JSONL + hit/miss + primary vs fallback; Google also wait/timeout ms
Unit tests	mostly no browser — pure selectors + pure decisions
Live loopback tests	Meta `SubmitGuardAndEngineTests` (Playwright vs fixture)
Mutation	Stryker scoped to selectors/variation/telemetry pure files

Key full paths (copy pack for TRAVERSAL LAB)¶

Template / structure - C:\Users\fives\source\repos\WW-ats-labs\docs\ATS-LABS-TODO.md - C:\Users\fives\source\repos\WW-ats-labs\tools\WorkWingman.ScraperLab.Lever\ (canonical template)

Icims (philosophy + iframe hop) - ...\src\WorkWingman.Infrastructure\Automation\IcimsSelectors.cs - ...\tools\WorkWingman.ScraperLab.Icims\IcimsAutomationEngine.cs - ...\tests\WorkWingman.ScraperLab.Icims.Tests\IcimsSelectorAndJudgementTests.cs - ...\tests\WorkWingman.ScraperLab.Icims.Tests\IframeSwitchLogicTests.cs - ...\docs\technical\scraper-automation-learnings-icims.md

Meta (negative class ban) - ...\tools\WorkWingman.ScraperLab.Meta\MetaSelectors.cs - ...\tests\WorkWingman.ScraperLab.Meta.Tests\MetaSelectorsTests.cs - ...\docs\technical\scraper-automation-learnings-meta.md

Google (deep chain + telemetry) - ...\tools\WorkWingman.ScraperLab.Google\GoogleSelectors.cs - ...\tools\WorkWingman.ScraperLab.Google\SelectorChain.cs - ...\tests\WorkWingman.ScraperLab.Google.Tests\GoogleLabTests.cs

Microsoft - ...\src\WorkWingman.Infrastructure\Automation\MicrosoftSelectors.cs - ...\tests\WorkWingman.ScraperLab.Microsoft.Tests\MicrosoftSelectorTests.cs

Apply-link / redirect (pre-lab, for TRAVERSAL) - ...\src\WorkWingman.Infrastructure\Automation\AtsDetector.cs (impl) - ...\tests\WorkWingman.Tests\AtsDetectorTests.cs

Recipe to write TRAVERSAL LAB in same style¶

Tool project tools/WorkWingman.ScraperLab.Traversal/ (or name matching hop domain): Exe, net10.0, Playwright, Core+Infrastructure refs, namespace WorkWingman.ScraperLab.Traversal.
TraversalSelectors.cs (or hop stages): static ordered string[] / factory chains; XML doc states lead strategy + what is banned.
Pure decision API on engine (like DecideFrameContext): hop success/fail/redirect-unwrap without browser.
FakeTraversalSite.cs: loopback multi-hop pages (listing → redirect wrapper → apply).
SiteVariation: drop apply link, wrap LinkedIn redirect, asset-before-apply, relative href.
Tests tests/WorkWingman.ScraperLab.Traversal.Tests/:
positive: lead selector/order
negative philosophy (e.g. never prefer script asset over apply; never keep outer LinkedIn URL as ApplyUrl)
[Theory] over hop topologies / wrapper shapes
Invariants: no real egress; submit locate-only if form reached; log-and-continue loop; JSONL metrics.
Learnings doc docs/technical/scraper-automation-learnings-traversal.md mirroring Icims/Meta structure.

Philosophy test naming house style:
Field_chains_lead_with_<strategy>_not_<anti-pattern> / NoChain_LeadsWithAnExactIdSelector / Core_chain_never_contains_a_bare_class_selector.I have the house pattern. Pulling AtsDetector, iCIMS selector philosophy, and any posting-id patterns so the hop map stays implementable.I have the house pattern and the measured hop chain. Delivering the full Traversal Lab spec as an implementable lab document — philosophy, ordered chains, negatives, hop map from the Adzuna payload, and philosophy-asserting tests.Already on the ats-labs worktree. Writing the full Traversal Lab spec into docs/technical.# Traversal Lab spec — delivered

Written to:

Planned path: docs/technical/scraper-lab-traversal-spec.md (the transcript records it on the separate ats-labs branch; it is not present in this tree).

on branch wt/WT-a17e/ats-labs. It follows the house ScraperLab pattern (static ordered chains + philosophy-asserting tests, same shape as iCIMS id$= / Meta “no bare class” / Google data-fieldid first).

What it encodes¶

Hop map from the Adzuna API object (not a bare URL)¶

Hop	Surface	Class	Action	Fail closed
0	API → `redirect_url`	start	Build `TraversalStart` (`id`, `redirect_url`, `adref` as metadata only)	bad/missing URL → Unknown
1	`adzuna.com/land` → `/details`	Blocked	Headful warm profile; consent dismiss; `a[href*='aztt=']`	403 / 200-challenge → Blocked; no `aztt` → Unknown
2	ZipRecruiter	Aggregator	External apply only	same-host Easy Apply only → Unknown
3	`jobs.spectrum.com`	Employer	Anchor href host wins	body first-marker → Lever is banned; `/application-process` banned
4	`careers-charter.icims.com/jobs/77665/…`	Ats	`TerminalAts`, Kind=Icims, PostingId=`77665`	—

Per-class philosophy (mirrors iCIMS “why the lead selector”)¶

Class	Structural truth	Lead rule
Blocked	CloudFront/403; headful works; fresh context hides `aztt`	Real browser as itself + warm profile; never stealth
Aggregator	Reseller, never SoR	Outbound external only; unwrap nested redirects
Employer	Vanity front-end; multi-marker body is a trap	ATS-host anchor href before any body scan
Ats	System of record	Terminate; extract posting id only when confident
Portal	Login-walled by design (USAJobs)	`TerminalPortal`, not form-fill

Hard negatives (measured traps as tests)¶

Spectrum multi-marker HTML → must not resolve to Lever
/application-process must never win
HTTP 200 + CloudFront body → StopBlocked
Adzuna is Blocked, not Aggregator
gh_src is never a posting id; gh_jid is

Posting IDs¶

HIGH: iCIMS /jobs/{digits}; Greenhouse gh_jid + greenhouse /jobs/{digits}
UNKNOWN (explicit): Workday, Lever, most others — null beats a wrong dedupe key
Adzuna API id = listing id only, never ATS posting id

Implementable skeleton¶

Spec includes project layout (tools/WorkWingman.ScraperLab.Traversal/ + tests), pure types (HostClass, HopDisposition, TraversalStart), ordered selector arrays, pure Decide* APIs, FakeTraversalSite multi-hop loopback, and xunit philosophy tests in house style (Assert.DoesNotContain, [Theory]/[InlineData] on measured URLs).

Explicit conflict with today’s code¶

AtsDetector.DetectFromDescriptionPage is first-marker-wins on body HTML. That is the Spectrum→Lever bug. Traversal must not use it as the hop oracle; promotion step is to stop call sites that do.

Not yet implemented — this is the lab spec only (selectors + tests as skeletons). Next build step is pure HostClassifier / *HopRules / PostingIdExtractor + offline fake multi-hop, then warm headful Adzuna behind the app profile (out of default CI).

Hard line held: no stealth, no webdriver patch, no CAPTCHA, no proxy dodge — refuse → Unknown/Blocked.