Linkedin Gronktayvius
I'll produce a LinkedIn-only TRAVERSAL LAB spec in the existing ScraperLab style. First I'm mapping the house pattern and the LinkedIn scraper/parser code so the philosophy, selectors, and tests match what's already there.Mapping ScraperLab house pattern + LinkedIn code. Searching repos now.Focusing main WorkWingman + ATS lab trees; listing ScraperLab projects.Reading iCIMS lab (house pattern) + LinkedIn automation sources.# ScraperLab house pattern + LinkedIn / traversal map
Canonical tree: C:\Users\fives\source\repos\WorkWingman
1. EXISTING SCRAPER LABS¶
Inventory (main repo)¶
tests/WorkWingman.ScraperLab.*.Tests (assembly trait Category=Lab):
| Project | Files |
|---|---|
| WorkWingman.ScraperLab.Icims.Tests | IcimsSelectorAndJudgementTests.cs, IframeSwitchLogicTests.cs, AssemblyInfo.cs |
| WorkWingman.ScraperLab.Google.Tests | GoogleLabTests.cs, FixtureHydrationTests.cs |
| WorkWingman.ScraperLab.Meta.Tests | MetaSelectorsTests, SubmitGuardAndEngineTests, JudgementPauseTests, PickIdStyleBoundaryTests, SiteVariationRandomizerTests, LabMetricsTests |
| WorkWingman.ScraperLab.Microsoft.Tests | MicrosoftSelectorTests, MicrosoftEngineDecisionTests |
tools/WorkWingman.ScraperLab.* labs (exe harnesses):
Icims, Google, Meta, Microsoft, Lever, Greenhouse, Amazon, Adp, Dayforce, Oracle, Paycom, Paycor, SuccessFactors, Ukg, base WorkWingman.ScraperLab (Workday)
Also tests under tools/: Adp, Amazon, Dayforce, Greenhouse, Lever, Oracle, Paycom, SuccessFactors (+ Paycor nested)
NOT FOUND: WorkWingman.ScraperLab.LinkedIn / any LinkedIn ScraperLab
Extra labs only in WW-ats-labs: Ashby, Eightfold, Jobvite, Phenom, RippleHire, SalesforceCareers, SmartRecruiters, Workable (recon targets from docs/ATS-LABS-TODO.md)
iCIMS lab skeleton (canonical house pattern)¶
tools/WorkWingman.ScraperLab.Icims/
WorkWingman.ScraperLab.Icims.csproj # Exe, net10.0, IsPackable=false
Program.cs # offline loop CLI
FakeIcimsSite.cs # loopback HttpListener fixture
FixtureVariation.cs # per-iteration DOM variance
IcimsAutomationEngine.cs # lab engine (uses prod IcimsSelectors)
LabProfile.cs / LabVault.cs / LabMetrics.cs
LoopRunner.cs
tests/WorkWingman.ScraperLab.Icims.Tests/
AssemblyInfo.cs # [assembly: AssemblyTrait("Category","Lab")]
IcimsSelectorAndJudgementTests.cs
IframeSwitchLogicTests.cs
*.csproj → Core + Infrastructure + lab project
Namespaces
- Lab: WorkWingman.ScraperLab.Icims
- Tests: WorkWingman.ScraperLab.Icims.Tests
- Selectors (prod, shared): WorkWingman.Infrastructure.Automation.IcimsSelectors
- Test attrs: xunit [Fact], [Theory] + [InlineData], assembly AssemblyTrait("Category","Lab")
- Filter: dotnet test WorkWingman.slnx --filter Category!=Lab
csproj pattern (tools/.../Icims):
- OutputType=Exe, RootNamespace/AssemblyName = WorkWingman.ScraperLab.Icims
- Refs: Playwright, Core, Infrastructure, TestSupport
Selector philosophy (iCIMS) — OBSERVED comments + asserted tests¶
Prod comment (IcimsSelectors.cs:3-32):
- Nested iframe #icims_content_iframe (+ optional inner iApply)
- Generated ASP.NET ids → lead with ends-with/substring, not exact id
- Table layout; forced account gate; multi-page
- Opposite of Workday exact-data-automation-id-first
Philosophy assertion style (iCIMS):
```14:24:WorkWingman/tests/WorkWingman.ScraperLab.Icims.Tests/IcimsSelectorAndJudgementTests.cs public void Field_chains_lead_with_a_suffix_or_substring_match_not_an_exact_id() { Assert.StartsWith("input[id$='FirstName']", IcimsSelectors.FirstName[0]); // ... foreach (var chain in new[] { IcimsSelectors.FirstName, IcimsSelectors.LastName, IcimsSelectors.School }) Assert.DoesNotContain("[id='", chain[0]); }
**Theory/InlineData (degree similarity):**
```26:38:WorkWingman/tests/WorkWingman.ScraperLab.Icims.Tests/IcimsSelectorAndJudgementTests.cs
[Theory]
[InlineData("Bachelor of Science (BS)", "B.S. Game Development", true)]
[InlineData("Bachelor of Arts (BA)", "B.S. Game Development", false)]
[InlineData("High School Diploma", "B.S. Game Development", false)]
public void Degree_similarity_scores_reasonable_matches_higher(...)
Negative / must-never (iCIMS):
- Chains must not lead with exact [id='...'
- Low-confidence degree → RunStatus.AwaitingJudgement, never guess
- Iframe present but field missing → FrameContext.NotFound (no silent success)
- Bare iframe fallback deliberately absent from InnerIframe
- Submit located never clicked (Program/LoopRunner/FakeSite invariant)
- Account gate simulated never real
Meta philosophy (contrast): lead semantic #id, degrade substring/name/aria, never class (hashed React).
Microsoft philosophy: lead exact semantic id → name → aria/label.
Safety invariants (every lab, docs/ATS-LABS-TODO.md)¶
- Never click Submit (locate only)
- Never type stored passwords / never create accounts
- Loopback fixture only
2. LINKEDIN CODE (OBSERVED)¶
Core symbols¶
| Path | Symbol | Role |
|---|---|---|
src/.../Automation/LinkedInJobsScraper.cs |
LinkedInJobsScraper |
Saved-jobs harvest + detail scrape |
src/.../Automation/LinkedInJobLdParser.cs |
LinkedInJobLdParser |
schema.org JobPosting ld+json |
src/.../Automation/LinkedInVoyagerJobParser.cs |
LinkedInVoyagerJobParser |
Voyager job JSON (title/company/apply) |
src/.../Core/Models/JobSource.cs:8 |
JobSource.LinkedIn |
enum value |
src/.../Core/Interfaces/ILinkedInJobsScraper.cs |
ILinkedInJobsScraper |
scraper contract (Source => LinkedIn) |
src/.../Automation/AtsDetector.cs |
DetectFromDescriptionPage |
ATS kind + ApplyUrl from HTML/JSON |
src/.../Automation/RedirectChainAtsResolver.cs |
RedirectChainAtsResolver |
hop-follow for aggregators |
src/.../Services/JobQueueService.cs:339-347 |
IsResolutionEligible / ResolveCandidateAsync |
only SimplifyJobs/Adzuna |
Also LinkedIn profile stack (out of traversal scope but present):
LinkedInProfileImporter, LinkedInProfileLdParser, LinkedInDomProfileParser, LinkedIn*Extractor, LinkedInVoyagerProfileParser, primer Advisor/Primers/linkedin.md (profile SPA notes, not jobs).
Facts — comments vs behavior¶
Saved-jobs URLs (code behavior):
https://www.linkedin.com/jobs-tracker/ // preferred 2026
https://www.linkedin.com/my-items/saved-jobs/
https://www.linkedin.com/jobs/tracker/saved
Job id / /jobs/view/ (behavior):
- Harvest: a[href*="/jobs/view/"] via progressive scroll
- Voyager fetch: pathname match /jobs/view/(\d+)/ →
GET /voyager/api/jobs/jobPostings/${id} with csrf-token = JSESSIONID
Closed shadow DOM (comment + behavior, live-DOM probe 2026-07-09):
- Authenticated /jobs/view/ content in CLOSED shadow root
- ld+json often absent; Playwright shadow-piercing empty
- Survivors: document.title, body.innerText, same-origin Voyager
- Wait: poll body.innerText past ~4k chars (15s ceiling)
Apply URL / “company website” (behavior, not UI text scrape):
- Voyager path: applyMethod.*.companyApplyUrl under union key
e.g. com.linkedin.voyager.jobs.OffsiteApply
- Scraper: AtsDetector.DetectFromDescriptionPage(pageHtml + voyagerJson)
- Fallback: if job.Ats.ApplyUrl empty, set from voyager.ApplyUrl
- Comment explicitly: shadow wall keeps apply link out of pageHtml
Easy Apply button / “Apply on company website” UI selectors:
NOT FOUND in production code (only business-doc mention of competitors clicking Easy Apply).
JobSource.LinkedIn assignment: always on scrape path (Source = JobSource.LinkedIn).
Compressed key snippets (for TRAVERSAL LAB)¶
Detail scrape pipeline (LinkedInJobsScraper.cs:309-458):
1. Goto /jobs/view/{id}
2. Wait rendered content (shadow hydration)
3. Primary: LinkedInJobLdParser.Parse(ld+json)
4. Fallback: class-prefix DOM selectors
5. Last resort: title + innerText + Voyager
6. ATS: detector on HTML+JSON; fill ApplyUrl from Voyager if needed
Voyager apply extract (LinkedInVoyagerJobParser.cs:158-168):
// applyMethod.{unionKey}.companyApplyUrl
if (!data.TryGetProperty("applyMethod", out var am)) return null;
foreach (var union in am.EnumerateObject())
if (union.Value.TryGetProperty("companyApplyUrl", out var cu))
return cu.GetString();
Job id extraction in-page (LinkedInJobsScraper.cs:490-503):
const m = location.pathname.match(/\/jobs\/view\/(\d+)/);
fetch(`/voyager/api/jobs/jobPostings/${m[1]}`, {
headers: { 'csrf-token': jsession, 'accept': 'application/vnd.linkedin.normalized+json+2.1' }
})
Existing unit tests (NOT a ScraperLab)¶
tests/WorkWingman.Tests/LinkedInJobLdParserTests.cstests/WorkWingman.Tests/LinkedInVoyagerJobParserTests.cs(includes OffsiteApply fixture)tests/WorkWingman.Tests/LinkedInJobsScraperEmploymentTypeTests.cstests/WorkWingman.Tests/AtsResolutionTests.cs→RedirectChainAtsResolverTests
3. TRAVERSAL / APPLY-URL RESOLUTION (existing)¶
RedirectChainAtsResolver — the hop engine¶
Namespace: WorkWingman.Infrastructure.Automation
Interface: WorkWingman.Core.Interfaces.IAtsResolver
DI: Program.cs AddHttpClient<IAtsResolver, RedirectChainAtsResolver>()
Behavior (headers-only, max 5 hops):
- Follow 300/301/302/303/307/308 only (not whole 3xx)
- Accept known ATS only when hop URL passes AtsDetector host gate
- Never read body; never guess Unknown → ATS
- Safe URI: http(s), default ports, public DNS, no IP literals, no localhost
- Host spacing 1 req/s; process-wide concurrency 3
- Cache collection ats-resolutions; aggregator TTL 10d vs ATS host 30d
- Aggregator hosts (TTL class only): simplify.jobs, *.adzuna.com
Who calls resolution¶
```339:347:WorkWingman/src/WorkWingman.Infrastructure/Services/JobQueueService.cs private bool IsResolutionEligible(...) => ... sourceName is "SimplifyJobs" or "Adzuna" ...;
// ResolveAsync(candidate.LinkedInUrl) // field is historical name for listing URL
**LinkedIn is NOT in the resolution-eligible set.**
LinkedIn apply URL comes from scrape-time Voyager/`AtsDetector`, not redirect hops.
### Related guards
- `AtsDetector.ApplyUrlMatchesKind` — blocks driving vanity/unrelated host after mislabel
(comment names Voyager vanity risk explicitly, `AtsDetector.cs:97-105`)
- `SnapshotAnchorChecker` has its own manual redirect hop loop (reference anchors, not job ATS)
### Search hits that are NOT job-link traversal
- Path-traversal security (FleetDispatch, StudyVisualStore) — unrelated
- “Company Website” as **source-of-hire form option** in ATS labs — unrelated
- Job-board aggregators in `JobSourceSelector` / audience scoping — product routing, not hop resolve
---
## 4. LAB FILE SKELETON TEMPLATE (from real iCIMS)
tests/WorkWingman.ScraperLab.Template doc: `docs/ATS-LABS-TODO.md` says copy from **Lever**; iCIMS is the richest philosophy/negative-rule exemplar.
---
## 5. GAPS — nothing measured yet for a TRAVERSAL LAB
| Gap | Status |
|---|---|
| `WorkWingman.ScraperLab.LinkedIn` / Traversal lab project | **NOT FOUND** |
| Fake LinkedIn detail page (closed shadow / ld+json / Voyager JSON fixtures as site) | **NOT FOUND** (unit fixtures only for parsers) |
| Lab covering hop chain LinkedIn listing → companyApplyUrl → ATS host | **NOT FOUND** |
| Lab asserting Easy Apply vs OffsiteApply branching | **NOT FOUND** (parser has OffsiteApply fixture only) |
| Redirect resolution wired for `JobSource.LinkedIn` URLs | **NOT FOUND** (SimplifyJobs/Adzuna only) |
| ScraperLab measuring multi-hop aggregator vanity shortlinks → ATS | Partial: `RedirectChainAtsResolverTests` unit suite, **not** a ScraperLab loop |
| UI selectors for “Apply on company website” button | **NOT FOUND** |
| LinkedIn Easy Apply automation engine | **NOT FOUND** (by design posture: external ATS apply) |
| ScraperLab for hop-follow safety (SSRF/DNS/https-downgrade) | Unit tests exist; **no** ScraperLab |
### What a TRAVERSAL LAB would pin (from existing code, not invented selectors)
1. Job id from `/jobs/view/{digits}`
2. Voyager envelope shapes: embedded OffsiteApply + normalized `included`
3. `companyApplyUrl` → `AtsDetector` Kind + ApplyUrl host match
4. Shadow-wall fallback order: ld+json → DOM → title/innerText → Voyager
5. Negative: never treat LinkedIn host as ATS; never auto-drive vanity ApplyUrl when Kind/host mismatch (`ApplyUrlMatchesKind`)
6. Optional separate module: header-only hop resolve (reuse `RedirectChainAtsResolver` rules) for non-LinkedIn aggregators — already unit-tested
---
## 6. Exact names cheat-sheet
**2 defs-style summary:** 4 labs under `tests/ScraperLab.*` (Google/Icims/Meta/Microsoft) + ~15 tool labs; LinkedIn scrape/parse/apply-url live in Infrastructure; hop resolver exists but **not** LinkedIn-wired; **zero** LinkedIn/Traversal ScraperLab.I have the house pattern. Next I'll pull the LinkedIn scrapers, Voyager parser, ATS detector, and iCIMS philosophy tests so the lab spec uses real code shapes and assertion style.I'll measure what a public, non-session fetch gets from a `/jobs/view/` URL so the lab can separate OBSERVED walls from code-backed claims.# LinkedIn TRAVERSAL LAB Spec
**Host class:** `network` (LinkedIn-specific). Not an ATS, not a pure aggregator, not blocked.
**Role in the hop graph:** intermediate listing host that either (a) hands off to an employer ATS via Offsite Apply, or (b) terminates on-platform as Easy Apply / login wall / Unknown.
**Scope:** `/jobs/view/` traversal to employer apply URL only. Not profile import, not Easy Apply form fill, not stealth.
**Evidence legend**
| Tag | Meaning |
|---|---|
| **OBSERVED (code)** | Behavior encoded in `WorkWingman` today with comments/tests |
| **OBSERVED (fetch 2026-08-05)** | Measured this session with plain `HttpClient`-equivalent GET |
| **INFERRED** | Reasoned from structure; lab must fail closed until verified |
---
## 1. TRAVERSAL PHILOSOPHY
LinkedIn is a **session-gated job network**, not a system of record. The durable job key is the **numeric path id** in `/jobs/view/{id}`; the slug text is SEO decoration. The employer ATS URL is **not** a stable light-DOM `href` on the authenticated 2026 page shape — content lives in a **closed shadow root**, so Playwright CSS and even shadow-piercing locators miss the apply control. The page’s own same-origin **Voyager** record (`applyMethod.*.companyApplyUrl`) is the machine-readable hand-off. Guest HTML can show an **offsite-shaped** Apply button without ever embedding the destination URL (sign-in modal). **Never** treat `linkedin.com` as an ATS host, **never** invent a `companyApplyUrl`, and **never** “first-marker-wins” scan page text for greenhouse/lever/icims strings that appear in prose or recommendations.
Mirror of iCIMS: iCIMS leads with suffix ids because ASP.NET trees regenerate; LinkedIn leads with **Voyager union-key-tolerant JSON** (and path job id) because the light DOM is a lie.
---
## 2. ORDERED RULE / SELECTOR CHAIN
Most specific first. Playwright-compatible where UI is involved; pure C# for path/JSON.
### A. Input normalization (Hop 0)
| # | Rule | Why |
|---|---|---|
| A1 | Accept only `https://www.linkedin.com/jobs/view/...` (and `//` / `http` upgraded) | Other LI hosts (guest m., country TLDs) are **INFERRED** unstable for this lab |
| A2 | Extract job id: `pathname` match `/jobs/view/(?:[^/?#]*-)?(\d+)(?:/|$|\?)` → capture group 1 | Digits are stable; slug is not (**OBSERVED code** uses `/jobs/view/(\d+)/`) |
| A3 | Canonical detail URL: `https://www.linkedin.com/jobs/view/{id}/` | Strip query (`?refId=`, tracking) before hop cache keys |
| A4 | If no digits → **Unknown**, stop | Never invent id |
```csharp
// LinkedInTraversalSelectors — proposed lab-local static class
public static class LinkedInTraversalSelectors
{
/// <summary>Path job id. Lead rule for every hop. Never use slug tokens as the key.</summary>
public static readonly Regex JobIdFromPath = new(
@"/jobs/view/(?:[^/?#]*-)?(?<id>\d+)(?:/|$|\?)",
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Compiled);
public static string? TryGetJobId(string urlOrPath)
{
var m = JobIdFromPath.Match(urlOrPath);
return m.Success ? m.Groups["id"].Value : null;
}
public static string CanonicalViewUrl(string jobId)
=> $"https://www.linkedin.com/jobs/view/{jobId}/";
}
B. Session / wall classification (Hop 1)¶
| # | Predicate | Action |
|---|---|---|
| B1 | Final URL matches login wall shapes | STOP → Unknown (public-only path) or product path may wait for manual login — lab asserts wall, does not bypass |
| B2 | Guest SSR: meta[name=pageKey][content=d_jobs_guest_details] |
Guest mode: limited extraction only |
| B3 | Authenticated detail (not wall, not guest pageKey) | Full Voyager path |
Login wall URL predicates (OBSERVED code LinkedInJobsScraper.IsLoginWall):
url contains /login
url contains authwall
url contains /checkpoint
url contains /uas/
Guest apply UI (OBSERVED fetch 2026-08-05 on https://www.linkedin.com/jobs/view/4418859576/):
button#topbar-apply
text: "Apply"
icon class fragment: apply-button__offsite-apply-icon-svg
sibling/following:
div.contextual-sign-in-modal
data-impression-id="public_jobs_apply-link-offsite_contextual-sign-in-modal"
anchors under modal:
a[href*="/signup/cold-join"][href*="session_redirect="]
trk token: public_jobs_apply-link-offsite_contextual-sign-in-modal_join-link
What is NOT on guest HTML (OBSERVED fetch):
companyApplyUrl, OffsiteApply, applyMethod, Easy Apply string, ld+json JobPosting blocks, any absolute non-LinkedIn careers URL. The offsite shape is visible; the destination is not.
C. Apply hand-off resolution (Hop 2) — ordered¶
| # | Source | Selector / extract | When |
|---|---|---|---|
| C1 | Voyager JSON (authoritative for external apply) | In-page same-origin fetch (session cookies + JSESSIONID csrf) | Authenticated context only |
| C2 | Parse data.applyMethod.<unionKey>.companyApplyUrl |
Union key varies; property name is stable | OffsiteApply shape (OBSERVED code fixture) |
| C3 | Feed pageHtml + "\n" + voyagerJson into AtsDetector.DetectFromDescriptionPage |
Host markers in URL/JSON, not prose-first | When C1 returned body |
| C4 | If detector Kind set but ApplyUrl empty → fill from Voyager companyApplyUrl |
Existing scraper behavior | |
| C5 | AtsDetector.ApplyUrlMatchesKind(applyUrl, kind) |
Gate: refuse to drive mismatched vanity host | Before hand-off to ATS engine |
| C6 | ld+json JobPosting | script[type="application/ld+json"] → LinkedInJobLdParser |
Metadata only; does not replace C1 for apply URL |
| C7 | Light-DOM class-prefix chips | Title/company/description fallbacks only | Never primary apply source on 2026 auth shape |
| C8 | Guest: detect offsite-shaped Apply + sign-in modal | Assert “handoff gated”; no ATS URL | Fail closed |
Voyager fetch (OBSERVED code — do not re-invent):
// runs inside authenticated page context only
const m = location.pathname.match(/\/jobs\/view\/(\d+)/);
const jsession = document.cookie.split('; ')
.find(c => c.startsWith('JSESSIONID='))?.split('=')[1]?.replaceAll('"', '');
const r = await fetch(`/voyager/api/jobs/jobPostings/${m[1]}`, {
headers: {
'csrf-token': jsession,
'accept': 'application/vnd.linkedin.normalized+json+2.1'
}
});
return r.ok ? await r.text() : null;
Apply URL extraction order (OBSERVED code LinkedInVoyagerJobParser.ExtractApplyUrl):
1. data.applyMethod must be object
2. foreach union property under applyMethod
3. if child has string companyApplyUrl → return it
4. else null (Easy Apply / missing / unknown shape → no external hop)
UI text “Apply on company website”: NOT FOUND in production selectors. Product path does not click that button; it reads Voyager. Lab may assert UI text only if a future live probe records it — until then mark INFERRED / optional secondary, never primary.
D. What happens after companyApplyUrl (Hop 3)¶
| Outcome | Next action |
|---|---|
| Absolute https URL, host is known ATS marker | Terminal for this lab: emit (AtsKind, ApplyUrl, LinkedInJobId); downstream ATS lab owns posting-id extraction |
| Absolute https URL, employer vanity / unknown host | Emit ApplyUrl + AtsKind.Unknown (or detector Phenom fingerprint later); do not invent vendor |
Easy Apply / no companyApplyUrl |
Terminal on LinkedIn for external ATS purposes → ApplyMode = EasyApplyOrOnsite, no hop |
| Wall / no session / Voyager 401/403/null | Unknown, stop |
3. NEGATIVE RULES (must NEVER)¶
| # | Never | Why | Evidence |
|---|---|---|---|
| N1 | Never lead with exact hashed React class equality for apply | Classes churn; closed shadow empties them | OBSERVED code comments 2026-07-09 probe |
| N2 | Never treat first host-marker string in page HTML as ATS when LinkedIn host is still the page | Recommendations / “people also viewed” / description prose can mention competitors | Spectrum-class trap applied here |
| N3 | Never resolve Apply destination from guest sign-in modal session_redirect |
Redirect points back to LinkedIn view URL, not employer ATS | OBSERVED fetch |
| N4 | Never use slug text as job id (ai-application-security-engineer-at-stifel-...) |
Only trailing digits are the id | OBSERVED fetch both shapes present |
| N5 | Never claim LinkedIn host is AtsKind anything |
LinkedIn is not an ATS in AtsDetector |
OBSERVED code |
| N6 | Never drive Easy Apply form as if it were external ATS | Product posture: external ATS apply; Easy Apply is LinkedIn-hosted | OBSERVED code absence of Easy Apply engine |
| N7 | Never pick first Company in Voyager included |
Decoys / ads / “people also viewed” | OBSERVED tests URN match / multi-company null |
| N8 | Never deep-scan root for companyResolutionResult.name |
Leaks decoy included entities | OBSERVED code scans job node only |
| N9 | Never patch navigator.webdriver, spoof UA/TLS, rotate proxies, solve CAPTCHA, use stealth plugins |
Hard product line | Constraint |
| N10 | Never invent companyApplyUrl when Voyager omits it |
Wrong ATS URL corrupts apply + dedupe | Philosophy |
| N11 | Never use cold automated public GET as proof of apply URL | Guest shows offsite shape only | OBSERVED fetch |
| N12 | Never reorder chain to put light-DOM apply click before Voyager | Would reintroduce shadow-wall false negatives | Philosophy assertion |
4. TERMINATION PREDICATE¶
function Decide(linkedinUrl, context):
id = TryGetJobId(linkedinUrl)
if id is null → FailClosed(Unknown, reason: BadUrl)
if context is PublicOnly:
// Lab + constrained product mode
fetch guest HTML without login
if login wall URL → FailClosed(Unknown, LoginWall)
if guest offsite Apply + sign-in modal and no companyApplyUrl in HTML
→ FailClosed(Unknown, HandoffGatedByLogin) // NOT a success
// Do not click Apply. Do not open modal. Stop.
if context is ProductSession (existing BrowserSession persistent profile):
open CanonicalViewUrl(id)
if IsLoginWall(finalUrl) or login form visible:
→ FailClosed(Unknown, LoginWall) for pure public lab
// Product may headed-wait for manual login; lab does not automate credentials
voyager = FetchVoyagerJobJson()
if voyager is null → FailClosed(Unknown, VoyagerUnavailable) // keep LinkedInUrl; no ATS hop
apply = ExtractApplyUrl(voyager) // companyApplyUrl under any union key
if apply is null or empty:
→ Terminal(LinkedInEasyOrOnsite, LinkedInJobId=id, ApplyUrl="") // no external hop
if apply host is linkedin.com → FailClosed(Unknown, SelfHandoff) // never hop to self
ats = AtsDetector.DetectFromDescriptionPage(html + voyagerJson)
if ats.ApplyUrl empty → ats.ApplyUrl = apply
if ats.Kind != Unknown && !ApplyUrlMatchesKind(ats.ApplyUrl, ats.Kind):
→ Terminal(UnknownKindOrVanity, ApplyUrl=apply, LinkedInJobId=id) // URL kept, engine not auto-driven
→ Terminal(ExternalAts, Kind=ats.Kind, ApplyUrl=ats.ApplyUrl, LinkedInJobId=id)
// NEXT hop is employer/ATS lab — out of LinkedIn lab scope
Keep hopping vs stop
| State | Decision |
|---|---|
| Login wall / authwall / checkpoint | STOP Unknown |
| Guest offsite Apply, no destination URL | STOP Unknown (gated) |
| Voyager missing / failed | STOP Unknown |
companyApplyUrl present, non-LinkedIn host |
STOP this lab; answer = that URL (+ detector Kind) |
No companyApplyUrl (Easy Apply / onsite) |
STOP; LinkedIn-terminal for external ATS |
| LinkedIn never “keep hopping” to another LinkedIn URL for apply resolution | One detail page is enough |
5. END-TO-END HOP MAP (LinkedIn-only)¶
Start object is not Adzuna. Product already imports LinkedIn as JobSource.LinkedIn. Two entry shapes:
Entry shape L0-A — saved-jobs harvest (product)¶
| Know | User has persistent BrowserSession profile; may be signed in |
| Need | List of /jobs/view/{id} URLs |
| Action | Open SavedJobsUrls in order: /jobs-tracker/ → /my-items/saved-jobs/ → /jobs/tracker/saved (OBSERVED: tracker must lead) |
| Selector | a[href*="/jobs/view/"] progressive scroll harvest |
| Fail closed | Login wall without user; zero links after all three list shapes |
Entry shape L0-B — single listing URL¶
| Know | A string URL or job id from queue row LinkedInUrl |
| Need | Canonical id |
| Action | A1–A4 normalization |
| Fail closed | No digits |
Hop L1 — open detail¶
| Know | Canonical .../jobs/view/{id}/ |
| Need | Guest vs auth vs wall; rendered content |
| Action | Playwright Goto DOMContentLoaded; if auth, WaitForRenderedContent body.innerText ≥ ~4000 chars / 15s (OBSERVED code) |
| Fail closed | Timeout + empty title → null job / Unknown |
Hop L2 — machine apply URL¶
| Know | Session cookies on linkedin.com (auth path) |
| Need | companyApplyUrl or explicit absence |
| Action | In-page Voyager GET /voyager/api/jobs/jobPostings/{id} with csrf = JSESSIONID |
| Fail closed | No JSESSIONID, non-OK response, garbage JSON → ApplyUrl null |
Hop L3 — classify external vs onsite¶
| Know | Voyager parse result |
| Need | Whether to leave LinkedIn |
| Action | If companyApplyUrl → AtsDetector + ApplyUrlMatchesKind; else Easy/Onsite terminal |
| Fail closed | Self-link to linkedin.com; Kind/host mismatch → do not auto-drive wrong engine |
Hop L4 — hand-off (out of lab)¶
| Know | Employer ATS or vanity URL |
| Need | That vendor’s posting id / form engine |
| Action | Existing ATS labs / RedirectChainAtsResolver only if intermediate aggregator (LinkedIn itself is not in IsResolutionEligible today — OBSERVED code SimplifyJobs/Adzuna only) |
| Fail closed | Unknown vendor → store URL, Kind Unknown |
6. PUBLIC-ONLY WALL (plain statement)¶
| Context | What you get | Apply hand-off |
|---|---|---|
| Plain HTTP GET, no cookies (OBSERVED fetch 2026-08-05) | HTTP 200, pageKey=d_jobs_guest_details, SSR title/h1/description, offsite-shaped Apply + sign-in modal |
No employer URL in HTML |
| Headless/headful Playwright without real user session | Expected: authwall / login / guest modal (not re-measured with Playwright this session → lab records as must probe, fail closed if wall) | None |
| Product persistent profile with user-completed login (OBSERVED code) | Authenticated /jobs/view/; closed shadow; Voyager works |
companyApplyUrl when OffsiteApply |
Hard line: lab does not log in, does not store passwords, does not bypass the modal. Public-only mode’s correct answer is often Unknown + reason HandoffGatedByLogin, not a fabricated careers URL.
Legitimate product state (already shipped, not a bypass):
- Persistent Chromium profile via
BrowserSession - Manual sign-in in headed window when wall appears (180s)
- Same-origin Voyager as the page itself uses
- No stealth, no fingerprint spoofing
7. POSTING ID EXTRACTION¶
| System | Extraction | Confidence |
|---|---|---|
| LinkedIn job id | Path digits: /jobs/view/(?:[^/?#]*-)?(\d+) |
HIGH — OBSERVED code + OBSERVED fetch (4418859576 in path, slug, and body) |
| LinkedIn dedupe key | Prefer linkedin:{jobId}; do not use raw full URL with tracking query |
HIGH (design) |
| External ATS posting id from LinkedIn alone | UNKNOWN until Hop L3 URL is classified | Never regex-guess from LinkedIn description |
Greenhouse gh_jid on vanity |
Only after hop lands on URL containing gh_jid / greenhouse host |
N/A until hand-off URL known |
iCIMS path /jobs/{n}/ |
Only after hop lands on *.icims.com |
N/A until hand-off |
OffsiteApply fixture URL https://careers.internationalmotors.com/apply/456 |
456 is fixture-only, not a general careers.com rule |
Do not generalize |
Wrong regex on external id silently corrupts dedupe — LinkedIn lab emits LinkedIn id + raw ApplyUrl; ATS lab owns vendor id.
8. TEST ASSERTIONS (house style)¶
Project skeleton (mirror iCIMS):
tools/WorkWingman.ScraperLab.LinkedInTraversal/ # optional offline CLI later
tests/WorkWingman.ScraperLab.LinkedInTraversal.Tests/
AssemblyInfo.cs # [assembly: AssemblyTrait("Category","Lab")]
LinkedInTraversalPhilosophyTests.cs
LinkedInTraversalNegativeRulesTests.cs
LinkedInTraversalHopMapTests.cs
Selectors under test: lab-local LinkedInTraversalSelectors + prod LinkedInVoyagerJobParser / LinkedInJobsScraper / AtsDetector (assert philosophy against prod where it already encodes the rules).
Philosophy tests¶
using WorkWingman.Infrastructure.Automation;
namespace WorkWingman.ScraperLab.LinkedInTraversal.Tests;
/// <summary>
/// LinkedIn TRAVERSAL philosophy: path digits are the job key; Voyager companyApplyUrl is the
/// external hand-off; light-DOM / guest Apply UI never invents an ATS URL; linkedin.com is never an ATS.
/// </summary>
public class LinkedInTraversalPhilosophyTests
{
[Fact]
public void Job_id_chain_leads_with_path_digits_not_slug_tokens()
{
// Unlike guest SEO slugs, the stable key is always the trailing numeric id.
Assert.Equal("4418859576",
LinkedInTraversalSelectors.TryGetJobId(
"https://www.linkedin.com/jobs/view/4418859576/"));
Assert.Equal("4418859576",
LinkedInTraversalSelectors.TryGetJobId(
"https://www.linkedin.com/jobs/view/ai-application-security-engineer-at-stifel-financial-corp-4418859576"));
Assert.Equal("4418859576",
LinkedInTraversalSelectors.TryGetJobId(
"https://www.linkedin.com/jobs/view/4418859576/?refId=abc&trackingId=xyz"));
}
[Theory]
[InlineData("https://www.linkedin.com/jobs/view/4418859576/", "4418859576")]
[InlineData("https://www.linkedin.com/jobs/view/4418859576", "4418859576")]
[InlineData("/jobs/view/4418859576/", "4418859576")]
[InlineData(
"https://www.linkedin.com/jobs/view/ai-application-security-engineer-at-stifel-financial-corp-4418859576",
"4418859576")]
[InlineData("https://www.linkedin.com/my-items/saved-jobs/", null)] // list page — no job id
[InlineData("https://careers.internationalmotors.com/apply/456", null)]
public void Job_id_extraction_theory(string url, string? expected)
=> Assert.Equal(expected, LinkedInTraversalSelectors.TryGetJobId(url));
[Fact]
public void Apply_url_chain_leads_with_voyager_companyApplyUrl_not_light_dom()
{
// Philosophy: first machine source for external apply is applyMethod.*.companyApplyUrl.
// If a future "UI click chain" is added, it must NOT reorder ahead of Voyager.
var ordered = LinkedInTraversalSelectors.ExternalApplySourceOrder;
Assert.Equal("VoyagerCompanyApplyUrl", ordered[0]);
Assert.DoesNotContain("ClickApplyButton", ordered.Take(1));
Assert.DoesNotContain("GuestSignInModalSessionRedirect", ordered);
}
[Fact]
public void Voyager_offsite_fixture_exposes_employer_url_not_linkedin_host()
{
// Verbatim shape already in LinkedInVoyagerJobParserTests (embedded OffsiteApply).
const string embedded = """
{
"data": {
"title": "Lead Software Engineer - Enterprise Applications",
"applyMethod": {
"com.linkedin.voyager.jobs.OffsiteApply": {
"companyApplyUrl": "https://careers.internationalmotors.com/apply/456"
}
}
}
}
""";
var v = LinkedInVoyagerJobParser.Parse(embedded);
Assert.Equal("https://careers.internationalmotors.com/apply/456", v.ApplyUrl);
Assert.DoesNotContain("linkedin.com", v.ApplyUrl!, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void Missing_companyApplyUrl_is_terminal_not_a_guess()
{
var v = LinkedInVoyagerJobParser.Parse("""
{ "data": { "title": "Engineer",
"applyMethod": {
"com.linkedin.voyager.jobs.ComplexOnsiteApply": { "easyApplyUrl": "/easy-apply/1" }
} } }
""");
// Even if Easy Apply union appears, without companyApplyUrl there is NO external hop.
// (ComplexOnsiteApply property names beyond this are INFERRED — assert null ApplyUrl only.)
Assert.Null(v.ApplyUrl);
}
[Fact]
public void Canonical_view_url_strips_tracking_and_uses_digits_only()
{
var id = LinkedInTraversalSelectors.TryGetJobId(
"https://www.linkedin.com/jobs/view/4418859576/?refId=x")!;
Assert.Equal("https://www.linkedin.com/jobs/view/4418859576/",
LinkedInTraversalSelectors.CanonicalViewUrl(id));
}
}
Negative rules tests¶
public class LinkedInTraversalNegativeRulesTests
{
[Fact]
public void LinkedIn_host_is_never_classified_as_an_ATS_kind()
{
var ats = AtsDetector.DetectFromDescriptionPage(
"<html>https://www.linkedin.com/jobs/view/4418859576/</html>");
// linkedin.com is not in ApplyUrlMarkers for any AtsKind
Assert.Equal(AtsKind.Unknown, ats.Kind);
Assert.Equal("", ats.ApplyUrl);
}
[Fact]
public void Guest_sign_in_session_redirect_must_not_be_treated_as_employer_apply_url()
{
// OBSERVED fetch 2026-08-05: modal join link redirects back to LinkedIn view, not ATS.
const string guestModalHref =
"https://www.linkedin.com/signup/cold-join?source=jobs_registration" +
"&session_redirect=https%3A%2F%2Fwww.linkedin.com%2Fjobs%2Fview%2F" +
"ai-application-security-engineer-at-stifel-financial-corp-4418859576" +
"&trk=public_jobs_apply-link-offsite_contextual-sign-in-modal_join-link";
var hop = LinkedInTraversalSelectors.ClassifyGuestApplyHref(guestModalHref);
Assert.Equal(GuestApplyHrefKind.LinkedInAuthGate, hop);
Assert.Null(LinkedInTraversalSelectors.TryGetExternalApplyUrlFromGuestHtml(
$"<a href=\"{guestModalHref}\">Join</a>"));
}
[Fact]
public void Description_prose_mentioning_lever_must_not_win_over_missing_companyApplyUrl()
{
// Spectrum-class trap: markers in text without an anchor/JSON apply URL.
const string html = """
<html><body>We migrated from Lever last year.
Apply on our careers site after you leave LinkedIn.
https://www.linkedin.com/jobs/view/4418859576/</body></html>
""";
var ats = AtsDetector.DetectFromDescriptionPage(html);
// Detector may set Kind=Lever from hostless string "Lever" — if it does, ApplyUrl must still
// fail ApplyUrlMatchesKind for any non-lever URL. Preferred lab assertion: without a lever.co
// URL in the blob, ApplyUrl stays empty (current ExtractUrl requires host marker in a URL).
Assert.True(ats.ApplyUrl.Length == 0 ||
AtsDetector.ApplyUrlMatchesKind(ats.ApplyUrl, ats.Kind));
Assert.DoesNotContain("linkedin.com", ats.ApplyUrl, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void Decoy_included_company_must_not_become_employer()
{
var v = LinkedInVoyagerJobParser.Parse("""
{
"data": { "title": "Backend Engineer",
"*companyDetails": "urn:li:fs_normalized_company:REAL" },
"included": [
{ "$type": "com.linkedin.voyager.organization.Company",
"name": "Decoy Ads Co", "entityUrn": "urn:li:fs_normalized_company:DECOY" },
{ "$type": "com.linkedin.voyager.organization.Company",
"name": "Real Employer", "entityUrn": "urn:li:fs_normalized_company:REAL" }
]
}
""");
Assert.Equal("Real Employer", v.Company);
Assert.NotEqual("Decoy Ads Co", v.Company);
}
[Fact]
public void Voyager_apply_to_linkedin_host_is_rejected_as_external_handoff()
{
var v = LinkedInVoyagerJobParser.Parse("""
{ "data": { "applyMethod": {
"com.linkedin.voyager.jobs.OffsiteApply": {
"companyApplyUrl": "https://www.linkedin.com/jobs/view/4418859576/"
} } } }
""");
Assert.False(LinkedInTraversalSelectors.IsExternalEmployerApplyUrl(v.ApplyUrl));
}
[Theory]
[InlineData("https://www.linkedin.com/login")]
[InlineData("https://www.linkedin.com/authwall?trk=guest")]
[InlineData("https://www.linkedin.com/checkpoint/challenge/abc")]
[InlineData("https://www.linkedin.com/uas/login-submit")]
public void Login_wall_urls_fail_closed(string url)
=> Assert.True(LinkedInJobsScraper.IsLoginWall(url));
[Fact]
public void Saved_jobs_list_order_leads_with_job_tracker()
{
// Live-DOM probe 2026-07-09: tracker first or harvest under-counts.
Assert.Equal("https://www.linkedin.com/jobs-tracker/",
LinkedInJobsScraper.SavedJobsUrls[0]);
}
}
Guest fixture negative (verbatim measured markers)¶
[Fact]
public void Guest_ssr_fixture_has_offsite_shape_but_no_employer_url()
{
// Minimal fixture distilled from OBSERVED fetch 2026-08-05 (job 4418859576).
// Full HTML saved locally during lab authoring; CI uses distilled markers only.
const string guest = """
<meta name="pageKey" content="d_jobs_guest_details">
<h1 class="top-card-layout__title topcard__title">AI-Application Security Engineer</h1>
<button id="topbar-apply" data-modal="job-details-subnav-apply-modal">
Apply
<icon data-svg-class-name="apply-button__offsite-apply-icon-svg"></icon>
</button>
<div class="contextual-sign-in-modal"
data-impression-id="public_jobs_apply-link-offsite_contextual-sign-in-modal"></div>
""";
var decision = LinkedInTraversalSelectors.DecideFromGuestHtml(guest, jobId: "4418859576");
Assert.Equal(TraversalTerminal.Unknown, decision.Terminal);
Assert.Equal(TraversalFailReason.HandoffGatedByLogin, decision.Reason);
Assert.Null(decision.ExternalApplyUrl);
Assert.True(decision.OffsiteApplyShaped);
// NEGATIVE: must not invent Stifel careers URL
Assert.DoesNotContain("stifel", decision.ExternalApplyUrl ?? "", StringComparison.OrdinalIgnoreCase);
}
Shadow-wall / source order¶
[Fact]
public void Detail_scrape_source_order_is_ld_then_dom_then_voyager_for_metadata_but_apply_is_voyager_led()
{
// Metadata (title/company): ld+json → DOM chips → Voyager → document title (prod scraper)
// Apply URL: Voyager companyApplyUrl (+ detector on html+json) — NOT DOM
var meta = LinkedInTraversalSelectors.MetadataSourceOrder;
Assert.Equal(new[] { "JsonLd", "DomClassPrefix", "Voyager", "DocumentTitle" }, meta);
var apply = LinkedInTraversalSelectors.ExternalApplySourceOrder;
Assert.Equal("VoyagerCompanyApplyUrl", apply[0]);
Assert.DoesNotContain("DomClassPrefix", apply);
}
9. PROPOSED LAB SELECTOR / RULE TYPE (implementable skeleton)¶
namespace WorkWingman.ScraperLab.LinkedInTraversal;
public enum TraversalTerminal { ExternalAts, LinkedInOnsiteOrEasyApply, Unknown }
public enum TraversalFailReason
{
None, BadUrl, LoginWall, HandoffGatedByLogin, VoyagerUnavailable, SelfHandoff
}
public enum GuestApplyHrefKind { LinkedInAuthGate, ExternalHttp, Unknown }
public static class LinkedInTraversalSelectors
{
public static readonly string[] MetadataSourceOrder =
["JsonLd", "DomClassPrefix", "Voyager", "DocumentTitle"];
public static readonly string[] ExternalApplySourceOrder =
["VoyagerCompanyApplyUrl", "AtsDetectorUrlInHtmlPlusVoyagerJson"];
// Guest SSR markers (OBSERVED fetch 2026-08-05) — philosophy tests, not auth path.
public const string GuestPageKey = "d_jobs_guest_details";
public const string GuestApplyButton = "#topbar-apply";
public const string GuestOffsiteIconFragment = "apply-button__offsite-apply-icon";
public const string GuestSignInModalImpression =
"public_jobs_apply-link-offsite_contextual-sign-in-modal";
// Auth path: no stable light-DOM apply selector is VERIFIED for closed shadow.
// Optional INFERRED secondary (do not enable until live probe with session records it):
// public const string InferredCompanyWebsiteButton = "text=Apply on company website";
public static bool IsExternalEmployerApplyUrl(string? url)
{
if (!Uri.TryCreate(url, UriKind.Absolute, out var u)) return false;
if (u.Scheme is not ("http" or "https")) return false;
var h = u.Host;
if (h.Equals("linkedin.com", StringComparison.OrdinalIgnoreCase)) return false;
if (h.EndsWith(".linkedin.com", StringComparison.OrdinalIgnoreCase)) return false;
if (h.EndsWith(".licdn.com", StringComparison.OrdinalIgnoreCase)) return false;
return true;
}
// ... TryGetJobId, CanonicalViewUrl, DecideFromGuestHtml, ClassifyGuestApplyHref ...
}
10. CLOSED SHADOW DOM — WHAT THE LAB PROVES¶
| Claim | Status |
|---|---|
| Authenticated detail content in closed shadow root | OBSERVED code (live-DOM probe 2026-07-09) |
| Playwright shadow-piercing locators empty | OBSERVED code |
Survivors: document.title, body.innerText, same-origin Voyager |
OBSERVED code |
Wait strategy: poll body.innerText length ≥ 4000, 15s |
OBSERVED code |
Guest SSR page still has light-DOM h1.top-card-layout__title and description markup |
OBSERVED fetch (guest is a different pageKey) |
| Apply control clickable in light DOM on auth page | NOT VERIFIED this session — do not assert a CSS apply chain for auth |
Lab stance: auth apply resolution is JSON-first; any future UI click path is additive after a logged live probe, never a reorder.
11. RELATION TO EXISTING CODE (no duplicate product path)¶
| Existing | Role |
|---|---|
LinkedInJobsScraper |
Product harvest + detail scrape (session) |
LinkedInVoyagerJobParser |
Pure apply/company/title parse |
LinkedInJobLdParser |
Metadata when ld+json present |
AtsDetector / ApplyUrlMatchesKind |
Kind + drive gate after hand-off URL known |
RedirectChainAtsResolver |
Not LinkedIn-wired; headers-only hop for SimplifyJobs/Adzuna |
JobQueueService.IsResolutionEligible |
LinkedIn excluded |
This lab pins philosophy so a future LinkedInTraversal helper cannot regress into: light-DOM-first apply, guest-modal-as-URL, linkedin-as-ATS, or decoy company/marker wins.
12. SAFETY INVARIANTS (lab + product)¶
- No Submit clicks (N/A for traversal; still state it).
- No credential automation; no password vault use for LinkedIn login in lab.
- No stealth / webdriver patch / UA spoof / proxy rotate / CAPTCHA solve.
- Public-only lab runs: loopback fixtures + distilled guest HTML fixture from measured markers; optional live guest GET that expects Unknown.
- Authenticated live loop: only on Andrew’s machine with his profile, offline CLI opt-in, never CI.
- Unknown is a first-class terminal result.
13. IMPLEMENTATION CHECKLIST (when ticketed)¶
- Add
LinkedInTraversalSelectors(lab or Infrastructure — prefer lab-first, promote if product needs). - Add
tests/WorkWingman.ScraperLab.LinkedInTraversal.TestswithCategory=Lab. - Distill guest fixture from measured HTML (markers only; no PII).
- Reuse
LinkedInVoyagerJobParserTestsOffsiteApply fixture as hop L2 golden. - Wire philosophy tests that fail if
ExternalApplySourceOrder[0]changes off Voyager. - Document public wall in lab README: Unknown ≠ bug when gated.
- Do not add LinkedIn to
RedirectChainAtsResolvereligibility without a separate design — hop is in-session Voyager, not HTTP 302 chain.
Measured sample (Hop L0-B example)¶
| Field | Value |
|---|---|
| Input URL | https://www.linkedin.com/jobs/view/4418859576/ |
| Job id | 4418859576 |
| Guest title | Stifel Financial Corp. hiring AI-Application Security Engineer in St Louis, MO \| LinkedIn |
| Guest h1 | AI-Application Security Engineer |
| Guest apply | Offsite-shaped #topbar-apply → sign-in modal |
| External ApplyUrl (public) | none — fail closed |
| External ApplyUrl (auth + OffsiteApply) | Only via Voyager companyApplyUrl — not measured this session without session |
Bottom line: LinkedIn traversal is id-from-path → session-or-wall → Voyager companyApplyUrl → AtsDetector gate → hand off. Guest offsite Apply without a destination is a measured dead end, not a puzzle to force open. That is the same honesty posture as iCIMS “never lead with exact id” — the structure of the host dictates the chain, and the tests fail if someone reorders it for convenience.