Skip to content

WorkWingman — Job Signals (Technical)

Two enrichment signals surfaced on a job posting, both built to a hard rule: every source must be legally/officially published or a read-only parse of a page the user already opened — never a bulk scrape that violates a site's Terms of Service. This document covers what each signal shows, exactly where the data comes from, the legal/ToS reasoning, and caching/rate guidance.

Signal A — LinkedIn repost count + original post date

What it shows: how many times a LinkedIn job posting has been reposted, its original post date (when LinkedIn renders one distinctly), and the current "posted N ago" age text — in addition to the existing PostedAt field already scraped by LinkedInJobsScraper.

Feasibility verdict: legal only in a narrow read-only form — not as a bulk feature. LinkedIn has no public API for a job posting's repost history or original post date. Third-party tools that claim to expose this either (a) scrape LinkedIn's rendered pages in bulk, which violates LinkedIn's User Agreement (automated data collection / scraping is explicitly prohibited) and risks account suspension — already flagged as a risk for this project — or (b) don't actually have the data. WorkWingman does not build a bulk scraper for this signal.

What WorkWingman actually does: LinkedInPostingSignalExtractor (src/WorkWingman.Infrastructure/Automation/LinkedInPostingSignalExtractor.cs) parses the HTML of one job-detail page the user's own browser session already navigated to via the existing LinkedInJobsScraper (the same page load that already extracts title, company, description, etc.). It reads the text LinkedIn's own UI already rendered — the "Reposted N days ago" / "Reposted N times" badge and the "Posted N ago" text — using HtmlAgilityPack (already a project dependency) with a layout-tolerant fallback selector chain, the same philosophy as LinkedInJobsScraper.FirstTextAsync. It is:

  • Read-only — no writes, no interaction, just text extraction from markup already in memory.
  • One page at a time — it runs exactly once per job page the scraper already visits; it adds no additional page loads, no additional requests, and no hidden/private API calls.
  • Rate-normal — it rides along on the existing 900ms-paced scrape (BrowserSession.PageDelay); it does not increase LinkedIn traffic at all.

Fields returned (LinkedInPostingSignal, src/WorkWingman.Core/Models/LinkedInPostingSignal.cs):

Field Type Reliability
RepostCount int? Populated when LinkedIn renders "Reposted N times"; defaults to 1 when it renders bare "Reposted" with no count; null when no repost text is present at all.
OriginalPostedDate DateOnly? Populated only when LinkedIn renders a distinct "Originally posted …" string. LinkedIn does not reliably render this — most postings only show the current "Reposted/Posted N ago" text, not a separate original date. Expect this to be null most of the time.
CurrentPostedAgeText string? The raw "N days/weeks/months ago" / "just now" text, when present.
IsUnavailable bool true when every field above is null — a normal, expected outcome, not an error.

Graceful degradation is the default, not the exception. Because LinkedIn's markup changes frequently and rarely exposes a true "original post date," most jobs will come back with RepostCount = null and OriginalPostedDate = null. The extractor never throws and never guesses; a null field means "not rendered," not "parse failed."

ToS/legal note: this feature must stay exactly as scoped — one page, read-only, rider on the user's own legitimate scrape. Do not: add a scheduled/background job that revisits job pages solely to refresh this signal; add a bulk crawl across search results; or call any undocumented/internal LinkedIn API. If a future contributor is tempted to do any of those to get more reliable data, that is the point to stop — it moves the feature from "legal read of a page the user opened" to "the ToS-violating bulk scraping this design deliberately avoids."

Signal B — Company layoff history

What it shows: for a job's company, the date of the last layoffs, the number of employees affected, and (when derivable) the percent of the workforce — sourced only from legally/officially published records.

Sources evaluated and the decision:

Source Verdict Why
State WARN Act notices Used (WarnNoticeSource) Federal law (29 U.S.C. § 2101 et seq., the Worker Adjustment and Retraining Notification Act) requires covered employers (100+ employees) to file 60-day advance layoff notices with the state; states publish them as a public record. This is the strongest legal primary source for exact counts + dates.
SEC EDGAR full-text search Used (SecEdgarSource) Free, official, unauthenticated SEC API (https://efts.sec.gov/LATEST/search-index). Public companies must disclose material restructuring/workforce reductions under Form 8-K Item 2.05 ("Costs Associated with Exit or Disposal Activities"). Coverage is limited to public companies that filed a qualifying 8-K.
layoffs.fyi Adapter registered, but intentionally a no-op today (LayoffsFyiSource) layoffs.fyi's own press page states data is "free to use with attribution" editorially, but documents no public API, no CSV/JSON export, and no bulk endpoint — a full data export is available only "upon request" to the site's maintainer. The only programmatic paths that exist (e.g. third-party Apify actors) work by scraping the live site without a documented redistribution license — exactly the LinkedIn-style pattern this codebase avoids. See the class doc on LayoffsFyiSource for the full reasoning.

State coverage caveat (WARN): state labor departments do not publish WARN data uniformly. Texas Workforce Commission publishes its WARN notices as structured JSON on the Texas Open Data Portal (Socrata SODA API) — confirmed live, queryable, and exactly the shape WarnNoticeSource targets. California (EDD), New York (DOL), and Washington (ESD) also publish WARN notices, but as an XLSX download, a Tableau dashboard, or PDF filings rather than a queryable API — pulling those into this adapter would mean either scraping a dashboard or a scheduled-download-and-parse job, which is a different integration shape. WarnNoticeSource is deliberately parameterized by feed URL (feedUrl constructor parameter, defaulting to WarnNoticeSource.TexasFeedUrl) so adding a state with a genuine open-data JSON/CSV endpoint is a one-line change; states that only publish XLSX/dashboards are out of scope until a separate scheduled-download adapter is built.

Normalized model (LayoffEvent, src/WorkWingman.Core/Models/LayoffEvent.cs):

public class LayoffEvent
{
    public string CompanyName { get; set; }
    public DateOnly Date { get; set; }
    public int? Count { get; set; }              // null when the source doesn't report a headcount
    public double? WorkforcePercent { get; set; } // null for every source today (see below)
    public LayoffSourceKind Source { get; set; }  // WarnNotice | SecEdgar | LayoffsFyi
    public string SourceUrl { get; set; }          // direct link for attribution
}

WorkforcePercent is always null today. Neither the WARN notice fields nor SEC EDGAR's full-text search metadata carry a workforce-percentage figure directly — that number typically lives inside the free-text body of an 8-K filing (e.g. "approximately 8% of our global workforce"), which would require parsing filing prose, not just search-result metadata. This is flagged rather than guessed at.

Aggregation (CompanyLayoffService, src/WorkWingman.Infrastructure/Enrichment/CompanyLayoffService.cs): queries every registered ILayoffSourceAdapter (WARN, EDGAR, layoffs.fyi), merges the results, deduplicates events that share a company name (case-insensitive) and date — keeping the highest-priority source's record (WARN's hard headcount > EDGAR's filing > layoffs.fyi) — and returns the list newest-first. One source failing (network error, malformed response) never blocks the others; each adapter's LookupAsync degrades to an empty list rather than throwing.

Source adapter interface:

public interface ILayoffSourceAdapter
{
    LayoffSourceKind Source { get; }
    Task<IReadOnlyList<LayoffEvent>> LookupAsync(string companyName, CancellationToken ct = default);
}

Wiring

Both signals are registered in Program.cs the way every other service in this codebase is — plain IServiceCollection singletons, no special-case DI:

builder.Services.AddSingleton<ILayoffSourceAdapter, WarnNoticeSource>();
builder.Services.AddSingleton<ILayoffSourceAdapter, SecEdgarSource>();
builder.Services.AddSingleton<ILayoffSourceAdapter, LayoffsFyiSource>();
builder.Services.AddSingleton<ICompanyLayoffService, CompanyLayoffService>();

LinkedInPostingSignalExtractor is a public static class (no DI registration needed) — a pure parse function, the same pattern as the existing AtsDetector.

JobPosting (src/WorkWingman.Core/Models/JobPosting.cs) carries both signals directly on the job model:

public LinkedInPostingSignal? PostingSignal { get; set; }
public List<LayoffEvent> LayoffHistory { get; set; } = [];

Where each field actually gets populated:

  • PostingSignal is set inside LinkedInJobsScraper.ScrapeJobDetailAsync — right after the page's HTML is fetched for the existing title/company/description/ATS extraction, the same HTML is handed to LinkedInPostingSignalExtractor.ExtractFromPageHtml. No extra request, no extra page load.
  • LayoffHistory is set inside JobQueueService.ResyncFromLinkedInAsync — for each newly-added job (by LinkedInUrl, after dedup against the existing queue), ICompanyLayoffService.GetLayoffHistoryAsync(job.Company) is called once and the result assigned before the job is persisted. Already-known jobs are not re-queried on every resync. A lookup failure/empty result leaves LayoffHistory as an empty list, never null, never blocking the rest of the sync.

Caching and rate guidance

  • LinkedIn posting signal: no caching needed beyond what already happens — it's parsed once, synchronously, from HTML already fetched during the existing scrape pass. There is no separate network call to cache.
  • WARN (Texas Socrata API): the Socrata SODA API has no documented hard rate limit for unauthenticated use but does throttle abusive traffic; cache lookups per company for at least 24 hours (layoff notices don't change intra-day) and reuse ResiliencePipelines.Default (3 retries, exponential backoff, handles 429).
  • SEC EDGAR full-text search: SEC.gov's fair-access policy expects a descriptive User-Agent (name + contact — see SecEdgarSource.UserAgent) and a self-imposed rate limit; treat 10 requests/second as a hard ceiling and cache per-company results for at least 24 hours. No API key required, no authentication.
  • layoffs.fyi: no calls are made (see above), so no rate guidance applies today. If a written data-sharing agreement with the maintainer is obtained in the future, wire the agreed-upon access method into LayoffsFyiSource and add caching consistent with that agreement's terms at that time.

Tests