Skip to content

WorkWingman — Compensation & Diversity Signals (Technical)

Two more enrichment signals surfaced on a job posting, built to the same hard rule as job-signals: every source must be legally/officially published, a paid licensed API, or a read-only parse of a page the user already opened — never a bulk scrape that violates a site's Terms of Service. This document covers what each signal shows, exactly where the data comes from, the legal reasoning (including two honest "we looked, and there is no legal path" verdicts), and caching/rate guidance.

Signal A — Compensation insight (salary)

What it shows: for a job's role + location, up to three independently-attributed wage estimates side by side — one from actual H-1B sponsorship filings, one from the U.S. government's official occupational wage survey, and one read directly off the employer's own job posting when pay-transparency law required them to publish it.

Sources evaluated and the decision

Source Verdict Why
levels.fyi Rejected for the free path; config-gated seam for the paid path (LevelsFyiSource) levels.fyi's own Terms and Conditions explicitly prohibit "any manual or automated software, devices or other processes (including but not limited to spiders, robots, scrapers, crawlers ... data mining tools or the like) to 'scrape' or download data" from the site. There is no free public API. levels.fyi does sell an official API/MCP/CLI product exposing company/level/percentile compensation data through a documented, licensed endpoint — a legitimate path if a paid key is obtained. LevelsFyiSource exists as a registered, config-gated no-op: it never scrapes, and the real HTTP call against the paid API's schema is intentionally left unimplemented until a real subscription exists to build and test against (writing request/response mapping code against an unverified paid contract would be speculative, not honest).
Glassdoor Rejected outright, no adapter built Glassdoor closed public API access around 2021 and, as of 2024, restricts API access to enterprise partners only — no public developer signup exists. Glassdoor's Terms of Use explicitly prohibit "any robot, spider, scraper, data mining tools ... or other automated means to access the Services... without our express written permission." There is no free or self-serve paid path, so unlike levels.fyi there is nothing to build a config-stubbed seam against — this source is simply not built.
DOL/OFLC H-1B LCA disclosure data Used (H1bLcaSource) Employers must file a Labor Condition Application (LCA) with the DOL before sponsoring an H-1B/H-1B1/E-3 visa (20 CFR § 655.760), and OFLC publishes the disclosure data — including the actual wage the employer told the government it would pay for the role — as public performance data, by law. This is public-domain U.S. government data; community sites like h1bdata.info exist only to make the same official files searchable, they add no private data.
BLS OEWS wage survey Used (BlsOewsSource) The Bureau of Labor Statistics' Occupational Employment and Wage Statistics survey is free, official, and explicitly designed for programmatic use via the BLS Public Data API (registration key optional, raises the daily quota from 25 to 500 requests). No scraping, no ToS ambiguity — this is a documented government API.
Employer-posted pay range Used (PostedRangeSource) Colorado, California, Washington, New York, and other states now legally require employers to publish a good-faith pay range on job postings viewable by their residents — including remote postings. This is the strongest possible legal footing: the range isn't fetched from anywhere new, it's parsed from the exact posting the app already scraped for title/company/description.

Why levels.fyi and Glassdoor get different treatment: both ban scraping outright, but levels.fyi additionally sells a documented, licensed API — a legal door that's simply locked behind a subscription WorkWingman doesn't currently have. Glassdoor's enterprise-only access has no public application path or documented schema at all. A config-gated seam only makes sense where there's a real, checkable contract to eventually wire up; Glassdoor doesn't offer one.

Normalized model (CompensationInsight, src/WorkWingman.Core/Models/CompensationInsight.cs)

public class CompensationInsight
{
    public string Role { get; set; }
    public string Location { get; set; }
    public decimal? P25 { get; set; }      // 25th-percentile annual base pay
    public decimal? Median { get; set; }
    public decimal? P75 { get; set; }      // 75th-percentile annual base pay
    public CompensationSourceKind Source { get; set; } // H1bLca | BlsOews | PostedRange | LevelsFyi
    public string SourceUrl { get; set; }
    public DateOnly AsOf { get; set; }
}

Unlike layoff history, insights are never merged into one number. CompanyLayoffService dedupes/merges because multiple sources reporting the same event should collapse to one record. Compensation is different: an H-1B filing wage, a BLS occupational median, and this specific employer's posted range are three legitimately different numbers with different meanings — collapsing them into an average would hide exactly the comparison a job-seeker wants to make ("is this posting's range in line with what BLS says the occupation typically pays?"). CompensationInsightService therefore returns every source's insight independently, each keeping its own Source/SourceUrl/AsOf attribution.

Source adapters in detail

H1bLcaSource (src/WorkWingman.Infrastructure/Enrichment/H1bLcaSource.cs) reads a small local JSON extract (role/location/wage columns only) rather than calling a live remote API — OFLC does not publish a live query API, only quarterly bulk Excel/CSV file downloads. The extract is intended to be ETL'd out-of-band from the quarterly OFLC disclosure file on a matching quarterly cadence (a scheduled job, not built in this change — see "Refresh cadence" below) and dropped at H1bLcaSource.DefaultExtractPath (%USERPROFILE%\Wingman\data\h1b-lca-extract.json), overridable via constructor parameter for tests. Title matching is bidirectional and decoration-tolerant (LCA data has no stable role-taxonomy join key across postings): either title containing the other matches, and as a fallback the scraped title is reduced to a "core" — seniority words (Senior, Sr., Lead, Principal, Staff, …) dropped and anything after the first comma trimmed — so a real scraped title like Senior Software Engineer, AI Platform still matches an LCA row for Software Engineer instead of silently returning nothing. Worksite location is a case-insensitive substring match. Each matched wage is normalized to an annual figure first — OFLC discloses a wage rate plus its unit (Hour × 2080 / Week × 52 / Bi-Weekly × 26 / Month × 12 / Year × 1; a missing unit is assumed already-annual, OFLC's most common case), so an hourly $60/hour filing isn't skewed into the percentiles as a $60 annual salary. Percentiles use a simple nearest-rank calculation over the annualized wages, and AsOf is the latest real decision date among the matched filings (falling back to today only when no matched record carries a date at all — an honest "unknown vintage", never a claim that a stale figure is fresh). When the extract file doesn't exist yet, this degrades to an empty result — never a crash, never a fabricated number.

BlsOewsSource (src/WorkWingman.Infrastructure/Enrichment/BlsOewsSource.cs) calls the real BLS Public Data API (https://api.bls.gov/publicAPI/v2/timeseries/data/). BLS OEWS series IDs are a fixed 25-character code (OE + area type + area code + industry code + occupation code + datatype code) — there is no free-text "search by job title" endpoint, so this adapter is constructed with an explicit role/location -> (P25 series ID, median series ID, P75 series ID) map rather than attempting fuzzy title matching. A role/location pair with no configured mapping degrades to an empty result. The registration key (WorkWingman:BlsApiKey in configuration) is optional; without it the adapter still calls the API unauthenticated at the lower (25/day) quota rather than refusing to run.

Series mapping — why BLS is registered conditionally. Because there is no title-search endpoint, BlsOewsSource can only return something for role/location tuples present in its map, and it deliberately never guesses a 25-character series ID (guessing one would be exactly the fabricated-data failure this whole feature avoids). So an unmapped BlsOewsSource is inert — it returns [] for everything before making any HTTP call. To avoid a silent no-op that misleadingly implies live BLS coverage, Program.cs only registers BlsOewsSource into the pipeline once a verified map is actually loaded: BlsOewsSource.LoadSeriesMap(WorkWingman:BlsSeriesMapPath) reads an optional local JSON file (a list of { role, location, p25SeriesId, medianSeriesId, p75SeriesId } entries, each verified by a human against the live BLS API — same local-file, degrade-to-empty philosophy as H1bLcaSource's extract), and registration is skipped entirely when that map is empty/absent. IsConfigured reflects whether the source has any mapping. This keeps the shipped default honest: BLS is a real, buildable seam that lights up the moment a verified map is supplied, not a dead adapter pretending to work.

PostedRangeSource (src/WorkWingman.Infrastructure/Enrichment/PostedRangeSource.cs) is a pure text parse — no network call at all, same philosophy as LinkedInPostingSignalExtractor. It is deliberately conservative about false positives:

  • Bound to the pay phrase. It finds a pay-context phrase ("pay range", "salary range", "compensation range", "base salary", "annual salary") and then only accepts a $X - $Y range that appears within a short (160-char) window immediately after that phrase. So an unrelated dollar range elsewhere in the posting — e.g. a 401(k) match: $1 - $2 benefits line before the salary section — is never mistaken for the salary. It scans every pay-context phrase in order, not just the first, so an earlier phrase with no nearby range ("Base salary depends on experience… Pay range: $100k - $130k") doesn't suppress a real range that follows a later phrase; the first window that yields a valid range wins, and if none do it degrades to empty.
  • k shorthand is per-token. The trailing k/K ($100k → 100000) is captured on each amount individually in the regex, tied strictly to the matched salary token — never inferred from a fuzzy scan of surrounding text (which would wrongly fire on 401k benefits copy and inflate a $100,000 salary to $100,000,000).
  • Hourly ranges are annualized. When the posting expresses the range per hour — $50 - $70/hour, /hr, per hour, an hour, or hourly — the values are multiplied by 2080 (40 hrs/week × 52 weeks) before storage, so every value stays in the annual-base-pay unit CompensationInsight documents rather than silently surfacing $50 as an annual salary.

An inverted range (low > high) is rejected as malformed rather than silently accepted. Median is the arithmetic midpoint of the parsed low/high — a best-effort approximation, not a true percentile, since a posted range is a single interval, not a distribution. The parser leaves SourceUrl empty because it only sees the description text, not the posting URL; the caller (JobQueueService) stamps the job's own LinkedInUrl onto each PostedRange insight after the lookup, so persisted compensation data always cites the exact posting it was read from.

LevelsFyiSource (src/WorkWingman.Infrastructure/Enrichment/LevelsFyiSource.cs) is a documented no-op today, exactly like LayoffsFyiSource's treatment of layoffs.fyi in job-signals.md. It exists as a registered seam (IsConfigured reflects whether WorkWingman:LevelsFyiApiKey is set) but does not implement the actual paid-API HTTP call — that follow-up requires a live subscription to verify the real request/response schema against.

Aggregation (CompensationInsightService, src/WorkWingman.Infrastructure/Enrichment/CompensationInsightService.cs)

Queries every registered ICompensationSourceAdapter, fans the postingHtml parameter out to all of them (only PostedRangeSource uses it today), and returns the concatenated list of every insight every source returned — no merge, no dedupe. One source failing (network error, missing local extract file, malformed JSON) never blocks the others; each adapter's LookupAsync degrades to an empty list rather than throwing, and CompensationInsightService additionally catches any exception a misbehaving adapter might still raise (matching CompanyLayoffService's graceful-degradation contract exactly), re-throwing only OperationCanceledException so cancellation still propagates.

Source adapter interface:

public interface ICompensationSourceAdapter
{
    CompensationSourceKind Source { get; }
    Task<IReadOnlyList<CompensationInsight>> LookupAsync(
        string role, string location, string? postingHtml, CancellationToken ct = default);
}

Signal B — Company diversity snapshot

What it shows: for a job's company, the most recently published percent of the workforce identifying as women and/or an underrepresented racial/ethnic minority (URM), when a curated public source has published one.

Sources evaluated and the decision

Source Verdict Why
Company DEI reports Used (CompanyDiversitySource, kind CompanyDeiReport) Many large-cap employers voluntarily publish an annual workforce-demographics page or PDF. These are public, citable, company-authored documents — this source only ever stores a direct URL to the employer's own published figure.
EEO-1 Component 1 federal contractor disclosures Used, with a staleness caveat (CompanyDiversitySource, kind Eeo1FederalContractorDisclosure) Federal contractors with 50+ employees and a qualifying contract must file an EEO-1 report; in response to FOIA litigation the Department of Labor has released consolidated (company-wide, not establishment-level) EEO-1 data for non-objecting contractors — e.g. Component 1 reports for 2016–2020 released in February 2026. This data is real and legally obtained, but is frequently years stale by the time it's released, and only ever consolidated at the whole-company level, not by role or location.
A general-purpose bulk "diversity API" Does not exist — not built There is no legal bulk source for current, per-company gender/race breakdowns the way there is for salary (LCA) or wages (BLS). Most private companies publish nothing at all. The EEOC's own EEO-1 aggregate reporting requirement is, as of this writing (mid-2026), itself the subject of a proposed rule to rescind it — the pipeline that produces this data at all may shrink further, not grow.

Why this is a curated table, not a crawler. Attempting to "search the web for this company's diversity report" programmatically would be unreliable (no stable URL pattern), unattributable (hard to verify a scraped page is actually the company's own report and not a stale mirror or someone else's summary), and impossible to keep legally clean at scale (some of those pages belong to sites with their own scraping restrictions). CompanyDiversitySource is intentionally a lookup against a small, hand-curated table (CuratedEntries, a company name -> DiversitySnapshot dictionary, always re-keyed to an ordinal case-insensitive comparer regardless of how the caller built it) — every entry is added only once a human has verified the company's own published figure and recorded a direct source URL. A company with no curated entry returns null, meaning "not curated yet," never "no diversity data exists" and never a fabricated estimate.

Normalized model (DiversitySnapshot, src/WorkWingman.Core/Models/DiversitySnapshot.cs)

public class DiversitySnapshot
{
    public double? WomenPercent { get; set; } // null if not published
    public double? UrmPercent { get; set; }   // null if not published
    public DiversitySourceKind Source { get; set; } // CompanyDeiReport | Eeo1FederalContractorDisclosure
    public string SourceUrl { get; set; }
    public DateOnly AsOf { get; set; } // filing/publication year — judge staleness yourself
}

Every numeric field is independently nullable: a company might publish WomenPercent in its DEI report without breaking out a comparable URM figure, and this must be representable without inventing a number for the missing side.

Source lookup interface:

public interface ICompanyDiversitySource
{
    Task<DiversitySnapshot?> LookupAsync(string companyName, CancellationToken ct = default);
}

Wiring

Registered in Program.cs the same way every service in this codebase is — plain IServiceCollection singletons:

builder.Services.AddSingleton<ICompensationSourceAdapter, H1bLcaSource>();
builder.Services.AddSingleton<ICompensationSourceAdapter, PostedRangeSource>();
builder.Services.AddSingleton<ICompensationSourceAdapter>(
    new LevelsFyiSource(apiKey: builder.Configuration["WorkWingman:LevelsFyiApiKey"]));
// BLS is only wired in once a verified role/location -> series-ID map is loaded (see
// "Series mapping" above) — an unmapped BlsOewsSource would be a silent no-op.
var blsSeriesMap = BlsOewsSource.LoadSeriesMap(builder.Configuration["WorkWingman:BlsSeriesMapPath"]);
if (blsSeriesMap.Count > 0)
{
    builder.Services.AddSingleton<ICompensationSourceAdapter>(
        new BlsOewsSource(blsSeriesMap, builder.Configuration["WorkWingman:BlsApiKey"]));
}
builder.Services.AddSingleton<ICompensationInsightService, CompensationInsightService>();
builder.Services.AddSingleton<ICompanyDiversitySource, CompanyDiversitySource>();

JobPosting (src/WorkWingman.Core/Models/JobPosting.cs) carries both signals directly on the job model:

public List<CompensationInsight> CompensationInsights { get; set; } = [];
public DiversitySnapshot? Diversity { get; set; }

Where each field actually gets populated: inside JobQueueService.ResyncFromLinkedInAsync, for each newly-added job (by LinkedInUrl, after dedup against the existing queue) — alongside the existing LayoffHistory enrichment call:

job.CompensationInsights = (await compensation.GetInsightsAsync(
    job.Title, job.Location, job.Description, ct)).ToList();
job.Diversity = await diversity.LookupAsync(job.Company, ct);

job.Description (the plain job-description text already scraped for the existing Description field) is passed as the postingHtml argument — PostedRangeSource's HTML parser (HtmlAgilityPack.LoadHtml) accepts plain text gracefully (it becomes one text node), so no raw page HTML needs to be persisted on the job model or re-fetched with an extra page load, which would violate the same "no extra requests" discipline LinkedInPostingSignalExtractor follows. Already-known jobs are not re-queried on every resync. A lookup failure/empty result leaves CompensationInsights as an empty list and Diversity as null, never blocking the rest of the sync.

Caching, rate guidance, and refresh cadence

  • H-1B LCA extract: this is a local file read, not a live call — no per-request rate limit applies. Refresh cadence: OFLC publishes new disclosure data quarterly; the local extract should be regenerated on the same quarterly cadence via a separate out-of-band ETL step (not built in this change — the adapter only defines where it reads from, H1bLcaSource.DefaultExtractPath). Until that ETL step exists, this adapter is a registered, buildable seam that degrades gracefully to empty results (same pattern as WarnNoticeSource's state-coverage gaps in job-signals.md).
  • BLS OEWS API: the registered key raises the quota to 500 requests/day; without one, 25/day. OEWS itself is only republished annually (each May) — cache per role/location series lookups for at least 30 days; there is no value in calling more often than the underlying survey updates.
  • Posted pay range: no caching needed — it's parsed once, synchronously, from text already fetched during the existing scrape pass, same as the LinkedIn posting signal in job-signals.md.
  • levels.fyi: no calls are made (see above), so no rate guidance applies today.
  • Company diversity snapshot: a local in-memory/curated lookup, no network call, no caching needed. Re-curating an entry (updating its AsOf/figures when a company publishes a new report) is a manual, periodic maintenance task, not an automated refresh.

Tests

  • H1bLcaSourceTests.cs — real temp-file JSON extracts, no network; covers matching, percentile math, and a missing/malformed extract file.
  • BlsOewsSourceTests.cs — Flurl HttpTest stubs against the real BLS API shape; zero real network.
  • PostedRangeSourceTests.cs — fixture HTML/plain-text strings, no network at all (pure text parse).
  • LevelsFyiSourceTests.cs — pins the documented no-op behavior with and without a configured key.
  • CompensationInsightServiceTests.cs — fan-out/no-merge/graceful-degradation logic against mocked ICompensationSourceAdapters.
  • CompanyDiversitySourceTests.cs — in-memory curated-table lookups, case-insensitivity, and null-degradation.
  • JobQueueServiceTests.cs — new compensation/diversity enrichment-wiring cases added alongside the existing layoff-history ones.
  • All new source files are covered by the existing **/Enrichment/*.cs glob in stryker-config.json — no config change was needed since the new adapters live in that same folder.