Why some “green” tests still shipped broken features¶
Technical version: Producer contract tests (WING-324)
The short story¶
Several times the app’s automated tests were all green while a real screen was completely broken for users. The shared root cause was simple and ugly:
The server sent one shape of data. The app expected a different shape. The test was written from the app’s wrong expectation, so it congratulated itself.
Examples that actually shipped:
| What broke | What went wrong |
|---|---|
| Demo persona picker | App looked for a nested feature flag; server sent a simple true/false at the top level. Gate failed closed and hid the feature for everyone. |
| Persona cards | Server field was personaKey; app read key. Everything looked empty or “DEFAULT.” |
| Saved / recommended job lists | Server added a new field; the app’s strict allow-list never learned it and rejected every real page of results. |
The rule we follow now¶
For any data that crosses from the real server to the browser UI:
- Capture a real response from a running server (sandbox data only — never a user’s private job list). Save it as a fixture with when and how it was taken.
- Run the real app code against that exact payload and prove it accepts it.
- Compare keys — if the server adds or renames a field, the test fails until the app is updated.
- Pin the old wrong shape as absent so the previous bug cannot quietly come back green.
Hand-built JSON that encodes the same wrong guess as the code is not a test. It is the bug wearing a lab coat.
What this protects you from as a user¶
Silent “feature missing” failures, empty confirm dialogs, and whole job lists rejected as invalid — the kinds of problems that never show up as a crash, only as “why is this screen dead?”