Skip to content

Why the Robots Kept Failing — In Plain English

(Plain-language twin of ci-self-hosted-runner.md. Same story, no jargon.)

Every time someone changes WorkWingman's code, a set of automated checks runs: build it, run the tests, scan for security holes, make sure the docs still line up. If any check fails, the change doesn't land.

For about four days, those checks failed constantly. Not one of the failures was caused by bad code. The code was fine the whole time. Every single failure was the machine that runs the checks being set up wrong.

This is the story of why, because the mistake is a good one to understand — and because it kept happening in the same shape, four times in a row.

The one idea behind every failure

The checks used to run on computers Microsoft rents out. Now some of them run on a PC in this house — cheaper, faster, and it does what it's told.

But a program running on a PC doesn't run as a person. It runs as an account, and the account here is a stripped-down system one called NETWORK SERVICE. It's not Andrew. It doesn't have his desktop, his settings, or his stuff. Deliberately: something that runs unattended and executes code from the internet should own as little as possible.

That's the whole story. Nearly every failure was something installed for Andrew, and therefore invisible to the account actually doing the work.

It is a genuinely nasty class of bug, because the natural way to check makes it disappear. You log in, you type the command, it works perfectly, and you conclude the machine is fine. It isn't. You just asked the wrong account.

Failure one: nowhere to put the toolchain

The first step needs the .NET toolchain. The installer's default home is inside Program Files — a protected folder that only administrators may write to.

NETWORK SERVICE is not an administrator. So the very first step failed, and because it failed, every later step was skipped. Nothing was ever built, tested, or scanned.

Fix: stop trying to write to the protected folder. The tool now installs into scratch space the account already owns. Nobody's permissions were widened — the job just stopped asking for something it never needed.

Failure two: a locked door for scripts

With the toolchain in place, the next step failed.

Windows can refuse to run script files at all — a sensible protection against double-clicking something nasty from an email. On this PC that protection was at its strictest: run no scripts, ever.

The problem is that the check system works by writing a little script for each step and running it. So the protection was, very correctly and very unhelpfully, blocking the build itself.

Fix: allow scripts written on this machine, while still refusing unsigned scripts downloaded from the internet — the standard setting for a build machine, and not "allow everything".

The subtle part: this had to be set for the whole machine, not for Andrew's account. Setting it for his account would have looked right, felt right, tested fine when he tried it, and changed nothing at all for the account that actually runs the checks.

Failure three: the program that existed and didn't

Next step, next failure:

pwsh : The term 'pwsh' is not recognized...

Except pwsh — a newer PowerShell — was installed. Andrew could type pwsh and it ran.

It was installed as an app inside his user profile. NETWORK SERVICE has no profile, no access to his folders, and no way to see it. The program existed and did not exist, depending on who asked.

Fix: install it properly, once, for the whole machine.

The tempting shortcut — "just use the older PowerShell that's already there" — doesn't work, and it's worth knowing why. The script being run loads a component built for modern .NET. Old PowerShell runs on an older foundation that physically cannot load it. That shortcut trades a clear error message for a confusing one.

What it looks like now

The build step, which had been dying instantly for days:

Install .NET            OK
Build solution          OK
Security scan of deps   OK
Install test browser    OK
Run the tests           3,635 passed, 1 failed

3,635 of 3,636. The code was always fine. It just never got the chance to prove it.

The two things still broken

The one failing test needs an AI model that's running on that PC — and can't reach it. Same bug, fourth time: the model is running for Andrew, not for the account doing the work. It's a bit of an odd thing to have in the standard checks at all, since it means the checks go red whenever that model happens to be switched off. That's an open decision.

Three security scanners still can't run, for a completely different reason: they run on Microsoft's rented computers, and the bill isn't paid. Nothing in the code can fix that. Only Andrew can, and until he does, those three stay red no matter how good the code is.

The lesson, if you only keep one thing

When something works for you but fails for the machine, don't check whether it works. Check who it works for.

Four failures, one cause. Every fix was "install it for everyone, not just for you." And the checks that verified the fixes had to run as the stripped-down account too — because a test that passes as the wrong user is just a more confident way of being wrong.

Learn more