Skip to content

Could WorkWingman use an AI browser-agent instead of Playwright? (plain version)

Engineer version with all the detail: ../technical/pageagent-feasibility.md

Someone pointed us at a project called Alibaba PageAgent and asked: instead of hand-coding "click this exact button" instructions for every job site, what if we just told an AI "fill in the Education section" and let it figure out the clicking? We spent a day digging into it. Short version: promising for the hard cases, not a replacement for what already works.

What PageAgent actually is

Today, WorkWingman fills out job applications the way a very precise robot would: it's told exactly where the email box is on a Workday page (a specific hidden label in the page's code), and it clicks exactly there. That's fast and 100% predictable, but someone has to write that exact instruction for every single company's application system, and it breaks the moment that company redesigns their site.

PageAgent flips this around. You give it a plain sentence — "fill the Education section with B.S. Game Development" — and it reads the page like a person would (by its visible text and labels, not screenshots), decides what to click or type, and does it. No screenshots, no per-site instructions. It's a small AI "brain" that lives inside the webpage itself.

Can it use Claude?

Not automatically, but yes with some work. PageAgent was built assuming you'd hook it up to a service like Alibaba's own Qwen AI. Wiring it up to Claude instead means writing a translator — a smallish, well-defined piece of code that speaks PageAgent's expected language on one side and Claude's actual language on the other. It's the same kind of "one obvious swap" WorkWingman already has planned for other AI features (there's a placeholder for a real Claude document-writer already in the code). Not free, but not scary either.

Where would it actually live?

This is the trickiest part. WorkWingman's real automation today runs in the invisible "engine room" (the .NET backend) — it's like a mechanic working under the hood while you watch through a window. PageAgent needs to run inside the browser page itself, more like a passenger reaching over and clicking things directly. Those are different rooms. To connect them, WorkWingman would need a new "intercom" between the engine room and the browser window — something that doesn't exist yet. The plumbing for a visible automation browser window is already sketched out as a future project; PageAgent would plug into that plumbing, not replace the whole engine.

Some parts wouldn't change at all: the part that remembers your logins, the part that keeps a real browser profile so you don't have to sign in every time, and the tests that check the app itself works — none of that is about clicking buttons on a job site, so none of it needs replacing.

Does "one sentence" really replace all that hand-coded per-site work?

Partly. The appeal is real: telling an AI "fill the Education section" doesn't care whether the company's website was built with one set of hidden labels or another — a human-shaped instruction generalizes in a way a hard-coded click target never can. That's genuinely valuable for the job sites WorkWingman hasn't automated yet.

But it's not a magic fix for everything. Some job sites bury their actual form inside a "frame within the page" (like a picture-in-picture window) that outside code — including PageAgent — simply isn't allowed to see into, for the same privacy/security reasons your browser doesn't let random websites read each other's tabs. Playwright's specialized tools can reach into those frames; an in-page AI agent generally can't. So the honest picture is: fewer hand-coded shortcuts, not zero.

The catches

  • It costs money and time per click. Every single field-fill is a call to an AI service. A full job application, click by click, could easily add up to 45–80 of these calls — call it a few minutes and somewhere between pocket change and a dollar, per application, just in AI usage. Today's approach costs nothing extra to run once it's built.
  • It's not perfectly predictable. The exact same webpage can, in principle, get a slightly different response from an AI on two different days. WorkWingman's testing philosophy is "flaky is a bug" — that's much harder to guarantee when an AI is making the call each time instead of a fixed rule.
  • The "never submit without you" promise doesn't come for free. WorkWingman's most important promise is that it will never hit the final Submit button without your OK. Today that's true because the code simply never tells the robot arm to press that button. An AI agent, left to its own judgement, could decide to click Submit like it clicks anything else — so that guarantee has to be deliberately rebuilt as a hard rule around the AI, not assumed to come built in.
  • Testing offline gets harder. Right now, tests can check "does this exact instruction find the right box on a saved copy of the page" — no live internet, no waiting, no cost. Testing an AI-driven version the same way means either paying for real AI calls in every test run (slow, costly, and can vary run to run) or faking the AI's answers (which only proves the wiring works, not that the AI actually reads the page correctly).

Bottom line

Don't rip out the current approach — there isn't even a finished version of it to rip out yet for job sites beyond Workday. The sensible plan is: finish the reliable, hand-coded path for the sites already targeted, and bring in the AI agent as a backup player — used when a hand-coded shortcut doesn't exist yet (a brand-new job site) or breaks unexpectedly (a site redesign), always double-checked by a person before anything gets submitted. Prove it out small, watch the real cost and accuracy, and only lean on it more once the data says it's earned that trust.