The naive agent test

I've been building a new API for a web application with Claude Code, and once it was usable I wanted it tried by somebody who hadn't been in the room while it was written. Which was me and the agent I used.

So I opened a fresh Claude Code session, gave it just a short note on where to find a token and where the API doc lived - and a real job that needed a handful of records creating and updating through it.

The idea is borrowed from usability testing. Steve Krug's Don't Make Me Think has a whole section on testing on the cheap: grab a few people who have never seen your site, give them a real task, and watch where they stumble, and the reason it works is that they don't know what you know. A fresh agent session is about as naive as a user gets: no memory of the build, and nothing to go on but what you hand it.

What the agent stumbled on

The job got done, and it did pretty well. Afterwards I asked the agent whether anything had tripped it up that should have been in the instructions, and the feedback was good: nothing in the docs was wrong, but there were a few absences. The create endpoint had a one-line description and no request body, so it had worked out what to send from the shape of a read response and guessed at which fields were required, and nowhere did the docs say whether a money field was in pounds or pence, so it had inferred that from a figure already in the system. It got every one of those guesses right.

Then I told it to be stupid

That made me wonder how naive it had really been, so I asked it. It admitted that it had arrived with everything its model already knows about how APIs usually work, and had used that to infer as much as it could from what I'd given it. I realised it had been trying to get everything right, so the API had only ever seen well-formed requests, and the validation and error handling hadn't been exercised. So I went back and told it to be stupid, or rather to try deliberately bad stuff at random: send malformed payloads, guess at field names, put the wrong types in, and see what the API does with it. That turned out to be really useful, generating another list of improvements.

Two agent sessions in a loop

This was only the first pass at the API, and what has made it useful is running it as a loop between two sessions with nothing shared: one building, one arriving cold and trying to use what the first one built. The builder can't test its own work for the same reason I can't - it was in the room, and it fills every gap in the docs from memory without noticing there was a gap. My job is carrying the tester's feedback back to the builder and deciding what to act on, and each round has changed the API: the contract is now served by the API itself, and the request schema is generated from the validators the code already uses. Then the next fresh session gets to try again. Looking back, I realised the two sessions do share one thing, the model itself, so wherever the builder had followed convention the tester was always going to guess right, which is probably why every guess in the first pass was good.

It's an interesting pattern to have in the agentic development toolbox, and one I expect will come in handy again. I'll most likely write it up as a Claude Code skill, so that it becomes a playbook I can run each time and improve as I learn what it misses.