Case study · OnTheClock · 2026
Tests that have to prove they can fail
I joined a workforce-management SaaS as its only quality engineer, in the middle of a platform migration where the software calculates customer payroll. There was no test library. There was one of me.
- Role
- Sole quality engineer
- Stack
- Playwright, Pytest, Appium, Claude Code, MCP, Zephyr Scale, GitHub Actions
- Platforms
- Web, iOS, Android
The problem
A migration like this needs a documented test for every behavior the old system had, because "it looks the same" is not evidence that payroll comes out the same. Writing that library by hand would take a team months. The release schedule did not have months.
What I did
Authoring with AI, under a rule
I drafted 67% of the library with Claude Code. The model does not get bored hundreds of cases in, so the cases it drafts are often more thorough than rushed ones written by hand. It will also happily produce tests that cannot fail.
That is the failure mode I designed against. A test that derives its expected value from the same place it reads the actual value will be green forever and check nothing. So every generated test has to prove it can fail before it counts: break the thing it guards, and the test must go red. A suite nobody believes is a suite nobody acts on, and AI will generate a thousand of those if you let it.
A test-impact agent
I built an agent on Claude and the Model Context Protocol that reads a pull request's changes and maps them to the test cases covering the affected features. It finds test accounts whose data already satisfies each case's preconditions, runs the automated subset in the cloud, and files the result as a test cycle. "Which tests should we run for this change?" went from a half-day of judgment to one command.
One source of truth
Defects and tests were tracked in four systems. I consolidated them to two, Jira and Zephyr Scale, retired a tool, wrote the adoption guide and trained the manual testers. I also built three reports that rebuild themselves from Jira, Zephyr and GitHub, so nobody has to type a percentage that can go stale.
Results
- 1,081test cases built from zero, across 24 functional areas
- 279cases automated on web, iOS and Android
- 70defects caught by automation, 92% of all automation-found defects company-wide
- 1false alarm among them
- 4,232executions across 69 release cycles
- 372pull requests merged across 6 repositories
What I would tell another team
Generating tests is not the same as being able to trust them. The one-false-alarm figure is not a by-product of this work. It is what makes the rest of it usable, because engineers act on a red build only when a red build has been right every time before.