Case study · OnTheClock · 2026
An AI wrote 39 locators. None of them existed.
Appium, .NET MAUI, and real iOS and Android devices on BrowserStack. On paper, mobile test automation is a solved tooling problem. Then you write the first test and find out the app has no idea you exist.
- Role
- Built the mobile suite from scratch
- Stack
- Appium, Pytest, BrowserStack, .NET MAUI, XAML
- Platforms
- iOS and Android, on real devices
The problem
Every mobile automation framework finds elements the same way: it asks the app what things are called. Appium reads the accessibility tree, and whatever the app publishes there is your entire vocabulary. A button with no automation ID might as well not be on the screen.
Ask a model to write a test for a screen and it hands you plausible identifiers, exactly what a sensible developer would have named them. The app had named them nothing. Of the locators AI had written for me, 39 had never existed.
So the tests did not fail. They skipped, with reasons like "not visible for this configuration" that sent people hunting through account settings for features nobody had built. A few were worse: they passed. One read the clock state off a set of labels, then asserted those same labels were present. It could only ever confirm itself.
I had filed three bugs off the back of those tests. All three were my locators, not the app. I spent a week disproving my own bug reports, and I made every one of those tests capable of failing.
What I did
A contract, not better prompting
I wrote down what the app has to expose for automation to be possible at all, and the app's development partner shipped automation IDs against it. That turns test generation from guessing into reading. The model stops inventing identifiers when there is a real list to work from.
Probe for the contract; do not trust it
I added a page-source probe that runs on a real device, dumps the accessibility tree, and asserts that the new IDs are genuinely there. On the first build that claimed to ship a new set, it found none of them, on both platforms.
The cause: an automation ID bound in XAML inside a DataTemplate does not reach the platform view. The code looked right and the review was right, but the IDs never arrived on the device. Assigning them in code-behind works.
Results
- 39phantom locators found and replaced
- 3bug reports disproved before anyone chased them
- 0tests written against IDs the device never received
What I would tell another team
AI has made writing mobile tests almost free. It has done nothing about whether your app is testable, and it will confidently paper over the difference. Publish the contract, then assert on it on a real device in CI.