Testing apps built with Cursor, Claude Code, and Lovable — the hidden bug class

If you’ve built an app with Cursor, Claude Code, Lovable, or any other “vibe coding” tool, you’ve already noticed the pattern: the AI ships features fast, the tests pass, the demo works, and then a real user hits a path the AI didn’t model and the whole thing collapses in an interesting way.

This post is about that specific bug class — what makes it different from traditional bugs, why your test suite probably doesn’t catch it, and how crowd testing fills the gap better than for hand-written code.

The bug class that AI codes consistently produce

After looking at enough AI-built apps in beta, a pattern emerges. The bugs cluster around trust boundaries — the places in the code where one component hands data or control to another:

• Auth tokens that look valid but reference a session that was rotated 5 minutes ago
• Validators that reject null but accept undefined (or vice versa) because the AI guessed which one mattered
• Error handlers that catch Exception when they should catch a more specific type — silently swallowing real bugs
• Navigation that returns to the right screen but loses the state that should have travelled with it
• API contracts that the AI wrote based on what the docs probably say, not what the API actually returns

These aren’t syntax errors. The code compiles, runs, and looks correct under review. The bugs only surface when real conditions intersect with the AI’s incomplete mental model. And here’s the kicker: the AI’s tests don’t catch them either, because the AI wrote both the code and the tests from the same flawed mental model.

Why your test suite gives you false confidence

If you let Cursor or Claude Code generate tests alongside the code, you almost certainly have ≥80% code coverage. Your CI is green. You feel safe.

You shouldn’t.

AI-generated tests cover the happy path because the AI understood the happy path well enough to write the code. They cover obvious failure modes because the AI thought of them. But the bugs that ship to production come from the failure modes the AI didn’t think of — and by definition, those are the ones the AI didn’t write tests for.

This isn’t a flaw in the model. It’s a structural property. The same brain that wrote the code wrote the tests. The blind spots are correlated.

Three things that genuinely help

1. Cross-model review

Have a different AI review the code. Not the same model. Different model.

Claude reviewing Cursor’s output catches things Cursor missed because the models have different training data, different priors, different patterns they’re confident about. A 10-minute cross-model pass over a fresh feature catches ~30% of issues that would have shipped otherwise.

Same-model review (Claude reviewing Claude’s output) catches almost nothing. The blind spots overlap perfectly.

2. Adversarial test prompts

Ask the AI to write tests adversarially. Specifically:

“Write tests that would catch a junior developer who got the auth logic almost right. Focus on the cases where it looks correct but isn’t.”

This shifts the AI from “test the happy path” to “imagine a wrong implementation and write tests that would expose it”. The tests it writes under this prompt catch a meaningfully different bug class than the default-prompt tests.

3. Human testers on real devices

This is the one that can’t be automated away. Real human testers on real devices intersect your code with conditions the AI couldn’t model:

• Network drops mid-request
• Background app suspension at the wrong moment
• Permissions denied after first grant
• iOS keyboard taking up 60% of the screen on an iPhone SE
• A user with VoiceOver enabled
• A user whose locale is de-DE and decimal separators are commas

AI doesn’t know about your tester’s specific Pixel 6a with 2GB free RAM, and it can’t simulate the timing of a notification arriving exactly during a screen transition. Crowd testing on real hardware is the only reliable way to surface these.

What to crowd-test specifically in an AI-built app

Generic crowd-testing tasks (“try the app”) return generic findings (“nice UI”). Your test tasks need to specifically cross the seams the AI is bad at. A useful tasking pattern for vibe-coded apps:

These are the tasks that surface the AI’s blind spots. They’re also tasks no AI test suite is going to generate on its own, because the AI doesn’t know these are interesting failure modes for the app it just wrote.

The honest pricing math for AI-built indie apps

A typical AI-coded indie app from a single dev:

• 30–50 features
• Built in 2-6 weeks
• 0 dedicated QA budget

For testing, your options:

The crowd-testing line item is the one that genuinely buys you something the AI can’t deliver. €100 spent on 10 real testers on real devices catches the bugs that would otherwise hit your first 100 real users.

When AI built the app, who should crowd-test it?

The right tester pool for an AI-built indie app:

• Non-technical users (testers who don’t know what TestFlight is) catch the UX bugs the AI wrote past
• Diverse devices — the AI’s “it works on my machine” is your test simulator. Real devices spread across iOS + Android + various sizes/ages catch the bugs that hide on the dev’s M4 Mac
• Real-world tasks — not “open the app and tap around”, but specific tasks your real users would do

TesterPayKit is built specifically for this cohort. If you’ve shipped with Cursor, Claude Code, or Lovable and need real-tester signal in 48 hours, this is the path of least friction. €0 to start, pay-per-tester.

Part of TesterPayKit’s crowd-testing series. The pillar: What is crowd testing?. Adjacent: How to find beta testers for your Flutter app.