AI Code Review in Production: What Actually Breaks
- 6 - 8 min read
Every AI code review vendor’s homepage shows the same demo: a clean PR, a couple of well-reasoned comments, a green checkmark. That’s not what a year of running one in production actually looks like. I built CodeBot, a dual-model reviewer (Gemini + Claude) that reads every PR on a real repo, and here’s what it actually caught, missed, and got wrong.
What It’s Actually Good At
Catching the boring stuff nobody re-reads. Off-by-one errors, unhandled null cases, a forgotten await, an inconsistent naming convention against the rest of the file. This is where AI review earns its keep: the failure mode of human review is fatigue on exactly this kind of thing, and the model doesn’t get tired on PR 40 of the week.
Learning house style over time. After enough PRs, it stops flagging patterns the team has explicitly decided are fine and starts flagging the ones that actually violate a convention the team cares about. This took real tuning, not zero-shot — more on that below.
Line-level comments that don’t require a human to triage first. A dual-model setup (one model drafts, the other checks the draft before it posts) cut down on the single biggest failure mode of early AI review tools: confidently wrong comments that erode trust fast enough that engineers start ignoring the bot entirely.
Where It Actually Breaks
Cross-file logic it can’t see in one PR’s diff. If a function’s contract changes in file A and file B (in a different, unrelated PR merged an hour earlier) relies on the old contract, the bot has no way to know. It reviews the diff in front of it, not the live state of the whole system. This is the single largest category of bugs it misses.
Context that lives outside the code. A PR that looks correct in isolation but violates a decision made in a meeting, a Slack thread, or a design doc — the model has none of that. It’s reviewing syntax and local logic, not intent that was never written down anywhere it can read.
Confidently wrong on unfamiliar frameworks. Early on, it flagged valid patterns in a framework it hadn’t seen much of in training as errors, with full confidence. The dual-model check catches some of this, not all of it. This is the failure mode that actually costs trust: a wrong comment stated as fact is worse than no comment.
Rubber-stamping repetitive PRs. On a run of near-identical PRs (a batch of similar config changes, say), review quality drops. The model pattern-matches to “looks like the others” faster than it re-checks each one on its own merits. This is the automated-review equivalent of human reviewer fatigue, and it’s worth watching for specifically.
What Changed the Outcome
Two things mattered more than model choice. First, the dual-model check — one model proposing, a second one gating what actually gets posted — cut wrong comments enough that engineers stopped dismissing the bot by default. Second, letting it learn from explicit accept/reject feedback on its own comments, not just from the codebase, is what got it to stop re-flagging patterns the team had already decided were fine.
Neither of those shows up in a vendor demo, because neither of them is visible on PR one. They only show up after months of real use, which is exactly the data a demo can’t have.
The Actual Verdict
AI code review is worth running if you treat it as a second reviewer that never gets tired on the boring stuff, not as a replacement for the reviewer who knows why a contract changed last week. The gap between those two claims is where every vendor pitch gets vague, and where the actual failure modes above live.
Related Reading
Contact me if you’re evaluating AI code review for your team and want the unfiltered version of what it does and doesn’t catch.