Making LLM Code Review Actually Catch Things

The Illusion of One‑Pass LLM Reviews

One‑pass LLM reviews treat a single model output as a complete audit of a code change. The approach is attractive because it is fast and cheap. It assumes that the model can understand the surrounding context, the intended functionality, and the subtle edge cases that a human would spot. In practice the model often confuses stylistic preferences with genuine defects, and it can overlook logical errors that require deeper reasoning about state flow or external contracts. When the prompt is terse, the model may default to generic suggestions such as renaming a variable or adding a comment, which do not address security or performance problems.

LLM‑based code review catches around 30% of bugs in typical PRs (LLVM Discussion Forums). This figure comes from analysis of ~30 bugs found across 207 pull requests.

The remaining 70% of defects often involve subtle misuse of APIs, race conditions, or off‑by‑one errors that require multi‑step reasoning. A single model pass can surface syntax problems but rarely uncovers deeper architectural violations. Consequently teams that rely solely on a one‑pass review may ship code with hidden regressions. Introducing a second LLM as a reviewer creates a form of cross‑validation. By feeding the first model’s output back into a different model with a complementary prompt, teams can surface defects that the first model missed. This pattern mirrors human code review pairings and has been shown to improve detection rates when the second model is asked to focus on correctness rather than style.

Why Single‑Model, Single‑Prompt Checks Miss Real Bugs

Relying on a single model and a single prompt to identify bugs in code review processes often leads to incomplete results. Large language models (LLMs) are powerful but inherently limited by their understanding scope and prompt design. When a review is based solely on one prompt, the model may overlook subtle or complex issues that require deeper context or multiple perspectives. This creates a false sense of security, as the review appears thorough but misses critical defects.

One core problem is that simple prompts like “Please generate a code review for the following code” do not effectively guide the model to surface all relevant issues. As Shweta Ramesh et al. note, generating meaningful reviews requires careful consideration of data selection, prompt design, and context extraction. Without these, the model’s output tends to be superficial, focusing on obvious issues and ignoring more nuanced bugs. This is especially problematic in complex codebases where bugs can be subtle and context-dependent.

< div class=“stat-box” >
AI code review tools catch about half of bugs, according to a recent O’Reilly Radar article. This statistic highlights the limitations of single-pass, single-model checks, emphasizing that many bugs remain undetected in typical review setups.
< /div >

Furthermore, models trained on generic prompts often lack the specificity needed to identify certain classes of bugs. They may flag style issues or minor code smells but miss deeper logic errors or security vulnerabilities. The problem is compounded by the fact that models tend to generate responses based on patterns learned during training, which may not align with the unique context of a particular codebase.

The core issue is that a single prompt and model cannot fully emulate the multi-faceted nature of human review. Human reviewers consider multiple angles, ask questions, and revisit code sections. A single prompt model cannot replicate this iterative, multi-perspective process. As a result, relying solely on one prompt and one model leads to incomplete bug detection, risking overlooked defects that could cause failures in production.

< div class=“citation” >
“We observed that simple prompts such as “Please generate a code review for the following code” do not work well in practice; generating meaningful reviews requires careful consideration of data selection, prompt design, and context extraction.” , Shweta Ramesh et al., “Automated Code Review Using Large Language Models at Ericsson: An Experience Report,” arXiv:2507.19115v2 (2025) source
< /div >

Signal vs. Noise: Measuring Review Effectiveness

When an LLM reviews code, the first question is whether the feedback it produces is useful or merely noise. Engineers need a way to quantify signal versus noise so that they can trust the tool and allocate human effort efficiently. The metric is simple: the proportion of comments that correspond to actual defects versus those that are false positives.

A recent survey of production pipelines shows that AI code review tools catch between 50% and 80% of bugs. This range reflects the variance in model quality, prompt design, and the complexity of the codebase. However, the raw detection rate does not reveal how many of the generated comments are actionable. To separate signal from noise, teams track the ratio of bug‑related comments to total comments per pull request.

Signal measurement begins with a baseline: the number of bugs that would surface through a traditional manual review. By comparing the LLM’s bug count to this baseline, teams can compute a precision metric. Precision is the fraction of LLM comments that actually correspond to a defect. A precision below 30% indicates that the majority of feedback is noise and that the model or prompt needs adjustment.

In practice, noise manifests as repetitive style warnings, redundant variable renames, or comments that do not affect program correctness. A typical pull request may generate 10–20 comments, but only a fraction address real issues. Most AI code review tools generate 10–20 comments per pull request. The problem? 80% of those comments are noise.

By continuously monitoring precision and recall, teams can iterate on prompt templates, model selection, and post‑processing filters. When the signal‑to‑noise ratio exceeds a threshold, typically 70% precision, engineers can safely reduce manual triage. Otherwise, the review process should incorporate additional checks or a second LLM opinion to surface hidden defects.

Leveraging a Second LLM for a True Second Opinion

Relying on a single model for code review creates a single point of failure. Even the most capable models hallucinate or miss edge cases due to specific training biases or context window limitations. Introducing a second LLM breaks this echo chamber by providing an independent perspective. The underlying mechanism is straightforward but powerful. The first model performs a standard review, generating a list of potential issues. The second model receives the original diff plus the first model’s output. Its job is not just to review the code again, but to critique the critique. It must identify false positives, validate true positives, and surface issues the first model missed. This creates a feedback loop where the second model acts as a judge, weighing the evidence provided by the first against the source code.

# Pseudocode flow
diff = get_git_diff()
review_a = model_a.generate(f"Review this diff for logic errors: {diff}")
prompt_b = f"Diff: {diff}\nReview A: {review_a}\nCritique Review A. Identify false positives and missed bugs."
review_b = model_b.generate(prompt_b)

This pattern, often called ‘adversarial’ or ‘consensus’ review, forces the system to justify its findings. Failure modes include increased latency and compute cost. In practice, this means the review pipeline takes twice as long and costs twice as much. If both models share similar training data or architecture, they might agree on a hallucination. However, using different providers or model families mitigates this risk effectively. Compared to simply re-running the same prompt with higher temperature, a distinct model offers a genuinely different reasoning path. A single model with varied temperature often produces stylistic variations rather than distinct logical insights. A second model acts as a distinct cognitive agent. Use this tactic for security-sensitive changes or complex refactors where the cost of a missed bug outweighs the inference expense. It transforms the review from a passive check into an active debate, significantly raising the bar for code quality.

Prompt Engineering Patterns That Surface Actual Defects

General prompts like “review this code” are the primary cause of noisy, low-value feedback. They invite the model to optimize for surface-level issues such as variable naming or docstring formatting rather than substantive logic errors. To shift the model’s focus from style to correctness, you must constrain the scope of the inquiry and force a specific reasoning process.

The underlying mechanism relies on reducing the solution space. By assigning a specific persona or a narrow verification task, you limit the model’s tendency to hallucinate style critiques. For example, instead of asking for a general review, instruct the model to act as a security auditor focused solely on input sanitization. This forces the model to apply a stricter, more relevant set of heuristics to the code.

A concrete pattern involves asking for a trace of data flow. Consider the following prompt structure:

Review the attached code for SQL injection vulnerabilities.
1. Identify all user input sources.
2. Trace how these inputs flow into database queries.
3. Verify if each query uses parameterized binding.
4. Report only the queries that lack binding.

This structure eliminates the noise of style suggestions and directs the model’s attention to the execution path where the defect exists.

The failure mode of this approach is tunnel vision. If you ask the model to look only for SQL injection, it will ignore a concurrent race condition in the same function. Conversely, if the prompt is too vague, the model defaults to safe, low-value observations about code structure.

Compared to a general “catch-all” review, targeted prompting yields higher precision but lower recall for unrelated issues. It trades breadth for depth. Use this tactic when you know the specific risk profile of the code you are reviewing, such as handling untrusted input or managing complex state transitions. It is most effective in CI pipelines where you can chain multiple specialized prompts together to cover different defect categories.

Balancing Style Policing with Substantive Bug Detection

When an LLM flags every indentation error, the review can feel like a grammar check. Engineers quickly learn to scroll past style warnings and focus on the handful of substantive issues that the model surfaces. This shift can be dangerous: a cleanly formatted but logically incorrect block of code may slip through while a trivial missing semicolon gets highlighted repeatedly. To prevent such imbalance, reviewers should partition the review process into two passes. First, a lightweight style pass can use a dedicated static‑analysis engine or a fine‑tuned LLM that emits only formatting concerns. The output of this pass should be automatically applied as a code‑formatting patch, leaving the main review free of noise.

Second, a deeper pass should engage the LLM with a prompt explicitly requesting defect‑level analysis. The prompt can instruct the model to “explain why the logic may be incorrect, suggest edge‑case tests, and propose fixes.” By separating concerns, the model can maintain a higher signal‑to‑noise ratio, because it does not have to juggle style and logic simultaneously. Practically, this can be implemented as a two‑step GitHub Actions pipeline: the first step runs ruff or prettier to enforce style; the second step triggers the LLM review on the resulting diff. Engineers can also embed a short “style‑only” token in the prompt to tell the LLM to ignore formatting rules, thereby freeing cognitive bandwidth for bug hunting.

Another technique is to use a “style‑policing flag” that counts formatting violations but does not block the review. The LLM can return a separate tally of style issues, which the CI can surface as a badge. This visual cue reminds developers that the code meets the style guide while still allowing the

Building a Repeatable Multi‑Model Review Workflow

The final stage of a robust code review pipeline involves orchestrating multiple models to verify findings. Relying on a single model often leads to confirmation bias where the reviewer adopts the same logical blind spots as the original code generator. By implementing a multi-model workflow, you force the system to reconcile conflicting perspectives, which significantly increases the probability of catching subtle logic errors or security vulnerabilities.

A repeatable workflow requires a structured handoff between models. The first model acts as the primary reviewer, generating a list of potential issues. The second model, often a more capable or differently tuned variant, receives these findings along with the original source code. Its task is not to generate new feedback but to validate the existing findings. This verification step acts as a filter for false positives. If the second model cannot confirm the defect, the finding is flagged for human intervention or discarded.

To implement this, define a clear schema for the review output. Use JSON to ensure that both models operate on the same data structure. This allows you to programmatically compare the outputs of different models. For instance, you might use a fast, cost-effective model for initial triage and a high-reasoning model for final verification.

def verify_review(original_code, findings):
    prompt = f"Validate these findings: {findings}. Code: {original_code}"
    response = client.messages.create(
        model="claude-3-5-sonnet-20240620",
        messages=[{"role": "user", "content": prompt}]
    )
    return parse_validation(response)

Failure modes in this workflow typically stem from prompt drift or inconsistent output formats. If the first model changes its reporting style, the second model may fail to parse the input correctly. You must enforce strict schema validation at every transition point. When the models disagree, the system should default to a conservative state, such as marking the code as requiring manual review.

Choose this tactic when your codebase complexity exceeds the reasoning capacity of a single model pass. It is particularly effective for security-sensitive code where the cost of a missed bug outweighs the latency of an additional model call. By decoupling the detection phase from the verification phase, you create a system that is both scalable and resilient to the inherent unpredictability of individual LLM outputs.