In our first post we covered how Doistbot went from a scrappy GitHub Action to a serverless app, reviewing about 4,500 pull requests a month for less than $2 each. We kept the details around the review pipeline itself short there. This post digs deeper into it.

After starting with Gemini CLI running in GitHub Actions, Doistbot’s reviews went through two more major architectures: a council of frontier models voting on each other’s findings, and multi-model multi-focus reviews running in parallel.

We’ll also cover the evals we built to measure whether each change actually made reviews better.

Council of experts

Until early 2026, every review came from a single model. It worked, but the rate of hallucination was still quite high. Doistbot would confidently flag bugs that weren’t there, or comment on code the PR never touched. Every false flag chipped away at people’s trust in Doistbot.

So instead of trusting one model, we ran three models against the same diff and had them check each other’s work. Each provider ran in its own harness (Claude Code, Codex, and Gemini CLI), and every harness implemented the same interface:

type ProviderName = 'gemini' | 'openai' | 'anthropic'

type ReviewProviderConfig = {
    apiKey: string
    modelName?: string
    timeout?: number
    jsonSchema?: Record<string, unknown>
}

interface ReviewProvider {
    readonly name: string

    review(
        prompt: string,
        config: ReviewProviderConfig,
    ): Promise<ModelReviewResult>
}

Then came the voting phase. We grouped matching findings from different models, and the models that hadn’t raised a finding voted on it, with the surrounding code for context. A finding needed two-thirds of their votes to make it onto the PR.

const CONSENSUS_THRESHOLD = 0.67 // 2/3 approval required

async function calculateConsensusForGroup(
    group: FindingGroup,
    availableModels: ProviderName[],
): Promise<ConsensusResult> {
    const finding = group.primaryFinding

    // Every model raised it, so there's no one left to vote
    if (availableModels.every((m) => group.sourceModels.includes(m))) {
        return { findingId: finding.id, finding, approvalRatio: 1, isApproved: true, votes: [] }
    }

    // Models that raised the finding don't get a vote
    const voters = availableModels.filter((m) => !group.sourceModels.includes(m))
    const votes = await Promise.all(voters.map((voter) => getVote(voter, finding)))

    const approvals = votes.filter((v) => v.approve).length
    const approvalRatio = votes.length > 0 ? approvals / votes.length : 0

    return {
        findingId: finding.id,
        finding,
        approvalRatio,
        isApproved: approvalRatio >= CONSENSUS_THRESHOLD,
        votes,
    }
}

The vote worked because the models rarely hallucinated the same thing. A finding one model made up usually failed to convince the other two, while real issues got through. The approved findings were then summarized and posted as one review.

Together, the pipeline looked like this:

A pull request event reaches the webhook service, which starts a disposable Fargate sandbox. The sandbox prepares the review, then Claude Code, Codex, and Gemini CLI review the PR independently. Each model votes on the others' findings, only findings approved by at least two models are kept, and the approved findings are summarized and posted as one GitHub review.

The council was a big step up in review quality, but it meant keeping three harnesses running, each with its own quirks, limits and release cycles. Since we worked on Doistbot part-time while more and more of the company relied on it, every breaking change in one of them meant dropping what we were doing to fix it.

Migrating to Pi

By then, open harnesses were becoming popular. After a promising review from our CTO, Gonçalo, we took a closer look at Pi , an up-and-coming, minimal harness.

Pi was a breath of fresh air. It got a lot of decisions right, was extremely extensible, and had a small footprint, letting us customize as much or as little as we wanted. For what we needed, Pi fit like a glove.

Once we moved Doistbot to Pi, it was immediately obvious we’d made the right call. Every model now ran through the same harness, so there was one integration to maintain instead of three.

After the migration to Pi, a long-running ECS webhook service filters events and starts a disposable Fargate sandbox. The sandbox prepares the review, then runs every model through one shared Pi coding-agent harness: Gemini 3.1 Pro Preview and GPT-5.4 in parallel when the council is enabled, otherwise the first available model. Findings are validated, grouped, voted on, summarized, and posted.

Around the same time, we dropped Anthropic for two reasons:

  1. Anthropic’s policies on third-party harness usage kept changing, and working on their stack always felt like a worse DX.
  2. Anthropic had the highest failure rate of our three providers, from provider timeouts to model failures. At its peak, it caused more than half of all Doistbot failures.

A Datadog logs error outlier panel for @meta.model:anthropic, showing Anthropic accounted for 54.9% of total errors while making up only 10.7% of total logs.

Outgrowing a single prompt

Four months in, the council was running 1,500 reviews a month, and reviews were getting slower. Part of the problem was the prompt. Each model received one large prompt, about 40,000 tokens on average, containing general guidance, repository instructions, and the PR diff.

Diagram of the three prompts in the single-prompt council pipeline. Review prompts combine the review template, PR metadata with the body capped at 4,000 characters, the complete annotated diff, and all platform standards, and send a complete copy to Gemini and GPT-5.4. Voting prompts combine the voting template, the findings assigned to each voter, 10 lines of source context per finding, and all platform standards again, and each model votes on the other's findings. The summary prompt combines summary instructions, the PR title and body capped at 2,000 characters, and all approved finding bodies, and goes to Gemini.

Most of that size was repetition. Every review prompt carried the complete annotated diff and all of our platform standards, and the voting prompts carried the standards again. On top of that, our guidance told the models to explore the repository, and they often went down so many paths that they hit the timeout before returning anything.

Higher reasoning modes made this worse, and raising the timeouts only made reviews slower. We needed another approach.

Switching to multi-focus reviews

Newer models hallucinated far less, so voting was no longer worth its cost. That freed us to rethink the prompt itself. Rather than asking every model to look at everything, we split each review into focused passes, each with a shared base prompt plus a much smaller prompt for its area.

FocusWhat it looks for
General correctnessIncorrect behavior, crashes, broken contracts, and unhandled edge cases or error paths
ReuseReinvented helpers and utilities, and logic duplicated across the change
Code qualityLogic in the wrong layer, brittle APIs, weak typing, and drift from established patterns
EfficiencyDuplicate work, missed batching or concurrency, and hot-path or resource regressions
TestsOverly complex or brittle tests, unnecessary mocking, and untested high-value scenarios
Internal Todoist standardsClear violations of our engineering and platform standards, with a link to the relevant handbook page

These prompts weren’t written from scratch. The big labs had already done the hard work of tuning their coding agents for review, and much of it was public, so we borrowed from them. General correctness is based on Codex’s review prompt , and reuse, code quality, and efficiency are modeled on Claude Code’s /simplify command . Tests and internal Todoist standards came from our own experience and needs.

For instance here’s the tests focus as it shipped:

## Review Focus: Tests

For this pass, focus only on test quality and test-design issues introduced by the diff.

Flag findings when:
- the new tests are disproportionately complex for the behavior being covered
- the tests rely on unnecessary mocking or deep implementation knowledge
- the change duplicates existing coverage instead of extending nearby tests
- the highest-value scenario introduced by the diff is still untested
- the tests are brittle in a way that will create avoidable maintenance cost

Do not report production-code issues here unless they are directly tied to a test-quality problem.

If the diff does not introduce any findings for this focus, return an empty comments array.

Every focus prompt follows this pattern, with a narrow scope, a short list of what to flag, and explicit permission to come back empty-handed. That last line matters more than it looks. Models would often over-index on coming up with a finding, even if the finding was low value.

Every review now fanned out into 12 parallel passes, with each focus running on both Gemini and GPT-5.4. Their findings were collected, deduplicated, mapped to lines the PR actually changed, summarized, and posted as one review.

Most prompts got much smaller too. The platform standards alone were about 16,000 tokens, and they now went only to the standards passes instead of every prompt.

Diagram of the prompts in the multi-focus pipeline at launch. The shared base prompt combines the review template, PR metadata with the body capped at 4,000 characters, and the complete annotated diff, with no platform standards, and is built once per review. One of six focus prompts, each about 150 to 175 tokens, is appended to it for each pass: general correctness, reuse, code quality, efficiency, tests, and internal Todoist standards. Only the internal Todoist standards focus adds all platform standards, about 16,000 tokens. Each combined prompt goes to Gemini and GPT-5.4, for 12 parallel passes. Findings from all passes are deduplicated by Gemini, mapped to changed lines, summarized by Gemini, and posted as one GitHub review.

With the passes running side by side and no voting round to wait for, a medium-sized PR finished in 8–10 minutes, down from 18–20 with the council.

Splitting reviews into focuses also made the move to open models from our first post a gradual one. We could switch models one focus at a time, starting with tests in June and following with general correctness, quality, and efficiency in July. GPT stayed alongside the open models.

In July, we also added a security focus, bringing us to seven.

FocusModels today
General correctness, security, reuse, code quality, efficiencyDeepSeek, GLM, GPT
Tests, internal Todoist standardsDeepSeek, GLM

Measuring review quality

Until spring, we judged reviews mostly by feel and individual feedback. Craig from our Frontend team changed that by building an eval for Doistbot’s reviews.

He put together a gold set of 36 merged PRs from our web app’s history, each tagged with the issues a good review should catch and how severe they were. We measured every change against 31 of them and kept the other five out of all prompt work, as a blind check that we weren’t just tuning to the test. Every finding Doistbot posted was scored as either a real catch or a false positive. False positives ranged from nits and out-of-scope comments to issues that had already been fixed or were made up entirely. We tracked two numbers above all.

  • Recall: Whether the review caught what mattered (especially for critical issues).
  • Fabrication: Whether it invented problems.

Runs went into Langfuse so we could compare them side by side.

The first baseline, taken on the council, was precise but shallow. Its findings held up, but it caught only about half of the issues it should have. It read the diff faithfully but didn’t read the direction of the codebase. On a PR that passed a token in a URL, it found five implementation bugs but never questioned the design itself, and it ended up praising another PR for skipping flaky tests instead of fixing them.

Multi-focus moved the needle. On the blind set, it caught five of the six must-catch issues, including all five critical ones. On a PR that scrubbed passwords from request logs, it noticed that email addresses were still leaking into the same logs, which the council had missed. Five PRs is a small sample, but it was the first sign of progress on the bigger-picture issues.

Council (April)Multi-focus (end of May)
Recall55%77%
Critical recall54%79%

The evals also became our gate for model changes. A cheaper model could replace a more expensive one only if recall held and fabrication didn’t go up. Before the July swap, we ran eight recent PRs through both setups.

Gemini 3.1 Pro + GPTOpen models + GPT
False positives00
Trivial PRs left alone2 of 22 of 2
P2 findings (inline)49
P3 findings (summary)1625
Real issues only one setup caught16

They agreed on the headline issues, and each caught real problems the other missed. With one run per PR on each setup, that’s directional rather than precise, but it showed the cheaper mix held its own, and that different models looking at the same diff see different things.

Cutting the noise

Catching more issues wasn’t the same as being more useful. Multi-focus posted 64% more findings than the council on the same PRs, and some teams felt reviews had gotten noisier. Most of those findings were valid, so the question became which ones were worth someone’s immediate attention.

We moved P3 findings, the nits, out of inline comments and into a footnote in the review summary, and tightened the prompts. Inline comments dropped by about two-thirds, from roughly nine per PR to three.

A Doistbot review summary on a GitHub pull request. The summary explains that the PR widens an OAuth grant to include Comms write scopes, that the changes line up across configuration, docs, and tests, and that no inline issues were flagged. Below it, a collapsed section titled Optional follow-up notes lists three P3 findings: a code comment that now contradicts the scopes, a test constant that duplicates a scope already defined in settings, and a test that mints redundant subject tokens.

Scoring also showed where the worst noise came from. A stricter round of scoring, with a second pass that checked every finding against the diff, found that about 10% of findings were made up, rising to about 18% on mechanical PRs like renames, dependency bumps, and CI config changes. Even with permission to come back empty-handed, the models went looking for problems when there was nothing real to flag. That’s part of why we later gave mechanical PRs a smaller focus set, and skipped the review entirely for some.

Continuous improvement

Each version of the review engine fixed the biggest problem of the one before it. The council made reviews trustworthy while models still made things up, and multi-focus made them deeper once the models no longer needed to check each other’s work. Evals confirmed whether each step actually worked, and they’re still how we decide what to change next.

Some of what makes a review useful happens before it starts. Doistbot now flags PRs that are too big to review well and suggests how to split them. That’s one of several things we’ll cover in our next post, on what Doistbot does beyond just code reviews.