In our first post we covered how Doistbot went from a scrappy GitHub Action to a serverless app, reviewing about 4,500 pull requests a month for less than $2 each. We kept the details around the review pipeline itself short there. This post digs deeper into it.
After starting with Gemini CLI running in GitHub Actions, Doistbot’s reviews went through two more major architectures: a council of frontier models voting on each other’s findings, and multi-model multi-focus reviews running in parallel.
We’ll also cover the evals we built to measure whether each change actually made reviews better.
Council of experts
Until early 2026, every review came from a single model. It worked, but the rate of hallucination was still quite high. Doistbot would confidently flag bugs that weren’t there, or comment on code the PR never touched. Every false flag chipped away at people’s trust in Doistbot.
So instead of trusting one model, we ran three models against the same diff and had them check each other’s work. Each provider ran in its own harness (Claude Code, Codex, and Gemini CLI), and every harness implemented the same interface:
type ProviderName = 'gemini' | 'openai' | 'anthropic'
type ReviewProviderConfig = {
apiKey: string
modelName?: string
timeout?: number
jsonSchema?: Record<string, unknown>
}
interface ReviewProvider {
readonly name: string
review(
prompt: string,
config: ReviewProviderConfig,
): Promise<ModelReviewResult>
}
Then came the voting phase. We grouped matching findings from different models, and the models that hadn’t raised a finding voted on it, with the surrounding code for context. A finding needed two-thirds of their votes to make it onto the PR.
const CONSENSUS_THRESHOLD = 0.67 // 2/3 approval required
async function calculateConsensusForGroup(
group: FindingGroup,
availableModels: ProviderName[],
): Promise<ConsensusResult> {
const finding = group.primaryFinding
// Every model raised it, so there's no one left to vote
if (availableModels.every((m) => group.sourceModels.includes(m))) {
return { findingId: finding.id, finding, approvalRatio: 1, isApproved: true, votes: [] }
}
// Models that raised the finding don't get a vote
const voters = availableModels.filter((m) => !group.sourceModels.includes(m))
const votes = await Promise.all(voters.map((voter) => getVote(voter, finding)))
const approvals = votes.filter((v) => v.approve).length
const approvalRatio = votes.length > 0 ? approvals / votes.length : 0
return {
findingId: finding.id,
finding,
approvalRatio,
isApproved: approvalRatio >= CONSENSUS_THRESHOLD,
votes,
}
}
The vote worked because the models rarely hallucinated the same thing. A finding one model made up usually failed to convince the other two, while real issues got through. The approved findings were then summarized and posted as one review.
Together, the pipeline looked like this:
The council was a big step up in review quality, but it meant keeping three harnesses running, each with its own quirks, limits and release cycles. Since we worked on Doistbot part-time while more and more of the company relied on it, every breaking change in one of them meant dropping what we were doing to fix it.
Migrating to Pi
By then, open harnesses were becoming popular. After a promising review from our CTO, Gonçalo, we took a closer look at Pi , an up-and-coming, minimal harness.
Pi was a breath of fresh air. It got a lot of decisions right, was extremely extensible, and had a small footprint, letting us customize as much or as little as we wanted. For what we needed, Pi fit like a glove.
Once we moved Doistbot to Pi, it was immediately obvious we’d made the right call. Every model now ran through the same harness, so there was one integration to maintain instead of three.
Around the same time, we dropped Anthropic for two reasons:
- Anthropic’s policies on third-party harness usage kept changing, and working on their stack always felt like a worse DX.
- Anthropic had the highest failure rate of our three providers, from provider timeouts to model failures. At its peak, it caused more than half of all Doistbot failures.

Outgrowing a single prompt
Four months in, the council was running 1,500 reviews a month, and reviews were getting slower. Part of the problem was the prompt. Each model received one large prompt, about 40,000 tokens on average, containing general guidance, repository instructions, and the PR diff.
Most of that size was repetition. Every review prompt carried the complete annotated diff and all of our platform standards, and the voting prompts carried the standards again. On top of that, our guidance told the models to explore the repository, and they often went down so many paths that they hit the timeout before returning anything.
Higher reasoning modes made this worse, and raising the timeouts only made reviews slower. We needed another approach.
Switching to multi-focus reviews
Newer models hallucinated far less, so voting was no longer worth its cost. That freed us to rethink the prompt itself. Rather than asking every model to look at everything, we split each review into focused passes, each with a shared base prompt plus a much smaller prompt for its area.
| Focus | What it looks for |
|---|---|
| General correctness | Incorrect behavior, crashes, broken contracts, and unhandled edge cases or error paths |
| Reuse | Reinvented helpers and utilities, and logic duplicated across the change |
| Code quality | Logic in the wrong layer, brittle APIs, weak typing, and drift from established patterns |
| Efficiency | Duplicate work, missed batching or concurrency, and hot-path or resource regressions |
| Tests | Overly complex or brittle tests, unnecessary mocking, and untested high-value scenarios |
| Internal Todoist standards | Clear violations of our engineering and platform standards, with a link to the relevant handbook page |
These prompts weren’t written from scratch. The big labs had already done the hard work of tuning their coding agents for review, and much of it was public, so we borrowed from them. General correctness is based on Codex’s review prompt , and reuse, code quality, and efficiency are modeled on Claude Code’s /simplify command . Tests and internal Todoist standards came from our own experience and needs.
For instance here’s the tests focus as it shipped:
## Review Focus: Tests
For this pass, focus only on test quality and test-design issues introduced by the diff.
Flag findings when:
- the new tests are disproportionately complex for the behavior being covered
- the tests rely on unnecessary mocking or deep implementation knowledge
- the change duplicates existing coverage instead of extending nearby tests
- the highest-value scenario introduced by the diff is still untested
- the tests are brittle in a way that will create avoidable maintenance cost
Do not report production-code issues here unless they are directly tied to a test-quality problem.
If the diff does not introduce any findings for this focus, return an empty comments array.
Every focus prompt follows this pattern, with a narrow scope, a short list of what to flag, and explicit permission to come back empty-handed. That last line matters more than it looks. Models would often over-index on coming up with a finding, even if the finding was low value.
Every review now fanned out into 12 parallel passes, with each focus running on both Gemini and GPT-5.4. Their findings were collected, deduplicated, mapped to lines the PR actually changed, summarized, and posted as one review.
Most prompts got much smaller too. The platform standards alone were about 16,000 tokens, and they now went only to the standards passes instead of every prompt.
With the passes running side by side and no voting round to wait for, a medium-sized PR finished in 8–10 minutes, down from 18–20 with the council.
Splitting reviews into focuses also made the move to open models from our first post a gradual one. We could switch models one focus at a time, starting with tests in June and following with general correctness, quality, and efficiency in July. GPT stayed alongside the open models.
In July, we also added a security focus, bringing us to seven.
| Focus | Models today |
|---|---|
| General correctness, security, reuse, code quality, efficiency | DeepSeek, GLM, GPT |
| Tests, internal Todoist standards | DeepSeek, GLM |
Measuring review quality
Until spring, we judged reviews mostly by feel and individual feedback. Craig from our Frontend team changed that by building an eval for Doistbot’s reviews.
He put together a gold set of 36 merged PRs from our web app’s history, each tagged with the issues a good review should catch and how severe they were. We measured every change against 31 of them and kept the other five out of all prompt work, as a blind check that we weren’t just tuning to the test. Every finding Doistbot posted was scored as either a real catch or a false positive. False positives ranged from nits and out-of-scope comments to issues that had already been fixed or were made up entirely. We tracked two numbers above all.
- Recall: Whether the review caught what mattered (especially for critical issues).
- Fabrication: Whether it invented problems.
Runs went into Langfuse so we could compare them side by side.
The first baseline, taken on the council, was precise but shallow. Its findings held up, but it caught only about half of the issues it should have. It read the diff faithfully but didn’t read the direction of the codebase. On a PR that passed a token in a URL, it found five implementation bugs but never questioned the design itself, and it ended up praising another PR for skipping flaky tests instead of fixing them.
Multi-focus moved the needle. On the blind set, it caught five of the six must-catch issues, including all five critical ones. On a PR that scrubbed passwords from request logs, it noticed that email addresses were still leaking into the same logs, which the council had missed. Five PRs is a small sample, but it was the first sign of progress on the bigger-picture issues.
| Council (April) | Multi-focus (end of May) | |
|---|---|---|
| Recall | 55% | 77% |
| Critical recall | 54% | 79% |
The evals also became our gate for model changes. A cheaper model could replace a more expensive one only if recall held and fabrication didn’t go up. Before the July swap, we ran eight recent PRs through both setups.
| Gemini 3.1 Pro + GPT | Open models + GPT | |
|---|---|---|
| False positives | 0 | 0 |
| Trivial PRs left alone | 2 of 2 | 2 of 2 |
| P2 findings (inline) | 4 | 9 |
| P3 findings (summary) | 16 | 25 |
| Real issues only one setup caught | 1 | 6 |
They agreed on the headline issues, and each caught real problems the other missed. With one run per PR on each setup, that’s directional rather than precise, but it showed the cheaper mix held its own, and that different models looking at the same diff see different things.
Cutting the noise
Catching more issues wasn’t the same as being more useful. Multi-focus posted 64% more findings than the council on the same PRs, and some teams felt reviews had gotten noisier. Most of those findings were valid, so the question became which ones were worth someone’s immediate attention.
We moved P3 findings, the nits, out of inline comments and into a footnote in the review summary, and tightened the prompts. Inline comments dropped by about two-thirds, from roughly nine per PR to three.

Scoring also showed where the worst noise came from. A stricter round of scoring, with a second pass that checked every finding against the diff, found that about 10% of findings were made up, rising to about 18% on mechanical PRs like renames, dependency bumps, and CI config changes. Even with permission to come back empty-handed, the models went looking for problems when there was nothing real to flag. That’s part of why we later gave mechanical PRs a smaller focus set, and skipped the review entirely for some.
Continuous improvement
Each version of the review engine fixed the biggest problem of the one before it. The council made reviews trustworthy while models still made things up, and multi-focus made them deeper once the models no longer needed to check each other’s work. Evals confirmed whether each step actually worked, and they’re still how we decide what to change next.
Some of what makes a review useful happens before it starts. Doistbot now flags PRs that are too big to review well and suggests how to split them. That’s one of several things we’ll cover in our next post, on what Doistbot does beyond just code reviews.