State-of-the-art code reviews for less than $2

By August 2025, frontier models and harnesses were reliable enough to complete complex tasks. Our engineering teams were firing on all cylinders, and PR volume was growing by double-digit percent month over month.

This growth came with a problem: our engineers were suddenly bottlenecked by reviews. We built Doistbot, an agent that uses frontier models to analyze PRs and complement human code reviews. It delivers thorough, high-quality feedback with a level of attention to detail we had not seen from automated reviews before, while costing about one-tenth as much as comparable products from frontier labs.

A year later, Doistbot is critical to how we build and ship at Todoist. It runs about 4,500 reviews each month at an average cost below $2 per review.

Making it world-class took a lot of iterations. We went from a scrappy GitHub Action to a proper agentic review harness, then to open models and dynamic routing to keep cost in check.

Here’s how we built Doistbot and what we learned along the way.

Gemini CLI in a GitHub Action

The first version of our internal automated code review served our Android team. It was simple and scrappy.

We invoked the newly released Gemini CLI through GitHub Actions, gave it the PR diff and a system prompt with our conventions, architecture, and best practices, then let it review the changes.

name: '🔎 Gemini Review'

# Triggers, concurrency, permissions, and authentication
# ...

jobs:
  review:
    runs-on: ubuntu-latest

    steps:
      - name: 'Checkout repository'
        uses: actions/checkout@v5

      # Repository parsing
      # ...

      - name: 'Run Gemini pull request review'
        uses: google-github-actions/run-gemini-cli@v0
        env:
          GITHUB_TOKEN: '${{ github.token }}'
          PULL_REQUEST_NUMBER: '${{ github.event.pull_request.number }}'

        with:
          gemini_cli_version: '${{ vars.GEMINI_CLI_VERSION }}'

          # Google Cloud authentication
          # ...

          settings: |-
            {
              "maxSessionTurns": 25,
              "mcpServers": {
                "github": {
                  "command": "docker",
                  "args": ["run", "...", "ghcr.io/github/github-mcp-server"],
                  "includeTools": [
                    "get_pull_request_diff",
                    "get_pull_request_files",
                    "create_pending_pull_request_review",
                    "submit_pending_pull_request_review"
                  ]
                }
              }
            }

          prompt: |-
            # Define the agent role, review rules, and output format

This worked surprisingly well. Gemini wasn’t as capable as Claude or GPT, but it frequently came up with high-signal feedback and found issues in the code or ways to improve it.

To bring these benefits to other teams, we turned it into a shared action.

Shared action

We added quality-of-life improvements to the action and gave it an identity: Doistbot.

The first version kept its logic and prompt in one file. We later split it into a modular composite action with reusable prompt configuration, repository-specific guidance, and support for follow-up conversations with the bot.

The new system looked like this:

This kept consumer repositories simple. They didn’t have to deal with annotating or sanitizing diffs, assembling prompts, or invoking the agent. It was all abstracted away.

name: Doistbot Review 🤖

# Triggers, concurrency, permissions, and event guards
# ...

jobs:
  review:
    runs-on: ubicloud

    steps:
      - name: Run Doistbot Review
        uses: Doist/actions/doistbot-review@main
        with:
          prompt_file: .github/workflows/doistbot-review-prompt.md
          gemini_api_key: ${{ secrets.GEMINI_STUDIO_TOKEN }}
          github_token: ${{ secrets.DOISTBOT_REVIEW_TOKEN }}

Centralizing the logic made the action easy to use, but adoption still depended on each team finding time to integrate it. We wanted a one-click install.

Migrating to a GitHub App

Henning, our Head of Frontend, proposed moving Doistbot to a GitHub App to make adoption easier. We did it for convenience, but it became one of the most crucial decisions in shaping the product.

We turned what was essentially a pile of scripts into a serverless app on AWS.

The new setup was easier to run, but it caused a major regression.

The earlier action ran Gemini inside a checked-out repository, where it could inspect files and documentation alongside the diff. The GitHub App only received a webhook, fetched the diff, called a model API, and posted the result. Without repository access, Doistbot reasoned from stale or incomplete context. It could not inspect files, follow calls across modules, or apply repository guidance reliably.

We considered repository maps, GitHub’s MCP server, API file-reading tools, and external sandboxes.

Each option cost us something: maps lost detail, MCPs were wasteful and expensive (especially GitHub’s), and external sandboxes would move source code and credentials to another provider.

The best option was the original behavior: check out the branch and let a coding agent explore it with normal command-line tools.

We separated webhook handling from review execution. Two long-running Fargate services received and sanitized webhooks, then launched a disposable Fargate sandbox for each review.

That fixed the regression: Doistbot could explore the repository again. And like every architecture change before it, the sandbox gave us more room to experiment with the review pipeline.

Unifying the harness

The models were already extremely capable, but they still hallucinated a lot. In reviews, that showed up as confident comments about bugs that didn’t exist or code the PR never touched. Frequent hallucinations like these would erode trust in the system.

Council of experts

To address this, we installed native CLIs from several providers in the sandbox and had multiple models review the same diff and vote on each other’s findings, like a council of experts. Each PR went to Claude, GPT, and Gemini for independent reviews, and only findings approved by at least two models were posted on the PR. Different models make different mistakes, so a bogus finding from one rarely convinces the others.

The council produced high-quality reviews, but it came with its own can of worms. Each model ran in its own harness (Claude Code, Codex, and Gemini CLI), and working on this part-time, we were constantly fighting fires caused by breaking changes in all three.

Pi harness

So we moved every model to Pi , an open and extensible harness with a small footprint. It cut out a lot of complexity and gave us a solid foundation to build on. Around the same time, we dropped Anthropic, which had the highest failure rate of our three providers.

After about four months, the council was handling 1,500 reviews a month, and reliability and latency started to suffer. Each model received one large prompt covering everything, and the models often explored too far and timed out before returning a result.

Conveniently, the fix for our scaling problems was also the best thing for review quality. The models had improved: their analysis was stronger and hallucinations had dropped sharply, so we no longer needed them to vote on each other’s findings. Instead, we could send each model down a specific rabbit hole, like efficiency or tests.

Multi-focus reviews

So we switched to multi-focus reviews, splitting each review into six focus areas. Each pass got a shared base prompt plus a smaller and more focused prompt for its area:

• General correctness
• Reuse
• Code quality
• Efficiency
• Tests
• Internal Todoist standards

Each focus ran on both Gemini and GPT-5.4, for 12 parallel passes per review.

Each pass now had one narrow, well-defined job, so it could spend all of its attention and tokens there. That depth and focus is where the jump in review quality came from. Instead of one broad pass per model, we had six thorough ones.

Running the passes in parallel also cut review time in half. A medium-sized PR went from about 18–20 minutes with the council to 8–10 minutes in the new multi-focus setup.

After the change, the pipeline looked like this:

We’ll dig into the prompts, the pipelines, and the trade-offs behind each of these approaches in our next post.

Adopting open models

By the time multi-focus reviews rolled out, we were averaging around 4,500 reviews a month. With reliability and latency under control, we shifted our focus to operational efficiency.

The pipeline used two main models: GPT-5.4 and Gemini 3.1 Pro. OpenAI still offered good value, but Gemini had fallen behind and no longer made sense as a primary review model.

At the same time, open models from Chinese labs were catching up. On the metrics that mattered to us, they delivered 85–90% of frontier-model performance at one-tenth of the cost.

Adopting them was almost a no-brainer. Lucky for us, our earlier investments in unifying the harness with Pi and supporting multi-focus reviews made the switch a breeze.

We replaced Gemini 3.1 Pro as a primary model with GLM 5.2 and DeepSeek V4 Pro, both routed through Pi to OpenRouter .

OpenRouter also offered zero data retention (ZDR) for the providers and models we used, which met our security and zero-data-retention needs.

Early on, we hit some quirks with OpenRouter. Uptime varied a lot across providers, and every now and then we’d run into latency or timeout failures. This hurt our reliability metrics. To fix this, we added a one-time fallback to Gemini 3.6 Flash, which is cheap and fast enough not to slow down the pipeline.

Swapping Gemini 3.1 Pro for open models, with Gemini 3.6 Flash as a fallback, cut our overall costs by about 20%. It also gave our review focuses a more diverse set of models, which improved review quality. We’ll cover how we measure that with evals in our next post.

Optimizing costs with dynamic routing

Our push for efficiency led us to the next bottleneck: cost. At our peak in June, our monthly AWS and model costs hit about $10k.

Per-review cost was $2.11, but we knew we could do better. Every PR opened at Todoist, including automated ones from bots like Renovate , triggered a full-blown review with our most expensive models. Economically, this was not very smart.

Each engineering team had a fixed setup for cutting releases, bumping dependencies, syncing copy from Transifex , and updating color tokens. These PRs were low risk: most needed only a small, focused review, and some needed none at all.

We needed to turn those patterns into classifier rules that our pipeline could use to toggle models and review focuses. After considering several overengineered options, we took the lazy route and outsourced the job to GPT-5.6 Sol in Codex.

It spent more than an hour scanning over 5,000 PRs across our repositories and produced deterministic categories.

By then, we had also added a Security focus, bringing us to seven. We encoded the routing rules in Doistbot as six categories, each with its own focus areas and models.

Reviews became dynamic: mechanical PRs now get a smaller, cheaper, and much faster review, while full reviews are reserved for actually meaningful changes.

Dynamic routing cut our per-review cost by about 30%, from a peak of $2.11 in June to $1.46 in July and August.

Built by everyone

In the last six months alone, we shipped more than 150 improvements and fixes to Doistbot. No one maintains it full-time and there’s no dedicated squad, yet it’s one of the most active and collaborative projects at Todoist. People keep poking at it, finding flaws, sharing ideas, and making it better together.

More than 20 people from several teams have contributed, and they still do. That comes from an engineering culture that gives people room to work outside their area of expertise. It’s also a great way to learn: working alongside people from other teams helps build skills that day-to-day work rarely leaves time for.

A year later

Doistbot grew up alongside a big shift in how we build. Comparing the same quarter a year apart, counting merged PRs and reviews before merge:

Over the full 12 months, around 40 Todoisters outside engineering merged PRs, compared with one the year before. Doistbot is a big part of how we’ve kept up.

When we started, none of us imagined how critical Doistbot would become to how we ship value to customers. It’s been a huge success internally, and we’re still making it better.

Reviews are only part of the story, though. Doistbot also summarizes long discussions, triages and routes bug reports, reviews our work locally before we open a PR, and understands natural language requests. One of the newest features cut the number of stalled issues nearly in half in its first week. Doistbot can now even fix incoming bug reports on its own, all the way to a pull request.

More on that soon. 👀