The first version of our AI reviewer flagged 14 "critical" issues on a one-line config change. Twelve were imaginary. One was a real typo. One was a genuine bug nobody else had caught. That ratio — one signal buried under twelve fabrications — is exactly why most teams turn the bot off within a week and never turn it back on.

We didn't want that bot. We wanted a second reviewer that never gets tired but also never hallucinates a problem into existence. A reviewer that runs our actual test suite instead of guessing at one, comments once with what it found, and costs predictable money. This is the setup that, after a few embarrassing iterations, actually held up in our CI.

Everything here runs on open source: Octomind, the session-based agent, and octomind-action, the GitHub Action that wraps it. No SaaS reviewer, no per-seat pricing, no code leaving your runner except to the model provider you already pay for.

Why "self-hosted AI reviewer" instead of a SaaS bot

There's a healthy market of hosted AI code-review products now. They work. But for AI code review in CI, three things pushed us to a self-hosted AI reviewer running inside our own GitHub Actions:

  1. The reviewer should run our commands, not its idea of our commands. A hosted bot reads the diff. It does not run cargo clippy --all-targets with our exact lint config, or our integration suite behind the right feature flags. A reviewer that can't execute the project's real checks is reduced to pattern-matching on text, which is precisely where the hallucinations come from.
  2. The agent runs on the runner we already trust. Same checkout, same secrets boundary, same network egress rules. Nothing about the diff goes anywhere we haven't already authorized.
  3. It's a config file in the repo, reviewed like any other. The reviewer's behavior is a workflow YAML and a few .agents/tools/ scripts under version control. When it gets noisy, you fix it in a PR. When a new hire asks "what does the bot actually check," the answer is git log.

If you've read our earlier piece on why custom MCPs belong in your repo, this is the same philosophy pointed at CI: the tooling that drives your agent lives next to the code it operates on.

How octomind run behaves non-interactively

Everything starts with one fact about the agent: octomind run switches to a non-interactive mode the moment you give it a --format or pipe it stdin. From the CLI definition:

--format <FORMAT>   Output format: plain or jsonl. When set, runs
                    non-interactively (reads input from stdin).

Internally the runtime resolves an output mode from the flag and whether stdin is a terminal:

Invocation Mode
octomind run in a terminal Interactive (prompts, colors, animations)
echo "..." | octomind run (piped) NonInteractive
octomind run --format plain NonInteractive
octomind run --format jsonl Jsonl (one JSON object per line)

CI is the piped case, and --format jsonl is the one you want there. It suppresses the interactive chrome and emits a stream of structured events — assistant messages, a final cost event — that a workflow can parse without scraping prose. That's the whole reason the agent is automatable: there is no hidden prompt waiting for a human to press y.

You can drive this by hand to see it:

echo "Review the staged diff and report only real defects." \
  | octomind run developer:rust --format jsonl

The last assistant event carries the review text; a trailing cost event carries token and dollar totals. That's the same stream the GitHub Action parses.

The Action, and its actual inputs

octomind-action ships two entry points. We use both for different jobs:

  • muvon/octomind-action@v1 (alias of muvon/octomind-action/run@v1) runs a single octomind run session from a prompt.
  • muvon/octomind-action/workflow@v1 runs a multi-step octomind workflow from a TOML file.

For code review, the single-session run is enough. Here are its real inputs — copied from action.yml, not invented:

Input Default What it does
prompt (required) The task/message sent to octomind
role config default Role/agent tag, e.g. developer:rust
model — Model override, e.g. openrouter:anthropic/claude-sonnet-5
name — Named session (create or resume)
resume — Resume a named session
resume_recent false Resume the most recent session for the cwd
sandbox false Restrict filesystem writes to the working directory
version latest Octomind version to install
tap — Add a tap before running
config — Path to an octomind config file
comment none PR comment mode: full, compact, or none
github_token ${{ github.token }} Token used to post the PR comment

And the outputs you can read in later steps:

Output What it carries
result Last assistant message (the review itself)
session_id Session ID, for resuming in a later step
cost {"tokens": N, "cost": N}
raw_output Full JSONL stream
exit_code Process exit code

Two of these are load-bearing for a reviewer. comment: compact posts the review as a collapsible <details> block on the PR — one comment, foldable, with the cost in a <sub> footer — instead of a wall of text. And cost lets you put the price of every review in the logs, which matters more than it sounds like it should.

Wiring the team's real lint and test through .agents/tools/

Here is the part that separated "actually held up" from "turned off in a week." A reviewer that only reads text invents problems because text is all it has. Give it the ability to run your real checks and it stops guessing.

Octomind discovers project-local tools at <workdir>/.agents/tools/<name> — shebang scripts with a comment header that becomes a tool schema. They show up to the model as a local MCP server, no separate server process, no registration. We covered the mechanics in custom MCPs belong in your repo; here's the CI-relevant point: the same scripts your developers' agents use locally are the ones the review agent runs in CI, because they're committed to the repo.

So we put the team's exact commands behind named tools.

.agents/tools/lint:

#!/usr/bin/env bash
# @description Run the project linter exactly as CI does. Returns warnings and errors.
set -euo pipefail
cd "$OCTOMIND_WORKDIR"
cargo clippy --all-targets --all-features -- -D warnings

.agents/tools/test:

#!/usr/bin/env bash
# @description Run the test suite. Defaults to the fast unit set.
# @param scope string One of: unit, all (default: unit)
set -euo pipefail
cd "$OCTOMIND_WORKDIR"
scope="${OCTOMIND_PARAM_SCOPE:-unit}"
case "$scope" in
  unit) cargo test --lib ;;
  all)  cargo test --all-targets ;;
  *) echo "Unknown scope: $scope" >&2; exit 2 ;;
esac

Now the reviewer's prompt can say "run lint and test, and only report defects you can substantiate." The agent calls the named tool, gets a real exit code and real output, and grounds its review in what the project actually does. A claim like "this will fail clippy" is no longer a guess — the agent ran clippy. If clippy passed, the claim never gets written.

This is the single biggest lever against the hallucinated-problem failure mode. The model is bad at predicting whether your code compiles. It is perfectly fine at reading the output of the command that compiled it.

Scoping what the reviewer is allowed to touch

A reviewer should read, run checks, and report. It should not edit files, push, or open PRs. Octomind roles control this with server_refs (which tool servers the role sees) and allowed_tools (which tools within them). The shape, from the shipped role templates:

[[roles]]
name = "reviewer:strict"
system = """
You are a code reviewer. Run the project's lint and test tools to ground
every claim. Report only defects you can substantiate with tool output or a
specific line. If you find nothing, say so in one sentence. Never edit files.
"""
temperature = 0.1

[roles.mcp]
server_refs = ["filesystem"]
allowed_tools = ["view", "lint", "test"]

A few deliberate choices in there. temperature = 0.1 because a reviewer should be boring and repeatable, not creative. allowed_tools lists only read and check tools — no shell write, no edit — so even if the model decided to "fix" something, it has no tool to do it with. And the system prompt's last two sentences are the noise control: report only what you can substantiate; if you find nothing, say so in one sentence. That single instruction took our average comment from fourteen "issues" to, on a clean PR, "No defects found."

For extra belt-and-suspenders, the Action's sandbox: true input restricts any filesystem writes to the working directory — so a tool that does need to write a temp file can, but nothing escapes the checkout.

Permissions and safety in a non-interactive run

The thing people worry about with an agent in CI is approvals. In an interactive session you'd be prompted before a tool runs. In CI there's no one to prompt, and that's the point: in non-interactive mode there is no y/n gate to hang the job. Safety doesn't come from a human in the loop — it comes from the loop being narrow:

  • The role's allowed_tools is the allowlist for MCP-server tools. The reviewer cannot call a server tool that isn't listed — a static boundary, not a runtime judgement call the model could be talked out of. The exception is project-local .agents/tools/ scripts, which are always exposed: their boundary is what you commit to that directory, and a PR review sees every change to it.
  • sandbox: true keeps writes inside the checkout.
  • Secrets stay in the runner's env, never in the repo. The .agents/tools/ scripts read ACME_TOKEN and friends from the environment; the script is committed, the secret is a GitHub Actions secret. If a secret is missing, the tool exits non-zero with a clear message instead of inventing a result.
  • github_token is scoped to commenting. The default ${{ github.token }} posts the PR comment; it isn't handed to the model as a tool.

The reviewer's blast radius is: read files, run two whitelisted commands, write one PR comment. That's a boundary you can reason about, which is more than you can say for "the model promised it wouldn't touch anything."

The workflow that held up

Here's the actual review job, grounded entirely in the inputs above:

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]

permissions:
  contents: read
  pull-requests: write # needed to post the review comment

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # so the agent can diff against the base

      - uses: muvon/octomind-action@v1
        with:
          role: reviewer:strict
          model: openrouter:anthropic/claude-sonnet-5
          sandbox: true
          comment: compact
          prompt: |
            Review the changes on this branch against the base.
            Run the `lint` and `test` tools to ground your findings.
            Report ONLY defects you can substantiate with tool output or a
            specific changed line. Group by severity. If nothing is wrong,
            reply in one sentence. Do not restate the diff.
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}

Notes on the choices:

  • role: reviewer:strict carries the low temperature, the tool allowlist, and the "substantiate or stay quiet" system prompt. The prompt input is the task; the role is the personality and guardrails. Keep them separate so you tune them separately.
  • model: is pinned, not left to drift. Pinning the model is half your cost control and most of your reproducibility. We pin to one Sonnet-class model and only change it on purpose.
  • comment: compact gives one foldable comment per run. On synchronize (every push to the PR) you get a fresh review without the thread turning into a scroll.
  • OPENROUTER_API_KEY is the only secret the job needs. Octomind speaks to OpenRouter, Anthropic, OpenAI, DeepSeek, Google Vertex, AWS Bedrock, and Cloudflare; the env var you set picks the provider.

That's the entire reviewer. One job, one Action, two committed tool scripts, one role definition.

Keeping the cost bounded

An agent that can run tools can, in principle, loop forever and bill forever. In practice we bound it with a few dull, effective levers:

  • Pin the model. A pinned Sonnet-class model has predictable per-token cost. "Latest and greatest" is how you get a surprise invoice.
  • Trigger on the right events. opened and synchronize only. Not on every comment, not on draft PRs (gate with if: github.event.pull_request.draft == false when you want).
  • Read the cost output and log it. Every run emits cost as {"tokens": N, "cost": N}. Pipe it into the job summary. Once the price of each review is sitting in your Actions logs, cost stops being a vibe and becomes a number you can put a threshold on.
      - run: echo "Review cost: ${{ steps.review.outputs.cost }}" >> "$GITHUB_STEP_SUMMARY"

The largest cost driver is the tools, not the prompt. A reviewer that runs your full integration suite on every push is expensive because the suite is expensive, not because the agent is. Our test tool defaults to the fast unit set for exactly this reason; the agent only reaches for scope: all when the diff touches something that warrants it, and even then we can cap it at the role level.

The gotchas, honestly

Non-determinism. Two runs on the same diff can word the review differently, and occasionally one finds something the other doesn't. temperature = 0.1 shrinks this a lot but doesn't kill it. The fix isn't to chase determinism — it's to treat the AI review as one advisory check among several. It never blocks merge on its own. A human and the real CI checks do that.

Noise is a prompt problem, not a model problem. Our first reviewer was loud because we asked it to "review the code," which an eager model reads as "find fourteen things." The cure was the two sentences in the system prompt: substantiate every claim; if you find nothing, say so in one sentence. Plus the tools — half the noise was the model guessing at outcomes it could have just run. If your reviewer is noisy, fix the prompt and give it the ability to check, before you blame the model.

Secrets in tool output. A tool that dumps environment or verbose logs can leak a secret straight into a PR comment. Keep tool output tight: return the lint/test result, not the world. We learned this when a verbose test runner echoed a connection string into the review body. Now the tools print results, not environment.

The base-ref diff. Use fetch-depth: 0 on checkout or the agent can't compare against the base branch and ends up "reviewing" the whole file. Cheap mistake, annoying symptom.

It is a second reviewer, not the reviewer. The day you start trusting it to catch everything is the day it misses the one that matters. It's a tireless first pass. The tired humans still own the merge.

FAQ

Does the agent need write access to my repo?

No. The reviewer role lists only read and check tools in allowed_tools, sandbox: true confines any write to the checkout, and github_token is used only to post the comment. The model is never handed a push or edit tool.

Can I run the team's real lint/test instead of a generic one?

That's the whole point. Put your exact commands in .agents/tools/lint and .agents/tools/test, commit them, and the agent runs them in CI. Same scripts your developers' local agents use. See custom MCPs belong in your repo.

How do I read the cost of a review?

The Action exposes a cost output ({"tokens": N, "cost": N}) and the full JSONL via raw_output. Echo cost into $GITHUB_STEP_SUMMARY and it's in every run's log.

What if I want a multi-step pipeline, not a single review?

Use muvon/octomind-action/workflow@v1 with a TOML file — gather context in one step, review in the next, summarize in a third. Same provider env, same PR-comment plumbing, plus a dry_run input to validate the pipeline without spending tokens.

Which model providers work?

Whichever you already pay for: OpenRouter, Anthropic, OpenAI, DeepSeek, Google Vertex, AWS Bedrock, Cloudflare. Set the matching API key in env and pin the model with the model input.

The shift

The instinct with an AI reviewer is to make it smarter — better model, longer prompt, more clever. What actually held up for us was making it narrower and more grounded. Fewer tools, all of them ours. A low temperature. A system prompt whose entire job is to keep the model quiet unless it can prove the point. And the ability to run the real checks, so it reasons from cargo clippy output instead of from a hunch about what cargo clippy might say.

The result isn't a genius reviewer. It's a reliable one: it never gets tired, it runs the same checks every time, and — the part that took the longest to earn — it stopped hallucinating problems into existence. For the broader set of things that shipped around this, the June 2026 round-up has the rest.

Wire it on a low-stakes repo first. Watch a few reviews. Tune the prompt twice. Then let it sit quietly on your PRs being the second pair of eyes that never blinks.

— Don

Octomind and octomind-action are open source under Apache-2.0. If your reviewer is noisy, the fix is almost always the prompt and the tools — and if it isn't, open an issue with the JSONL and we'll look.