# AI Code Review in CI With an Open-Source Agent: The Setup That Actually Held Up > We wanted a second reviewer that never gets tired but also never hallucinates a problem into existence. Here is how we wired Octomind into GitHub Actions for automated code review — the real octomind-action inputs, running the team's actual lint and test commands through .agents/tools/, keeping CI non-interactive and safe, and bounding the cost. The first version of our AI reviewer flagged 14 "critical" issues on a one-line config change. Twelve were imaginary. One was a real typo. One was a genuine bug nobody else had caught. That ratio — one signal buried under twelve fabrications — is exactly why most teams turn the bot off within a week and never turn it back on. We didn't want that bot. We wanted a second reviewer that never gets tired but also never hallucinates a problem into existence. A reviewer that runs our _actual_ test suite instead of guessing at one, comments once with what it found, and costs predictable money. This is the setup that, after a few embarrassing iterations, actually held up in our CI. Everything here runs on open source: [Octomind](https://github.com/Muvon/octomind), the session-based agent, and [octomind-action](https://github.com/Muvon/octomind-action), the GitHub Action that wraps it. No SaaS reviewer, no per-seat pricing, no code leaving your runner except to the model provider you already pay for. --- ## Why "self-hosted AI reviewer" instead of a SaaS bot There's a healthy market of hosted AI code-review products now. They work. But for AI code review in CI, three things pushed us to a self-hosted AI reviewer running inside our own GitHub Actions: 1. **The reviewer should run our commands, not its idea of our commands.** A hosted bot reads the diff. It does not run `cargo clippy --all-targets` with our exact lint config, or our integration suite behind the right feature flags. A reviewer that can't execute the project's real checks is reduced to pattern-matching on text, which is precisely where the hallucinations come from. 2. **The agent runs on the runner we already trust.** Same checkout, same secrets boundary, same network egress rules. Nothing about the diff goes anywhere we haven't already authorized. 3. **It's a config file in the repo, reviewed like any other.** The reviewer's behavior is a workflow YAML and a few `.agents/tools/` scripts under version control. When it gets noisy, you fix it in a PR. When a new hire asks "what does the bot actually check," the answer is `git log`. If you've read our earlier piece on why [custom MCPs belong in your repo](/blog/custom-mcps-belong-in-your-repo), this is the same philosophy pointed at CI: the tooling that drives your agent lives next to the code it operates on. --- ## How `octomind run` behaves non-interactively Everything starts with one fact about the agent: `octomind run` switches to a non-interactive mode the moment you give it a `--format` or pipe it stdin. From the CLI definition: ``` --format Output format: plain or jsonl. When set, runs non-interactively (reads input from stdin). ``` Internally the runtime resolves an output mode from the flag and whether stdin is a terminal: | Invocation | Mode | | ------------------------------------ | ------------------------------------------- | | `octomind run` in a terminal | `Interactive` (prompts, colors, animations) | | `echo "..." \| octomind run` (piped) | `NonInteractive` | | `octomind run --format plain` | `NonInteractive` | | `octomind run --format jsonl` | `Jsonl` (one JSON object per line) | CI is the piped case, and `--format jsonl` is the one you want there. It suppresses the interactive chrome and emits a stream of structured events — assistant messages, a final cost event — that a workflow can parse without scraping prose. That's the whole reason the agent is automatable: there is no hidden prompt waiting for a human to press `y`. You can drive this by hand to see it: ```bash echo "Review the staged diff and report only real defects." \ | octomind run developer:rust --format jsonl ``` The last `assistant` event carries the review text; a trailing `cost` event carries token and dollar totals. That's the same stream the GitHub Action parses. --- ## The Action, and its actual inputs `octomind-action` ships two entry points. We use both for different jobs: - `muvon/octomind-action@v1` (alias of `muvon/octomind-action/run@v1`) runs a single `octomind run` session from a prompt. - `muvon/octomind-action/workflow@v1` runs a multi-step `octomind workflow` from a TOML file. For code review, the single-session `run` is enough. Here are its real inputs — copied from `action.yml`, not invented: | Input | Default | What it does | | --------------- | --------------------- | ----------------------------------------------------------- | | `prompt` | _(required)_ | The task/message sent to octomind | | `role` | config default | Role/agent tag, e.g. `developer:rust` | | `model` | — | Model override, e.g. `openrouter:anthropic/claude-sonnet-5` | | `name` | — | Named session (create or resume) | | `resume` | — | Resume a named session | | `resume_recent` | `false` | Resume the most recent session for the cwd | | `sandbox` | `false` | Restrict filesystem writes to the working directory | | `version` | `latest` | Octomind version to install | | `tap` | — | Add a tap before running | | `config` | — | Path to an octomind config file | | `comment` | `none` | PR comment mode: `full`, `compact`, or `none` | | `github_token` | `${{ github.token }}` | Token used to post the PR comment | And the outputs you can read in later steps: | Output | What it carries | | ------------ | ------------------------------------------ | | `result` | Last assistant message (the review itself) | | `session_id` | Session ID, for resuming in a later step | | `cost` | `{"tokens": N, "cost": N}` | | `raw_output` | Full JSONL stream | | `exit_code` | Process exit code | Two of these are load-bearing for a reviewer. `comment: compact` posts the review as a collapsible `
` block on the PR — one comment, foldable, with the cost in a `` footer — instead of a wall of text. And `cost` lets you put the price of every review in the logs, which matters more than it sounds like it should. --- ## Wiring the team's real `lint` and `test` through `.agents/tools/` Here is the part that separated "actually held up" from "turned off in a week." A reviewer that only reads text invents problems because text is all it has. Give it the ability to run your real checks and it stops guessing. Octomind discovers project-local tools at `/.agents/tools/` — shebang scripts with a comment header that becomes a tool schema. They show up to the model as a `local` MCP server, no separate server process, no registration. We covered the mechanics in [custom MCPs belong in your repo](/blog/custom-mcps-belong-in-your-repo); here's the CI-relevant point: the same scripts your developers' agents use locally are the ones the review agent runs in CI, because they're committed to the repo. So we put the team's _exact_ commands behind named tools. `.agents/tools/lint`: ```bash #!/usr/bin/env bash # @description Run the project linter exactly as CI does. Returns warnings and errors. set -euo pipefail cd "$OCTOMIND_WORKDIR" cargo clippy --all-targets --all-features -- -D warnings ``` `.agents/tools/test`: ```bash #!/usr/bin/env bash # @description Run the test suite. Defaults to the fast unit set. # @param scope string One of: unit, all (default: unit) set -euo pipefail cd "$OCTOMIND_WORKDIR" scope="${OCTOMIND_PARAM_SCOPE:-unit}" case "$scope" in unit) cargo test --lib ;; all) cargo test --all-targets ;; *) echo "Unknown scope: $scope" >&2; exit 2 ;; esac ``` Now the reviewer's prompt can say "run `lint` and `test`, and only report defects you can substantiate." The agent calls the named tool, gets a real exit code and real output, and grounds its review in what the project actually does. A claim like "this will fail clippy" is no longer a guess — the agent ran clippy. If clippy passed, the claim never gets written. This is the single biggest lever against the hallucinated-problem failure mode. The model is bad at predicting whether your code compiles. It is perfectly fine at _reading the output of the command that compiled it_. --- ## Scoping what the reviewer is allowed to touch A reviewer should read, run checks, and report. It should not edit files, push, or open PRs. Octomind roles control this with `server_refs` (which tool servers the role sees) and `allowed_tools` (which tools within them). The shape, from the shipped role templates: ```toml [[roles]] name = "reviewer:strict" system = """ You are a code reviewer. Run the project's lint and test tools to ground every claim. Report only defects you can substantiate with tool output or a specific line. If you find nothing, say so in one sentence. Never edit files. """ temperature = 0.1 [roles.mcp] server_refs = ["filesystem"] allowed_tools = ["view", "lint", "test"] ``` A few deliberate choices in there. `temperature = 0.1` because a reviewer should be boring and repeatable, not creative. `allowed_tools` lists only read and check tools — no shell write, no edit — so even if the model decided to "fix" something, it has no tool to do it with. And the system prompt's last two sentences are the noise control: _report only what you can substantiate; if you find nothing, say so in one sentence._ That single instruction took our average comment from fourteen "issues" to, on a clean PR, "No defects found." For extra belt-and-suspenders, the Action's `sandbox: true` input restricts any filesystem writes to the working directory — so a tool that does need to write a temp file can, but nothing escapes the checkout. --- ## Permissions and safety in a non-interactive run The thing people worry about with an agent in CI is approvals. In an interactive session you'd be prompted before a tool runs. In CI there's no one to prompt, and that's the point: in non-interactive mode there is no `y/n` gate to hang the job. Safety doesn't come from a human in the loop — it comes from the loop being narrow: - **The role's `allowed_tools` is the allowlist for MCP-server tools.** The reviewer cannot call a server tool that isn't listed — a static boundary, not a runtime judgement call the model could be talked out of. The exception is project-local `.agents/tools/` scripts, which are always exposed: their boundary is what you commit to that directory, and a PR review sees every change to it. - **`sandbox: true`** keeps writes inside the checkout. - **Secrets stay in the runner's env**, never in the repo. The `.agents/tools/` scripts read `ACME_TOKEN` and friends from the environment; the script is committed, the secret is a GitHub Actions secret. If a secret is missing, the tool exits non-zero with a clear message instead of inventing a result. - **`github_token` is scoped to commenting.** The default `${{ github.token }}` posts the PR comment; it isn't handed to the model as a tool. The reviewer's blast radius is: read files, run two whitelisted commands, write one PR comment. That's a boundary you can reason about, which is more than you can say for "the model promised it wouldn't touch anything." --- ## The workflow that held up Here's the actual review job, grounded entirely in the inputs above: ```yaml name: AI Code Review on: pull_request: types: [opened, synchronize] permissions: contents: read pull-requests: write # needed to post the review comment jobs: review: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 # so the agent can diff against the base - uses: muvon/octomind-action@v1 with: role: reviewer:strict model: openrouter:anthropic/claude-sonnet-5 sandbox: true comment: compact prompt: | Review the changes on this branch against the base. Run the `lint` and `test` tools to ground your findings. Report ONLY defects you can substantiate with tool output or a specific changed line. Group by severity. If nothing is wrong, reply in one sentence. Do not restate the diff. env: OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} ``` Notes on the choices: - **`role: reviewer:strict`** carries the low temperature, the tool allowlist, and the "substantiate or stay quiet" system prompt. The prompt input is the _task_; the role is the _personality and guardrails_. Keep them separate so you tune them separately. - **`model:` is pinned**, not left to drift. Pinning the model is half your cost control and most of your reproducibility. We pin to one Sonnet-class model and only change it on purpose. - **`comment: compact`** gives one foldable comment per run. On `synchronize` (every push to the PR) you get a fresh review without the thread turning into a scroll. - **`OPENROUTER_API_KEY`** is the only secret the job needs. Octomind speaks to OpenRouter, Anthropic, OpenAI, DeepSeek, Google Vertex, AWS Bedrock, and Cloudflare; the env var you set picks the provider. That's the entire reviewer. One job, one Action, two committed tool scripts, one role definition. --- ## Keeping the cost bounded An agent that can run tools can, in principle, loop forever and bill forever. In practice we bound it with a few dull, effective levers: - **Pin the model.** A pinned Sonnet-class model has predictable per-token cost. "Latest and greatest" is how you get a surprise invoice. - **Trigger on the right events.** `opened` and `synchronize` only. Not on every comment, not on draft PRs (gate with `if: github.event.pull_request.draft == false` when you want). - **Read the `cost` output and log it.** Every run emits `cost` as `{"tokens": N, "cost": N}`. Pipe it into the job summary. Once the price of each review is sitting in your Actions logs, cost stops being a vibe and becomes a number you can put a threshold on. ```yaml - run: echo "Review cost: ${{ steps.review.outputs.cost }}" >> "$GITHUB_STEP_SUMMARY" ``` The largest cost driver is the tools, not the prompt. A reviewer that runs your full integration suite on every push is expensive because the _suite_ is expensive, not because the agent is. Our `test` tool defaults to the fast unit set for exactly this reason; the agent only reaches for `scope: all` when the diff touches something that warrants it, and even then we can cap it at the role level. --- ## The gotchas, honestly **Non-determinism.** Two runs on the same diff can word the review differently, and occasionally one finds something the other doesn't. `temperature = 0.1` shrinks this a lot but doesn't kill it. The fix isn't to chase determinism — it's to treat the AI review as one _advisory_ check among several. It never blocks merge on its own. A human and the real CI checks do that. **Noise is a prompt problem, not a model problem.** Our first reviewer was loud because we asked it to "review the code," which an eager model reads as "find fourteen things." The cure was the two sentences in the system prompt: _substantiate every claim; if you find nothing, say so in one sentence._ Plus the tools — half the noise was the model guessing at outcomes it could have just run. If your reviewer is noisy, fix the prompt and give it the ability to check, before you blame the model. **Secrets in tool output.** A tool that dumps environment or verbose logs can leak a secret straight into a PR comment. Keep tool output tight: return the lint/test result, not the world. We learned this when a verbose test runner echoed a connection string into the review body. Now the tools print results, not environment. **The base-ref diff.** Use `fetch-depth: 0` on checkout or the agent can't compare against the base branch and ends up "reviewing" the whole file. Cheap mistake, annoying symptom. **It is a second reviewer, not the reviewer.** The day you start trusting it to catch everything is the day it misses the one that matters. It's a tireless first pass. The tired humans still own the merge. --- ## FAQ **Does the agent need write access to my repo?** No. The reviewer role lists only read and check tools in `allowed_tools`, `sandbox: true` confines any write to the checkout, and `github_token` is used only to post the comment. The model is never handed a push or edit tool. **Can I run the team's real lint/test instead of a generic one?** That's the whole point. Put your exact commands in `.agents/tools/lint` and `.agents/tools/test`, commit them, and the agent runs them in CI. Same scripts your developers' local agents use. See [custom MCPs belong in your repo](/blog/custom-mcps-belong-in-your-repo). **How do I read the cost of a review?** The Action exposes a `cost` output (`{"tokens": N, "cost": N}`) and the full JSONL via `raw_output`. Echo `cost` into `$GITHUB_STEP_SUMMARY` and it's in every run's log. **What if I want a multi-step pipeline, not a single review?** Use `muvon/octomind-action/workflow@v1` with a TOML file — gather context in one step, review in the next, summarize in a third. Same provider env, same PR-comment plumbing, plus a `dry_run` input to validate the pipeline without spending tokens. **Which model providers work?** Whichever you already pay for: OpenRouter, Anthropic, OpenAI, DeepSeek, Google Vertex, AWS Bedrock, Cloudflare. Set the matching API key in `env` and pin the model with the `model` input. --- ## The shift The instinct with an AI reviewer is to make it smarter — better model, longer prompt, more clever. What actually held up for us was making it _narrower and more grounded_. Fewer tools, all of them ours. A low temperature. A system prompt whose entire job is to keep the model quiet unless it can prove the point. And the ability to run the real checks, so it reasons from `cargo clippy` output instead of from a hunch about what `cargo clippy` might say. The result isn't a genius reviewer. It's a reliable one: it never gets tired, it runs the same checks every time, and — the part that took the longest to earn — it stopped hallucinating problems into existence. For the broader set of things that shipped around this, the [June 2026 round-up](/blog/release-round-june-2026) has the rest. Wire it on a low-stakes repo first. Watch a few reviews. Tune the prompt twice. Then let it sit quietly on your PRs being the second pair of eyes that never blinks. — Don --- _[Octomind](https://github.com/Muvon/octomind) and [octomind-action](https://github.com/Muvon/octomind-action) are open source under Apache-2.0. If your reviewer is noisy, the fix is almost always the prompt and the tools — and if it isn't, [open an issue](https://github.com/Muvon/octomind/issues) with the JSONL and we'll look._