0.17.1 let you run every embedding Octocode generates on hardware you control — any OpenAI-compatible server, any model, your code never leaving your network. That release was about where the search runs.
0.18.0 is about how good the search actually is — and proving it with numbers you can reproduce.
Most code-search tools claim quality. Very few ship a benchmark you can actually run. This release ships two: a curated retrieval benchmark against Octocode's own source, and a Loc-Bench harness over 560 real GitHub issues. It also adds GraphRAG file-level expansion — a search step that pulls in structurally related files before the reranker scores them.
Here's what changed, and what the numbers say — including the parts that didn't work.
Why Benchmark Code Search at All?
When you change a chunking heuristic or a fusion weight, you want to know if retrieval got better or worse. Eyeballing a few queries won't tell you — it just confirms what you already believed. The only honest answer is a fixed set of queries, fixed ground truth, and a number that moves.
So 0.18.0 ships a real one. It lives in benchmark/ in the repo, and it measures Octocode's full pipeline end to end: chunking → embedding → vector search → hybrid fusion → reranking.
The ground truth
code.csv holds 127 queries, each annotated with 1–3 correct code locations as line ranges:
query,result1,result2,result3
Each result is src/path/file.rs:start-end:relevance, where relevance 2 is primary (directly answers the query) and 1 is secondary (useful context). A search result matches by line-range overlap — a hit at lines 40–90 matches ground truth 45–92 because the ranges intersect.
The corpus is pinned to a single commit (b1771ba). Ground truth is annotated against those exact line numbers, so indexing any other checkout would drift the ranges and quietly break the scores. Pin first, then index.
How the ground truth was built matters, because a sloppy answer key makes any benchmark lie:
- AI-generated candidates across every major module and doc file.
- Source verification — every referenced file and line range was read and checked against the actual code.
- Multi-agent validation — parallel agents independently re-checked line ranges, paths, and relevance grades.
- Search-informed corrections — if search found the same logic in a valid alternate location, that location was added as a secondary result, not deleted.
The hard queries
The last 27 of the 127 are deliberately adversarial: natural language that doesn't mirror any function name or comment. They test semantic understanding, not keyword luck. A few:
| Query intent | Why it's hard |
|---|---|
| Preventing infinite loops in overlapping chunks | 5-line guard clause, no code keywords in the query |
| Skipping indexing when the repo is unchanged | buried 100+ lines into a 900-line function |
| Rough token estimation without a tokenizer | the answer is a single line: s.len() / 4 |
| Preventing concurrent reindexing in the MCP server | AtomicBool + compare_exchange, a concurrency pattern |
If standard queries score near 1.0 but hard queries lag, the benchmark is doing its job: it's exposing real retrieval weaknesses instead of flattering the tool.
The metrics
Four standard IR metrics, all "higher is better":
- Hit@k — did any correct result land in the top k? (coverage)
- MRR — reciprocal rank of the first correct result. (how high is the first hit?)
- NDCG@10 — rewards putting the most relevant results highest, weighting relevance grade and position. (ranking quality)
- Recall@k — what fraction of all ground-truth blocks were found? (completeness)
Definitions and a worked example are in benchmark/README.md.
The Numbers: A Local, No-Key Floor
These results use the fully local, no-API-key stack — jina-embeddings-v2-base-code via fastembed, no reranker. That makes them a floor, not a ceiling: a paid code-aware reranker pushes higher. Run on Octocode's own source, 127 code queries:
| Config | Hit@5 | Hit@10 | MRR | NDCG@10 | Recall@10 |
|---|---|---|---|---|---|
| Dense vector only | 0.598 | 0.717 | 0.485 | 0.528 | 0.671 |
| Hybrid, default RRF weights (0.7/0.3) | 0.598 | 0.717 | 0.485 | 0.528 | 0.671 |
| Hybrid, keyword-tuned (0.3/0.7) | 0.732 | 0.835 | 0.572 | 0.620 | 0.807 |
Two things jump out.
The default RRF weights behave like vector-only. With fusion tilted 0.7 toward the dense signal, the vector results dominate and the keyword signal barely registers — the two top rows are identical.
Tilting toward keyword is a free, large win for code. Flip the weights to 0.3/0.7 — letting the BM25/keyword signal lead — and Hit@5 jumps +22% and Recall@10 +20%, at zero added cost. Why? Code is full of exact identifiers — compare_exchange, get_code_blocks_by_path, RelationshipDirection — and exact-token matching carries disproportionate weight when the thing you're searching for is a token. Pure semantic similarity blurs that; keyword fusion sharpens it.
That's the kind of result you only find by measuring. It's not intuitive that for code you'd want keyword to outweigh the embedding.
What Didn't Help (And Why That's in the Benchmark)
A benchmark that only reports wins is marketing. This one reports the regressions too — the full 6-variant matrix is in benchmark/RESULTS.md.
A generic reranker made things worse. Dropping a local off-the-shelf cross-encoder (bge-reranker-base) on top of the best config didn't help — it regressed Hit@5 from 0.732 back down to 0.598, undoing the entire keyword-tuning win. And it was roughly 4× slower (≈1900s vs ≈490s for the run).
The lesson isn't "rerankers are bad." It's that code retrieval needs a code-aware reranker. A reranker trained on prose re-scores fn verify_hmac against "webhook signature validation" worse than the hybrid fusion already did. A code-tuned reranker like voyage:rerank-2.5 is a different story — but you have to verify that on your own eval, not assume it. The benchmark is what makes that possible.
GraphRAG File-Level Expansion
The headline feature alongside the benchmark. Dense and keyword search both score blocks in isolation — they don't know that the file you want imports the file they found, or calls a function defined next door. GraphRAG file-level expansion closes that gap.
After the initial hybrid search produces ranked candidates, Octocode takes the strongest few, walks the GraphRAG graph for files structurally related by imports, calls, or extends, pulls a handful of blocks from those neighbor files, and hands the enlarged candidate set to the reranker:
// Seeds are already RRF-ranked; only expand from the strongest few.
const SEED_LIMIT: usize = 10; // expand from top 10 hits
const MAX_NEIGHBOR_FILES: usize = 8; // pull at most 8 related files
const BLOCKS_PER_FILE: usize = 3; // 3 blocks per neighbor
Neighbors are scored by weight × confidence across all seed files, strongest links first. It's file-granular because the graph is file-granular (CodeNode.id == relative path). And it's best-effort: any graph error degrades to passing the original results straight through — it never breaks search.
Turn it on in config.toml:
[search]
graph_expansion = false # default; requires [graphrag] enabled
It ships off by default, on purpose. Here's the honest part from the benchmark:
+graph= GraphRAG file-level expansion, which only moves results once a reranker re-scores the enlarged set.
Without a reranker, expansion adds candidates but doesn't change the ranking — so hybrid_30_70 and hybrid_30_70+graph score identically in the local, no-key matrix above. Expansion's payoff comes when a (code-aware) reranker re-scores the bigger pool and promotes a structurally related file that raw similarity ranked too low. Naive expansion can also dilute precision. So the guidance baked into the config comment is blunt: A/B it on your own eval before trusting it. The benchmark exists precisely so you can do that.
Loc-Bench: Issue → Code, Nothing Self-Annotated
The curated benchmark is annotated by us. A fair question: are the queries too friendly to our own tool? 0.18.0 ships a second harness that removes us from the loop entirely.
benchmark/locbench.py evaluates Octocode on Loc-Bench V1 (czlll/Loc-Bench_V1): 560 real GitHub issues from popular Python repos. The query is the original issue text. The ground truth is the set of files and functions the real merged fix actually edited (the dataset's curated edit_functions field). Nothing is self-annotated — it's the industry "given a bug report, find the code to change" task.
Per instance, the harness clones the repo (blobless, cached), checks out the base commit, runs octocode index, searches with the issue text, and scores the retrieved files against the real fix — at both file level (hit@k, recall@k, MRR, accuracy) and hunk level (did a retrieved result overlap the lines the patch actually touched?), broken down by issue category:
python3 benchmark/locbench.py --limit 10 --verbose # smoke test
python3 benchmark/locbench.py # full 560
python3 benchmark/locbench.py --category "Bug Report"
This is a measurement tool, not a leaderboard claim — it's there so you can run real-world issue localization on your own hardware and models, and so every future change to Octocode can be measured against a benchmark we didn't write.
The Variant Matrix
score.py measures one configuration. run_matrix.py sweeps several over one shared index — so the only thing changing between rows is search-time parameters, not the corpus or the embeddings. It writes a comparison table to RESULTS.md (and raw RESULTS.json), and accumulates across runs so a focused re-run merges into the existing table.
# Pin the corpus to the ground-truth commit first — or the line ranges drift
git worktree add /tmp/corpus b1771ba
# Full local matrix (fastembed, no API key): vector-only vs hybrid weight sweep
CORPUS=/tmp/corpus python3 benchmark/run_matrix.py
# Add a local cross-encoder reranker (still no key) — reuses the same index
CORPUS=/tmp/corpus SKIP_INDEX=1 ONLY=rerank \
RERANK_MODEL=fastembed:bge-reranker-base python3 benchmark/run_matrix.py
Variants: vector_only, hybrid_70_30 (default weights), hybrid_30_70 (keyword-tilted), +graph (GraphRAG expansion), and +rerank (cross-encoder). If you set VOYAGE_API_KEY, the paid code-aware reranker variants enable automatically — that's how you find the real ceiling above the local floor.
Everything Else
MCP registry metadata and server schema updated. Keeps Octocode's published MCP server descriptor in step with the actual tool surface, so registries and clients see the right schema. Plus the usual dependency and tooling housekeeping — no behavior change.
Upgrade
# Homebrew
brew upgrade muvon/tap/octocode
# Universal installer
curl -fsSL https://raw.githubusercontent.com/Muvon/octocode/master/install.sh | sh
# Cargo
cargo install octocode --version 0.18.0
Nothing in this release changes your index or config — upgrading is safe, and your existing setup keeps working. Two things to try:
- Tilt your fusion weights toward keyword. If you search code more than prose, the benchmark says keyword-led RRF is a free accuracy win. Measure it on your repo.
- Run the benchmark. Clone Octocode, pin the corpus, and run
run_matrix.pyagainst your own embedding model and reranker. The numbers above are a local floor — find your own ceiling.
To try GraphRAG expansion, enable [graphrag], set search.graph_expansion = true, pair it with a code-aware reranker, and A/B it against your baseline before keeping it on.
Octocode is open source (Apache 2.0) at github.com/Muvon/octocode — benchmark, ground truth, and raw results included. It's the code-search engine behind Octomind, and now its retrieval quality is something you can measure yourself, not just take our word for.
