# Octocode 0.18.0: A Reproducible Retrieval Benchmark and GraphRAG File-Level Expansion > Octocode 0.18.0 ships a reproducible semantic code search benchmark — 127 curated queries with line-range ground truth, plus a Loc-Bench harness over 560 real GitHub issues — and adds GraphRAG file-level expansion that pulls structurally related files into search before reranking. Open source, local-first, Apache-2.0. [0.17.1 let you run every embedding Octocode generates on hardware you control](/blog/octocode-0-17-1-local-embedding-providers) — any OpenAI-compatible server, any model, your code never leaving your network. That release was about _where_ the search runs. **0.18.0 is about how good the search actually is — and proving it with numbers you can reproduce.** Most code-search tools claim quality. Very few ship a benchmark you can actually run. This release ships two: a curated retrieval benchmark against Octocode's own source, and a [Loc-Bench](https://github.com/czlll/Loc-Bench_V1) harness over 560 real GitHub issues. It also adds **GraphRAG file-level expansion** — a search step that pulls in structurally related files before the reranker scores them. Here's what changed, and what the numbers say — including the parts that didn't work. --- ## Why Benchmark Code Search at All? When you change a chunking heuristic or a fusion weight, you want to know if retrieval got better or worse. Eyeballing a few queries won't tell you — it just confirms what you already believed. The only honest answer is a fixed set of queries, fixed ground truth, and a number that moves. So 0.18.0 ships a real one. It lives in [`benchmark/`](https://github.com/Muvon/octocode/tree/master/benchmark) in the repo, and it measures Octocode's full pipeline end to end: chunking → embedding → vector search → hybrid fusion → reranking. ### The ground truth `code.csv` holds **127 queries**, each annotated with 1–3 correct code locations as line ranges: ``` query,result1,result2,result3 ``` Each result is `src/path/file.rs:start-end:relevance`, where relevance `2` is primary (directly answers the query) and `1` is secondary (useful context). A search result matches by **line-range overlap** — a hit at lines 40–90 matches ground truth 45–92 because the ranges intersect. The corpus is **pinned to a single commit** (`b1771ba`). Ground truth is annotated against those exact line numbers, so indexing any other checkout would drift the ranges and quietly break the scores. Pin first, then index. How the ground truth was built matters, because a sloppy answer key makes any benchmark lie: 1. **AI-generated candidates** across every major module and doc file. 2. **Source verification** — every referenced file and line range was read and checked against the actual code. 3. **Multi-agent validation** — parallel agents independently re-checked line ranges, paths, and relevance grades. 4. **Search-informed corrections** — if search found the same logic in a valid alternate location, that location was _added_ as a secondary result, not deleted. ### The hard queries The last 27 of the 127 are deliberately adversarial: **natural language that doesn't mirror any function name or comment.** They test semantic understanding, not keyword luck. A few: | Query intent | Why it's hard | | -------------------------------------------------- | -------------------------------------------------------- | | Preventing infinite loops in overlapping chunks | 5-line guard clause, no code keywords in the query | | Skipping indexing when the repo is unchanged | buried 100+ lines into a 900-line function | | Rough token estimation without a tokenizer | the answer is a single line: `s.len() / 4` | | Preventing concurrent reindexing in the MCP server | `AtomicBool` + `compare_exchange`, a concurrency pattern | If standard queries score near 1.0 but hard queries lag, the benchmark is doing its job: it's exposing real retrieval weaknesses instead of flattering the tool. ### The metrics Four standard IR metrics, all "higher is better": - **Hit@k** — did _any_ correct result land in the top k? (coverage) - **MRR** — reciprocal rank of the _first_ correct result. (how high is the first hit?) - **NDCG@10** — rewards putting the _most_ relevant results highest, weighting relevance grade and position. (ranking quality) - **Recall@k** — what fraction of _all_ ground-truth blocks were found? (completeness) Definitions and a worked example are in [`benchmark/README.md`](https://github.com/Muvon/octocode/tree/master/benchmark). --- ## The Numbers: A Local, No-Key Floor These results use the **fully local, no-API-key** stack — `jina-embeddings-v2-base-code` via `fastembed`, **no reranker**. That makes them a _floor_, not a ceiling: a paid code-aware reranker pushes higher. Run on Octocode's own source, 127 code queries: | Config | Hit@5 | Hit@10 | MRR | NDCG@10 | Recall@10 | | ------------------------------------- | --------- | --------- | --------- | --------- | --------- | | Dense vector only | 0.598 | 0.717 | 0.485 | 0.528 | 0.671 | | Hybrid, default RRF weights (0.7/0.3) | 0.598 | 0.717 | 0.485 | 0.528 | 0.671 | | **Hybrid, keyword-tuned (0.3/0.7)** | **0.732** | **0.835** | **0.572** | **0.620** | **0.807** | Two things jump out. **The default RRF weights behave like vector-only.** With fusion tilted 0.7 toward the dense signal, the vector results dominate and the keyword signal barely registers — the two top rows are identical. **Tilting toward keyword is a free, large win for code.** Flip the weights to 0.3/0.7 — letting the BM25/keyword signal lead — and Hit@5 jumps **+22%** and Recall@10 **+20%**, at zero added cost. Why? Code is full of exact identifiers — `compare_exchange`, `get_code_blocks_by_path`, `RelationshipDirection` — and exact-token matching carries disproportionate weight when the thing you're searching for _is_ a token. Pure semantic similarity blurs that; keyword fusion sharpens it. That's the kind of result you only find by measuring. It's not intuitive that for code you'd want keyword to _outweigh_ the embedding. --- ## What Didn't Help (And Why That's in the Benchmark) A benchmark that only reports wins is marketing. This one reports the regressions too — the full 6-variant matrix is in [`benchmark/RESULTS.md`](https://github.com/Muvon/octocode/tree/master/benchmark/RESULTS.md). **A generic reranker made things worse.** Dropping a local off-the-shelf cross-encoder (`bge-reranker-base`) on top of the best config didn't help — it _regressed_ Hit@5 from **0.732 back down to 0.598**, undoing the entire keyword-tuning win. And it was roughly **4× slower** (≈1900s vs ≈490s for the run). The lesson isn't "rerankers are bad." It's that **code retrieval needs a code-aware reranker.** A reranker trained on prose re-scores `fn verify_hmac` against "webhook signature validation" worse than the hybrid fusion already did. A code-tuned reranker like `voyage:rerank-2.5` is a different story — but you have to verify that on your own eval, not assume it. The benchmark is what makes that possible. --- ## GraphRAG File-Level Expansion The headline feature alongside the benchmark. Dense and keyword search both score blocks in isolation — they don't know that the file you want _imports_ the file they found, or _calls_ a function defined next door. **GraphRAG file-level expansion** closes that gap. After the initial hybrid search produces ranked candidates, Octocode takes the strongest few, walks the GraphRAG graph for files **structurally related by imports, calls, or extends**, pulls a handful of blocks from those neighbor files, and hands the _enlarged_ candidate set to the reranker: ```rust // Seeds are already RRF-ranked; only expand from the strongest few. const SEED_LIMIT: usize = 10; // expand from top 10 hits const MAX_NEIGHBOR_FILES: usize = 8; // pull at most 8 related files const BLOCKS_PER_FILE: usize = 3; // 3 blocks per neighbor ``` Neighbors are scored by `weight × confidence` across all seed files, strongest links first. It's file-granular because the graph is file-granular (`CodeNode.id == relative path`). And it's **best-effort**: any graph error degrades to passing the original results straight through — it never breaks search. Turn it on in `config.toml`: ```toml [search] graph_expansion = false # default; requires [graphrag] enabled ``` It ships **off by default**, on purpose. Here's the honest part from the benchmark: > `+graph` = GraphRAG file-level expansion, which **only moves results once a reranker re-scores the enlarged set.** Without a reranker, expansion adds candidates but doesn't change the ranking — so `hybrid_30_70` and `hybrid_30_70+graph` score identically in the local, no-key matrix above. Expansion's payoff comes when a (code-aware) reranker re-scores the bigger pool and promotes a structurally related file that raw similarity ranked too low. Naive expansion can also _dilute_ precision. So the guidance baked into the config comment is blunt: **A/B it on your own eval before trusting it.** The benchmark exists precisely so you can do that. --- ## Loc-Bench: Issue → Code, Nothing Self-Annotated The curated benchmark is annotated by us. A fair question: are the queries _too_ friendly to our own tool? 0.18.0 ships a second harness that removes us from the loop entirely. [`benchmark/locbench.py`](https://github.com/Muvon/octocode/tree/master/benchmark) evaluates Octocode on **Loc-Bench V1** (`czlll/Loc-Bench_V1`): **560 real GitHub issues** from popular Python repos. The query is the _original issue text_. The ground truth is the set of files and functions the _real merged fix_ actually edited (the dataset's curated `edit_functions` field). Nothing is self-annotated — it's the industry "given a bug report, find the code to change" task. Per instance, the harness clones the repo (blobless, cached), checks out the base commit, runs `octocode index`, searches with the issue text, and scores the retrieved files against the real fix — at both **file level** (hit@k, recall@k, MRR, accuracy) and **hunk level** (did a retrieved result overlap the lines the patch actually touched?), broken down by issue category: ```bash python3 benchmark/locbench.py --limit 10 --verbose # smoke test python3 benchmark/locbench.py # full 560 python3 benchmark/locbench.py --category "Bug Report" ``` This is a measurement tool, not a leaderboard claim — it's there so you can run real-world issue localization on your own hardware and models, and so every future change to Octocode can be measured against a benchmark we didn't write. --- ## The Variant Matrix `score.py` measures one configuration. [`run_matrix.py`](https://github.com/Muvon/octocode/tree/master/benchmark) sweeps several over **one shared index** — so the only thing changing between rows is search-time parameters, not the corpus or the embeddings. It writes a comparison table to `RESULTS.md` (and raw `RESULTS.json`), and accumulates across runs so a focused re-run merges into the existing table. ```bash # Pin the corpus to the ground-truth commit first — or the line ranges drift git worktree add /tmp/corpus b1771ba # Full local matrix (fastembed, no API key): vector-only vs hybrid weight sweep CORPUS=/tmp/corpus python3 benchmark/run_matrix.py # Add a local cross-encoder reranker (still no key) — reuses the same index CORPUS=/tmp/corpus SKIP_INDEX=1 ONLY=rerank \ RERANK_MODEL=fastembed:bge-reranker-base python3 benchmark/run_matrix.py ``` Variants: `vector_only`, `hybrid_70_30` (default weights), `hybrid_30_70` (keyword-tilted), `+graph` (GraphRAG expansion), and `+rerank` (cross-encoder). If you set `VOYAGE_API_KEY`, the paid code-aware reranker variants enable automatically — that's how you find the real ceiling above the local floor. --- ## Everything Else **MCP registry metadata and server schema updated.** Keeps Octocode's published MCP server descriptor in step with the actual tool surface, so registries and clients see the right schema. Plus the usual dependency and tooling housekeeping — no behavior change. --- ## Upgrade ```bash # Homebrew brew upgrade muvon/tap/octocode # Universal installer curl -fsSL https://raw.githubusercontent.com/Muvon/octocode/master/install.sh | sh # Cargo cargo install octocode --version 0.18.0 ``` Nothing in this release changes your index or config — upgrading is safe, and your existing setup keeps working. Two things to try: 1. **Tilt your fusion weights toward keyword.** If you search code more than prose, the benchmark says keyword-led RRF is a free accuracy win. Measure it on your repo. 2. **Run the benchmark.** Clone Octocode, pin the corpus, and run `run_matrix.py` against your own embedding model and reranker. The numbers above are a local floor — find your own ceiling. To try GraphRAG expansion, enable `[graphrag]`, set `search.graph_expansion = true`, pair it with a code-aware reranker, and A/B it against your baseline before keeping it on. --- Octocode is open source (Apache 2.0) at [github.com/Muvon/octocode](https://github.com/Muvon/octocode) — benchmark, ground truth, and raw results included. It's the code-search engine behind [Octomind](https://octomind.run), and now its retrieval quality is something you can measure yourself, not just take our word for.