Vector search ranks by similarity. Hybrid search ranks by similarity plus keyword overlap. But the thing you actually want out of code search is relevance — and similarity is not relevance. The snippet that shares the most tokens with your query is often not the code that answers it.
Ask "where do we decide a request is retryable" and a similarity ranker will happily hand you the retry config struct, the retry constant, and the test that names the word four times, while the one function that makes the call sits at rank seven. Every token matched. Nothing answered.
PageIndex made this point for documents: replace top-k similarity with an LLM that reasons about which sections are relevant. We wanted the same for code, where the structure is richer than a PDF's table of contents — you have symbols, a call graph, real bodies.
So we built it: an optional reasoning step that sits after hybrid retrieval, reads the candidate code, and re-ranks by whether it actually answers the query. It's off by default, gated behind one config flag, and it leaves structural_search as the plain deterministic grep it should be. On our benchmark it is a large, across-the-board win. This post is the honest write-up — the numbers, how we tuned it, and the two things that did not work.
The core idea, and the one trick that made it work
The pipeline is short. A query goes through hybrid retrieval exactly as before — vector similarity plus keyword matching over the local index. The top candidates come back with their full code bodies. An LLM reads them and ranks them by whether they answer the question. Then the two rankings are fused and the top results are returned. No new index, no second embedding pass, no reindexing.
The naive version of that middle step is obvious: take the candidates, ask "which of these answer the query, ranked," use that order. We did that first. It was a disaster for recall.
Here is the pure-reasoning reranker on 127 queries:
| metric | hybrid only | pure reasoning | Δ |
|---|---|---|---|
| MRR | 0.595 | 0.752 | +0.157 |
| NDCG@10 | 0.658 | 0.758 | +0.100 |
| Hit@10 | 0.913 | 0.843 | −0.071 |
| Recall@10 | 0.886 | 0.811 | −0.075 |
The ranking metrics jumped — the correct answer got pulled to the top. But Hit@10 and Recall@10 dropped, because the model prunes: it returns the handful it judges relevant and quietly discards the rest, including true positives that were sitting at ranks 6–10. A reranker that improves precision by throwing away recall is not a reranker you ship. It looks brilliant on the queries it gets right and it makes the hard ones unanswerable.
The fix is small and it's the whole reason this works: don't let the LLM replace the ranking — fuse it with the hybrid ranking via Reciprocal Rank Fusion.
score(candidate) = 1 / (k + hybrid_rank)
+ reasoning_weight * 1 / (k + reasoning_rank)
k is the usual RRF damping constant that stops any single rank position from dominating the sum. The important part is what each term does. The hybrid rank always contributes, so it acts as a recall floor — an LLM-demoted or omitted true positive is knocked down, not deleted. The reasoning rank, weighted, drives the head of the list. You keep the ranking gains and stop bleeding recall.
That one change turned the pure-reasoning trade-off into a clean win on every metric.
The numbers
Benchmark: 127 hand-annotated queries against the Octocode codebase pinned at a fixed commit (100 standard + 27 deliberately-hard queries that use natural language with no keyword overlap). Local fastembed embeddings, hybrid search on, same queries on both sides, deepseek:deepseek-v4-flash as the reasoning model. Ground truth is line-range overlap against verified source locations.
| metric | hybrid only | + reasoning | Δ |
|---|---|---|---|
| MRR | 0.595 | 0.809 | +0.214 (+36%) |
| NDCG@10 | 0.658 | 0.833 | +0.175 (+27%) |
| Hit@5 | 0.827 | 0.953 | +0.126 |
| Hit@10 | 0.913 | 0.969 | +0.055 |
| Recall@5 | 0.777 | 0.924 | +0.147 |
| Recall@10 | 0.886 | 0.944 | +0.058 |
Every metric up. The two that matter most for "find me the code that does X" — MRR (how high the first correct result lands) and NDCG@10 (whether the most relevant results rank highest) — moved 36% and 27%. Hit@5 went from 0.83 to 0.95: nineteen times out of twenty, the answer is now in the top five.
That last number is the one that changes how an agent behaves. When the answer is reliably in the top five, the agent reads five results and acts. When it isn't, the agent widens the search, reads more files, burns context on candidates that were never relevant, and sometimes gives up and greps. Ranking quality upstream is context saved downstream.
How we tuned it (and where more was worse)
We didn't guess the config. We swept it, on a shared index, one dimension at a time.
Reasoning weight — how hard the fusion leans on the LLM versus the hybrid floor. Swept 1, 2, 3, 5. Ranking climbs with weight and plateaus around 2–3; 5 buys nothing and costs a little recall, because at that point the hybrid floor stops mattering and you're back to trusting the model alone. On the full benchmark, 2.0 was the best all-rounder.
How many candidates to reason over. Swept 15, 25, 35, 40. More is not better — 40 was measurably worse than 25. Feeding the model more candidates dilutes the decision, and it costs more tokens to get the worse result. 25 is the sweet spot.
How much of each candidate the model sees. This one was decisive. Signatures only, a snippet, or the full body:
| context | Hit@5 | MRR | Recall@5 |
|---|---|---|---|
| signatures | 0.933 | 0.806 | 0.900 |
| snippets | 0.933 | 0.849 | 0.883 |
| full body | 1.000 | 0.857 | 0.967 |
(30-query subset; absolute numbers run higher on the subset, but the ordering is the point.)
Full code bodies win by a lot. The model needs to see the actual code to judge relevance — names and signatures are not enough. Signatures-only was the worst option, which is worth sitting with: it's the cheapest option and the one you'd reach for to save tokens, and it throws away exactly the information the reasoning step exists to use.
LLM temperature. We assumed lower (more deterministic) would rank better. Wrong — temperature 1.0 beat 0.3 and 0.0. The model reasons better with normal sampling.
What did not work: index-time contextual descriptions
The obvious way to push recall higher is to enrich the index: have an LLM write a one-line "what this code does" for every chunk and embed that alongside the code (Anthropic's contextual retrieval reported large drops in failed retrievals doing exactly this). We had the machinery already, so we tested it — full re-index with descriptions, then the same A/B.
It made things slightly worse:
| Hit@5 | MRR | NDCG@10 | Recall@10 | |
|---|---|---|---|---|
| plain index (+reasoning) | 0.953 | 0.809 | 0.833 | 0.944 |
| contextual index (+reasoning) | 0.945 | 0.836 | 0.859 | 0.925 |
| Δ | −0.008 | +0.027 | +0.026 | −0.019 |
It trades recall for ranking: MRR and NDCG tick up, but Hit and Recall tick down. Prepending an LLM description dilutes the code's own tokens in the embedding, and on clean, well-structured code with a decent code embedder that drifts recall down. The recall lift Anthropic measured was on prose and mixed corpora — not this. And the ranking value contextual would add? The reasoning step already captures it, which makes contextual redundant-to-harmful stacked on top.
So we didn't ship it. It stays off, and the cost of that decision is worth naming: a contextual index means an LLM call per chunk at index time, on every reindex, for a result that measured worse. Worth stating plainly because "add contextual retrieval" is cargo-culted everywhere — it is not a universal win, and on code it can cost you.
How to turn it on
Reasoning is off by default (it costs one LLM call per search). Enable it in config.toml:
[search.reasoning]
enabled = true
model = "deepseek:deepseek-v4-flash" # any provider:model
max_candidates = 25 # 25 beat 40 — more dilutes
context_level = "full" # full body wins; snippets/signatures are worse
reasoning_weight = 2.0 # RRF weight vs the hybrid recall floor
final_top_k = 10
Every parameter is defined in the template — strict config, no hidden defaults in code. deepseek-v4-flash is cheap and fast; any provider works, and the numbers above are what a cheap model gets you, not a frontier one. It runs only in the semantic search path — structural_search stays pure grep, no AI, as it should. If you want deterministic, zero-cost search, leave the flag off and nothing about your existing setup changes.
The honest ceiling
We're at Hit@5 0.953. Getting to 1.0 is not realistic on this benchmark — the last 27 queries are deliberately adversarial, and some ground truth is genuinely ambiguous. The remaining gap is not pool recall (the contextual test confirmed the candidate pool is already well-covered); it's genuinely hard queries. A stronger base embedder in production (a purpose-built code embedding model over the local fastembed default) would raise the floor further, but the reasoning approach itself is at its practical best here.
What we like about this result: it's a real synthesis. Hybrid search gives you recall, cheaply and deterministically. Reasoning gives you relevance, where you already spend on an LLM. RRF fuses them so you don't have to choose. No new index, no vectorless rewrite, no reindexing — one flag, one LLM call, and the correct code goes to the top.
FAQ
What is reasoning retrieval, in one sentence?
An optional step after hybrid retrieval where an LLM reads the candidate code bodies, ranks them by whether they answer the query, and that ranking is fused with the hybrid ranking via Reciprocal Rank Fusion instead of replacing it.
Does it replace vector or hybrid search?
No, and that's the point. Letting the LLM own the ranking cost us 7 points of Recall@10. The hybrid rank stays in the score as a recall floor, so a true positive the model overlooked gets demoted rather than deleted.
What does it cost?
One extra LLM call per semantic search, over at most 25 candidates. It's off by default, and the benchmark numbers were produced with deepseek:deepseek-v4-flash — a cheap, fast model. Any provider:model works if you want to spend more.
Does it slow down or change structural_search?
No. Reasoning runs only in the semantic search path. structural_search stays deterministic grep with no LLM in the loop.
Should I enable contextual descriptions at index time too?
On code, our measurement says no: it improved MRR and NDCG slightly but cost Hit@5 and Recall@10, and the ranking gain is already covered by the reasoning step. It also adds an LLM call per chunk on every reindex.
Do I need to reindex to use it?
No. Reasoning is a query-time step on top of the index you already have. Flip the flag, restart, done.
— Don
Octocode is open source under Apache-2.0 — benchmark, ground truth, and raw results included, so you can rerun these numbers on your own codebase. It's the code-search engine behind Octomind; the MCP server is how they talk.
