RAG evaluation harness

At ~4 ms per query, evaluating 1,000 queries takes seconds. Use search_many to run support-recall evals and regression suites offline.

The benchmark metric for retrieval is support recall@|gold|: of the passages that actually support the answer, how many did retrieval find? At ~4 ms per query on a CPU, you can run this over a full eval set in seconds — so regression suites and offline evals become cheap enough to run on every change.

The eval set

A list of queries, each with the set of document ids that support the answer:

eval_set.py

EVAL = [
    {"query": "which supplier contract conflicts with the 2026 policy?",
     "gold": {"supplier_a", "policy_2026"}},
    {"query": "which vendor has the earliest renewal date?",
     "gold": {"supplier_c"}},
    # … hundreds more
]

Run it with `search_many`

evaluate.py

from statistics import mean
from rekall import Rekall
from eval_set import EVAL

rekall = Rekall()
store = rekall.store("contracts")

# One batched call — ~4 ms/query, so 1,000 queries run in seconds.
results = store.search_many([row["query"] for row in EVAL])

recalls, latencies = [], []
for row, r in zip(EVAL, results):
    gold = row["gold"]
    retrieved = {p.doc_id for p in r.passages[: len(gold)]}   # recall@|gold|
    recall = len(retrieved & gold) / len(gold)
    recalls.append(recall)
    latencies.append(r.latency_ms)
    if recall < 1.0:
        print(f"MISS  recall={recall:.2f}  conf={r.confidence:.2f}  {row['query']!r}")

print(f"\nsupport recall@|gold|: {mean(recalls):.3f}")
print(f"mean latency:          {mean(latencies):.2f} ms on a CPU")

Output

MISS  recall=0.50  conf=0.44  'which vendors ship internationally?'
…
support recall@|gold|: 0.812
mean latency:          4.30 ms on a CPU

evaluate.ts

import { Rekall } from "rekall-search";
import { EVAL } from "./eval-set";

const rekall = new Rekall();
const store = await rekall.store("contracts");

// Batch with your preferred concurrency; each query is ~4 ms.
const results = await Promise.all(
  EVAL.map((row) => store.search(row.query)),
);

let recallSum = 0, latencySum = 0;
results.forEach((r, i) => {
  const gold = EVAL[i].gold;
  const retrieved = new Set(r.passages.slice(0, gold.size).map((p) => p.docId));
  const hit = [...retrieved].filter((d) => gold.has(d)).length;
  const recall = hit / gold.size;
  recallSum += recall;
  latencySum += r.latencyMs;
  if (recall < 1) console.log(`MISS recall=${recall.toFixed(2)} conf=${r.confidence}`);
});

console.log(`support recall@|gold|: ${(recallSum / results.length).toFixed(3)}`);
console.log(`mean latency: ${(latencySum / results.length).toFixed(2)} ms on a CPU`);

Why this is a killer use case

Fast enough to gate CI. A 1,000-query regression suite finishes in seconds — run it on every PR without a GPU.
The confidence signal is measurable. Log confidence alongside recall to calibrate your LLM-fallback threshold on real data.
hops_used tells you where effort went. Bucket misses by hop count to see whether a capability (multi-hop, contradiction) is regressing.

The eval set

Run it with search_many

Why this is a killer use case

On this page

Run it with `search_many`