Benchmarks

Rekall is benchmarked on a 25-capability synthetic reasoning benchmark. We lead with per-capability results, named against their baselines, because that is where the real story is: reasoning-shaped tasks at millisecond latency on a CPU.

Methodology

• Suite: 25-capability synthetic reasoning benchmark — 287 documents, 300 questions, a single shared store.
• Metric: support recall@|gold| — did the method retrieve the passages that actually support the answer?
• Baselines:
  • BM25 — lexical keyword search.
  • Embedding (8B) — Qwen3-Embedding-8B, an 8B-parameter embedding model.
  • Agentic (9B) — a 3-round agentic loop over a 9B LLM (Qwen3.5-9B).
• SKIM checkpoints: the hierarchical variant here is an early checkpoint (1B training tokens, still training); the transformer variant is more trained (5B tokens).
• Latency is CPU, engine-only, measured server-side. Agentic and embedding baselines run on a GPU.

Headline — macro across 25 capabilities

Agentic (Qwen 9B)

0.80

2,969 ms

SKIM transformer (5B)

0.69

4.4 ms

Embedding (Qwen 8B)

0.60

~ms

SKIM hierarchical (1B)

0.58

~4 ms

BM25

0.52

~ms

SKIM runs ~670× faster than the agentic loop — measured 4.4 ms vs 2,969 ms, on a CPU. On quality, the transformer variant reaches up to 87% of the agentic reasoning score (0.693 / 0.798) at 0.15% of the latency; the hierarchical checkpoint — still early in training — is at 72% (0.577 / 0.798), and already beats BM25 while approaching the 8B embedding model at a fraction of its size.

Where reasoning wins

On the tasks that break similarity search, SKIM matches or beats models far larger than itself. Every column names its baseline.

The single clearest result is contradiction detection: SKIM scores 1.00 where BM25, the 8B embedding model, and the 9B agentic loop all score 0.50. This is the thesis in one number — retrieval that reasons finds things similarity cannot.

BM25

0.50

Embed 8B

0.50

Agentic 9B

0.50

SKIM

1

BM25

0.04

Embed 8B

0.46

Agentic 9B

0.81

SKIM

0.71

Where it still lags

Honesty matters more than a clean chart. On structural and lexical tasks, the pure embedder's raw lexical match currently wins:

These improve as training continues — the base is at 2.3B tokens and rising, with reruns at 3B / 4B / 5B in progress. The hierarchical checkpoint here is early; the numbers will move.

Long-document ingestion

Whole-document, unchunked ingestion is practical because SKIM's hierarchical attention is ~O(L), not O(L²). Measured against dense attention at 4,000 tokens: 3.4× faster and 63× less attention memory. Chunking strategy — the fiddliest part of every RAG pipeline — disappears.

Reproduction

The methodology, corpus, and reproduction scripts are published. The benchmark table generator is build_table.py; the full method and comparison live in SKIM_COMPARISON.md. Model artifacts are in DO Spaces under skim_search_loop/{bases,eval_models}/.

Fine print

Synthetic reasoning benchmark; SKIM is an early checkpoint still in training — numbers will move. Metric: support recall@|gold|. Full methodology, corpus, and reproduction scripts published. We lead with per-capability results because aggregate claims hide more than they show.