LLM fallback on low confidence

Take the 4 ms path for most queries; escalate to an LLM only when Rekall's confidence says the question is hard. The core composability pattern.

Rekall returns a calibrated confidence — the sufficiency head's answer to "did I find enough?" That signal makes Rekall trivially composable: run the 4 ms path on every query, and spend LLM tokens only on the rare questions that need them.

Search with Rekall

One reasoning-loop query, ~4 ms on a CPU. You get passages, confidence, hops_used.

Branch on confidence

If confidence >= threshold, use the passages directly. Most queries stop here.

Escalate only when unsure

Below the threshold, fall back to a slower, more expensive agentic/LLM path.

The pattern

fallback.ts

import { Rekall } from "rekall-search";

const rekall = new Rekall();
const store = await rekall.store("contracts");

const THRESHOLD = 0.6;

async function answer(query: string) {
  const r = await store.search(query, { trace: true });
  console.log(`confidence=${r.confidence} hops=${r.hopsUsed} ${r.latencyMs}ms`);

  if (r.confidence >= THRESHOLD) {
    return r.passages;                 // 4 ms path — most queries end here
  }

  // Rare fallback: a slower agentic/LLM loop over the same store.
  return await expensiveAgenticSearch(query);
}

async function expensiveAgenticSearch(query: string) {
  // e.g. call an LLM with retrieved context; seconds, GPU/token cost.
  // const msg = await anthropic.messages.create({ model: "claude-opus-4-8", ... });
  return [];
}

fallback.py

from rekall import Rekall

rekall = Rekall()
store = rekall.store("contracts")

THRESHOLD = 0.6

def answer(query: str):
    r = store.search(query, trace=True)
    print(f"confidence={r.confidence} hops={r.hops_used} {r.latency_ms}ms")

    if r.confidence >= THRESHOLD:
        return r.passages                 # 4 ms path — most queries end here

    # Rare fallback: a slower agentic/LLM loop over the same store.
    return expensive_agentic_search(query)


def expensive_agentic_search(query: str):
    # e.g. call an LLM with retrieved context; seconds, GPU/token cost.
    # msg = anthropic.messages.create(model="claude-opus-4-8", ...)
    return []

Choosing a threshold

0.6 is a sensible default: it captures the queries where the reasoning loop found a coherent, sufficient answer.
Lower it (e.g. 0.4) if false escalations are cheap and you want maximum recall from the fast path.
Raise it if a wrong fast-path answer is costly and you'd rather pay for the LLM.
Log confidence, hops_used, and latency_ms across a sample of real traffic, then set the threshold where the fast path's precision meets your bar.

Why this works

Because most queries clear the threshold, your average latency and cost stay at commodity-search levels — you only pay agentic prices on the tail. That's the whole point of a first-class confidence signal.