Ingestion

Ingestion writes document encodings into a store. It is O(N) in corpus size, incremental, and queryable the moment it returns — there is no index build step. Pass whole documents; SKIM's hierarchical attention reads long documents natively, so chunking never enters the picture.

Ingest a folder

The CLI walks folders and globs and auto-detects formats. The SDKs take documents (or files) directly.

rekall ingest ./docs --store demo

  scanning ./docs … 291 files
  ingesting  ████████████████████  287/291   70 docs/s
✓ 287 documents ingested · queryable now  (4.1 s)
  4 skipped:
    contracts/.DS_Store          unsupported type
    reports/corrupt.pdf          extraction failed (no text layer)
    archive/huge.mbox            41.2 MB > 32 MB limit
    notes/empty.txt              empty document

import { Rekall } from "rekall-search";

const rekall = new Rekall();
const store = await rekall.store("demo");   // get-or-create

// Whole documents, no chunking. Queryable when this resolves.
const { ingested, docIds } = await store.ingest([
  { id: "supplier_a", text: supplierText, metadata: { team: "procurement" } },
  { id: "policy_2026", text: policyText },
]);

console.log(`ingested ${ingested}:`, docIds);

from rekall import Rekall

rekall = Rekall()
store = rekall.store("demo")   # get-or-create

# Whole documents, no chunking. Queryable when this returns.
res = store.ingest([
    {"id": "supplier_a", "text": supplier_text, "metadata": {"team": "procurement"}},
    {"id": "policy_2026", "text": policy_text},
])

print(f"ingested {res.ingested}:", res.doc_ids)

Raw text — no file needed

A document is just {id, text, metadata} — the API and SDKs take raw text natively (as in the tabs above). The CLI equivalent is rekall add:

rekall add "Vendors are reviewed quarterly by the security team." --id vendor-note
echo "billing terms are net thirty" | rekall add --meta topic=billing
rekall add - < meeting-notes.txt

Same upsert semantics as file ingest. Without --id, a stable id is derived from the text, so re-adding identical text updates rather than duplicates.

Formats

Text extraction is built in. The CLI auto-detects; force a format with --format.

rekall ingest ./exports --store demo --format md

The SDK file helpers (ingestFiles / ingest_files) share the exact same extractors as the CLI.

Images and scanned PDFs

The engine ingests text, so non-text media becomes text once at ingest time — search stays milliseconds, nothing multimodal runs at query time. Two providers, tried in order:

1. Vision enrichment (recommended). Point Rekall at any vision-capable, OpenAI-compatible model. The zero-ops path is OpenRouter with a cheap vision model (fractions of a cent per image):

export REKALL_VISION_MODEL=qwen/qwen3.5-flash-02-23
export REKALL_ENRICH_API_KEY=<your OpenRouter key>

rekall ingest ./slides ./charts ./receipts --store media

REKALL_VISION_MODEL rides alongside REKALL_ENRICH_MODEL, so a text-only answer model (say, a DeepSeek) and a vision model for ingestion coexist. If only REKALL_ENRICH_MODEL is set and it accepts images, it handles both. REKALL_ENRICH_URL overrides the default OpenRouter endpoint for self-hosted VLMs.

Self-hosted works identically — Ollama, vLLM, or any endpoint speaking the OpenAI chat API (REKALL_ENRICH_URL=http://localhost:11434/v1, REKALL_ENRICH_MODEL=qwen2.5-vl:7b).

Each image (and each page of a scanned PDF) is transcribed verbatim and then described — so a revenue bar chart becomes findable by "how did revenue develop", not just by its axis labels.

2. Local OCR. With tesseract installed (plus poppler-utils for scanned PDFs), Rekall transcribes locally — text only, no descriptions:

apt install tesseract-ocr poppler-utils   # or brew install tesseract poppler

Scanned PDFs are detected automatically: a .pdf whose text layer is empty is rasterised page-by-page (200 DPI) and read by whichever provider is available. When neither provider is configured, the file is skipped with an error that says exactly which of the two to set up — the rest of the batch continues.

Upsert semantics

Documents are keyed by a caller-chosen id. Re-ingesting the same id replaces the document's encoding in place — no reindex, no tombstones to manage, and the store stays live throughout.

# Re-ingest supplier_a after it changed — old encoding is replaced
rekall ingest ./docs/supplier_a.pdf --store demo

Deleting is symmetric and equally cheap:

rekall docs delete supplier_a --store demo

await store.documents.delete("supplier_a");

store.documents.delete("supplier_a")

Live updates with --watch

--watch re-ingests on file change so documents stay live while you edit. Edit a doc, save, search again — updated answer, no reindex.

rekall ingest ./docs --store demo --watch

✓ 287 documents ingested · watching ./docs for changes …
  ~ policy_2026.md changed → re-ingested (1 doc, 6 ms)

Metadata

Attach flat key–value metadata to any document. Values are strings, numbers, or booleans. Metadata powers the equality filter on search and is echoed back on every returned passage.

await store.ingest([
  { id: "supplier_a", text: supplierText,
    metadata: { source: "supplier_a.pdf", team: "procurement", year: 2026, signed: true } },
]);

store.ingest([
    {"id": "supplier_a", "text": supplier_text,
     "metadata": {"source": "supplier_a.pdf", "team": "procurement", "year": 2026, "signed": True}},
])

Batch limits and streaming big corpora

A single ingest request takes 1–1000 documents and ≤ 32 MB. The SDKs stream larger corpora transparently — batching and backpressuring so you can pipe a million documents through without minding the limits.

async function* documents() {
  for await (const row of readHugeDataset()) {
    yield { id: row.id, text: row.body, metadata: { source: row.path } };
  }
}

await store.ingestStream(documents(), {
  concurrency: 4,
  onProgress: ({ ingested }) => console.log(`ingested ${ingested}…`),
});

def documents():
    for row in read_huge_dataset():
        yield {"id": row.id, "text": row.body, "metadata": {"source": row.path}}

# O(N), backpressure-aware; shows a tqdm bar if tqdm is installed.
store.ingest_stream(documents(), batch_size=256)

Ingest from files

// Node only — extraction parity with the CLI
await store.ingestFiles("./docs/**/*.{pdf,md,txt}");

store.ingest_files("docs/", glob="**/*.pdf")

Next steps