Benchmarks & Transparency
We publish real production measurements, including the unflattering
ones. Every number ships with its methodology, measurement date, and sample size, produced by
scripts (scripts/intrecall.py) you can re-run. What we don't do: present a score
from a curated dataset and imply it equals your production experience.
Scale is the foundation of any memory product: beyond retrieval quality, start with how much it holds.
The methodology first: this doesn't ask how a retriever performs on a standard question set. It asks something harsher — did the system recall everything it itself wrote? We replay real task topics against the memory IDs written at the time; a hit counts. Then the attribution: case-by-case review of the misses shows the dominant cause is off-topic writes (the task was about X; what got written was a config snippet about Y). The problem is in what gets written, not in finding it — so the fix targets the write gate. We publish this number anyway, because it disciplines write quality.
A correction, same day: an earlier figure on this page (p50 1.2–6.0s, tail up to 41s) came from an out-of-process probe contending with the live service for write locks — an artifact of the measurement method, not the user path. Measured through the actual service API, user-facing retrieval is healthy. We keep the old numbers here as a methodology lesson: measure the user's path through the user's path.
Public benchmarks in the memory space are mostly question-answering over curated transcripts (LoCoMo, LongMemEval — vendors report 92.5% and 71.2% on those). They test "given a fixed conversation history, answer correctly". What we publish is production write–recall self-consistency — different questions, different data. Converting scores across methodologies misleads everyone. To evaluate your own scenario, use the free tier, load your own data, and measure — the scripts and methodology ship with the product.
We re-run every quarter and date each measurement with its sample size; methodology changes are logged on this page, never silently swapped. Next round: recall@5 improvement after the write gate ships, and a larger-sample (≥20 queries) API-path p95.
Methodology and scripts ship with the product:
scripts/intrecall.py (write–recall self-consistency), direct production database
reads (scale). Questions about any number here? Open a feedback thread
requesting a re-computation — especially welcome.