Evaluating recall quality
Run repeatable recall tests to check that the right memories still come back after you change settings or add data.
When you change settings (embedding provider, score weights, recall mode) or add a lot of data to memory, you want to be sure the right memories still come back. Agent Memory provides a repeatable recall test for this: queries, and for each query the memories that should (and should not) be returned.
From the console
On the Evaluations page you create and save test sets, run them, and compare runs with earlier results. Tests run with the console's playground identity.
From the API
curl -s -X POST $API/v1/memory/evals/run -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
-d "{
\"workspaceId\": \"$WS\",
\"agentId\": \"<key's agent>\",
\"k\": 10,
\"datasetVersion\": \"billing-v1\",
\"cases\": [
{\"query\": \"how are invoice numbers generated\",
\"expectedIds\": [\"<expected-memory-id>\"],
\"forbiddenIds\": [\"<id-that-must-not-appear>\"],
\"staleIds\": []}
]
}"| Field | Description |
|---|---|
cases | 1–200 tests. Each test contains a query, expected IDs (expectedIds), and optionally forbidden (forbiddenIds) and stale (staleIds) IDs. |
k | How many top results to look at (default 10, max 100). |
agentId | Required. Whose visibility to search with; must be an agent your key is bound to (for keys with no bound agents, the key's own identity apikey:<keyId>). |
datasetVersion | A label for comparing results. |
Response:
| Metric | Meaning |
|---|---|
hitRate, recallAtK, precisionAtK, mrr, ndcgAtK | Classic retrieval metrics (with confidence intervals, intervals). |
forbiddenRate, forbiddenSurfaced | Did a forbidden memory come back? A single forbidden result fails the test (passed: false). |
staleRate, zeroResultRate, abstentionAccuracy | Stale memories, empty results, and expected-empty behavior. |
latencyMs | p50 / p95 / p99. |
meanContextTokens, llmCalls | Context size and number of model calls. |
manifest | Comparability key: dataset, k, score version, mode, embedding generation, code version. |
perCase | Details for each query. |
Evaluation reads run in deterministic mode and leave no trace: usage counters are not incremented, no receipts are written, and no learning credit is given. A run counts against your quota as one operation.
A good test set
- Sample real agent queries (the console's Requests page helps).
- For each query, put at least one correct memory in
expectedIds, and anything that must never appear (e.g. another user's private memory, a superseded old decision) inforbiddenIds. - Run with the same
datasetVersionbefore and after a configuration change; if themanifestdiffers, don't compare the results directly.
Out of scope
This evaluation measures retrieval quality; it does not measure the correctness of the agent's answer (with an LLM judge). Separate research scripts for long-conversation memory benchmarks exist in the repository; their results do not reflect the current default settings.