netclaw-dev/netclaw/netclaw-memory

Netclaw Memory

Read this before using any memory tool. It defines how memory works and when to use each tool.

Audience and Feature Gating

Memory is subject to two independent gates:

• Audience gate: Public sessions have no access to memory tools, automatic

recall, or memory extraction. Memory is fully inert for Public — no reads, writes, or recall. Historical memories authored by Public sessions are also excluded from recall and search for all audiences.

• Deployment gate: Memory.Enabled in netclaw.json (default true).

When false, memory is disabled for ALL audiences — recall returns empty, memory tools are hidden from discovery, and the observation sidecar skips extraction.

Both gates must pass for memory to function.

How Memory Works

• Automatic recall runs before each user turn and injects relevant

durable_fact (and occasionally evidence) memories into the conversation.

• Recall is selective by design: candidates must clear a relevance floor

and a per-turn character budget, so many turns inject nothing at all. An absent [memory-recall] block means nothing relevant cleared the bar — this is the normal, healthy outcome for most turns, not a malfunction and not evidence that memory is broken. Never tell the user "my memory isn't working" just because a turn had no [memory-recall] block. Use find_memories when you believe relevant memories exist that automatic recall did not surface.

• Recall is policy-aware: audience and boundary still govern what

can be surfaced for the current turn.

• Recall resolves once at turn start and the same bundle is reused during

tool-loop follow-ups.

• Recalled memories may persist into session history for ongoing context, so

per-turn policy is first-contact gating, not a way to retroactively scrub information already surfaced earlier in the session.

• Explicit tools are a manual-control layer on top of automatic recall.
• Memory is SQLite-backed and cross-session only within the active

domain/boundary policy envelope.

• Duplicate detection is semantic when embeddings are enabled

(Memory.Embeddings.Enabled): a near-duplicate proposal is nominated by embedding similarity and adjudicated by the curator LLM (skip, update, consolidate, or create) — similarity alone never merges or skips anything. Merges are lossless-or-append: the curator writes a merged body that keeps every source fact, and a deterministic guard falls back to appending the proposal instead of overwriting when that check fails.

• Memory IDs shown by automatic recall, find_memories, and get_memories

(e.g. doc-… / rec-…) are stable, opaque handles. Copy them verbatim into get_memories or update_memory — do not rewrite or reformat them.

Hybrid Recall (semantic + lexical)

When Memory.Embeddings.Enabled is true, automatic recall is hybrid: candidates come from the union of full-text search (FTS5) and vector nearest-neighbor search, then a single fused ranking decides what (if anything) gets injected. When embeddings are disabled, recall is lexical-only — same candidate pool, no vector term or cosine floor.

• Fusion: each candidate's score is `VectorWeight × cosine similarity +

LexicalWeight × squashed lexical score`, class-prior adjusted, then recency-decayed (a half-life multiplier that favors fresher memories among otherwise similar candidates but never zeroes out an old one on age alone).

• Query prefix is automatic, per-model: the turn query is embedded using

whatever retrieval-query encoding the active embedding model documents — for the shipped default snowflake-arctic-embed-m-int8 (and the allowlisted fp32 snowflake-arctic-embed-m it's quantized from), that means a fixed instruction string is prepended before the query text. This is a property of the model, not something you configure; document-side embeddings (stored memories) are never prefixed, so this never requires re-embedding existing content.

• Absolute floor: independent of the fused score, any candidate whose raw

cosine similarity falls below the effective MinCosineSimilarity is dropped before ranking. If nothing clears the floor, nothing is injected — this is a correct, healthy outcome, not degraded recall. See the zero-injection note above: don't editorialize about memory being broken when this happens.

• The floor follows the active model's manifest by default.

Memory.Recall.MinCosineSimilarity (nullable) is null unless an operator explicitly overrides it — when null, the effective floor is whichever calibration is pinned to the currently active embedding model (0.24 for the shipped default snowflake-arctic-embed-m-int8 prefixed encoding; also 0.24 for the allowlisted fp32 snowflake-arctic-embed-m prefixed encoding, calibrated independently — int8 measured as a strict retrieval-quality improvement over fp32 on the same gold sets, not a size/latency tradeoff). The numeric meaning of this value is model- and encoding-specific: cosine distributions shift materially between models, and even for the same model between a prefixed and unprefixed encoding — an old value copied from a different model/encoding combination can silently break recall (measured: F0.5 = 0.0 when the pre-prefix 0.68 floor was applied to prefixed queries). Only set an explicit override after re-running the calibration-verification procedure against the model and encoding actually active. Memory.Recall.VectorWeight defaults 0.7, LexicalWeight 0.3, RecencyHalfLifeDays 30.

• Degradation is explicit and logged, not silent: a turn whose

query-embedding step misses its latency sub-budget (or has no embedder available) falls back to lexical-only scoring for that turn and logs memory_recall_vector_degraded. A model with no manifest-carried retrieval calibration and no explicit MinCosineSimilarity override degrades the same way, with reason missing_calibration — this is expected for a newly-added or not-yet-calibrated model variant, not a bug. A document candidate with no embedding row for the current model degrades to lexical-only scoring for that candidate alone (rather than being excluded) and logs memory_recall_coverage_gap. Records have no embeddings by design. They retain lexical admission and do not count as coverage gaps. All of these are self-healing or intentional, not persistent failures — see Diagnostics below.

• Backfilling an existing corpus: enabling Memory.Embeddings.Enabled

on a deployment that already has memories does not retroactively embed them. Until they're embedded, recall for those documents degrades to lexical scoring and memory_recall_coverage_gap keeps firing. Operators should run netclaw memory backfill-embeddings right after turning embeddings on so the gap closes immediately instead of waiting for embed-on-write to catch up opportunistically.

• **Upgrading onto a new default model id (e.g. the fp32→int8 default

flip)**: an existing install with vectors stored under the previous Memory.Embeddings.ModelId self-heals automatically — vector coverage, the curation nominator, and hybrid recall are all scoped to the current model id, so the daemon's startup gap-repair sweep sees every document as missing a current-model embedding and re-embeds the whole corpus under the new id with no operator action required. The old model's vectors are never deleted, just no longer read. Until gap repair finishes, recall degrades to lexical-only (same self-healing, logged degradation as any other coverage gap above), and netclaw doctor surfaces the interim mixed-model state as a warning recommending netclaw memory backfill-embeddings --force to force it immediately instead of waiting.

Relevance Gate (cross-encoder)

The cosine floor above answers "is this candidate on-topic?" — it does not answer "does this candidate actually help answer the question?" A second stage, the relevance gate, runs after the floor for exactly this reason: a tiny cross-encoder (ms-marco-minilm-l-6-v2) jointly scores (query, candidate) for each of the (≤3) floor survivors and drops anything below its calibrated threshold.

• Activation follows `Memory.Embeddings.Enabled` — one mental switch, no

second thing to discover. Memory.Recall.RelevanceGate.Enabled (nullable) is an explicit override for an operator who wants embeddings for dedup/hybrid-recall but not the extra per-turn cross-encoder latency; Memory.Recall.RelevanceGate.Threshold (nullable) is an explicit override of the manifest's calibrated operating point. Leave both null unless you have a specific reason to diverge — the manifest-carried default is what was validated out-of-sample.

• Only ever runs in hybrid mode, on the floor's own survivors — it never

sees a wider candidate pool and never runs when recall has already degraded to lexical-only.

• Zero survivors after the gate is a healthy outcome, identical in kind

to zero survivors at the floor: the [memory-recall] block is omitted entirely, not emitted empty. Do not treat an absent recall block as evidence the gate (or memory generally) is broken — see the zero-injection note above.

• Degradation is explicit and logged, not silent: when the relevance

model is unavailable, its sub-budget is exceeded, or recall is running in lexical mode, the gate step is skipped and the floor's own result is injected unfiltered — the exact pre-gate behavior. This fires memory_recall_gate_degraded (rate-limited, same cooldown pattern as memory_recall_vector_degraded). A degraded gate never silently changes what gets injected without this marker.

When to Use Explicit Tools

find_memories + get_memories

Use when:

• The user explicitly asks what you remember
• Automatic recall seems insufficient for the question
• You need targeted retrieval beyond the injected bundle

Pattern: find_memories("query") -> scan results -> get_memories("id1,id2")

Normal find_memories behavior:

• searches durable_fact plus current evidence
• excludes trace
• hides expired evidence by default
• respects the current turn's effective audience and boundary

store_memory

Use only for deliberate save requests:

• User explicitly says "remember this" or "save this for later"
• Pinning a high-value fact, decision, or preference

Do NOT call store_memory reflexively on routine turns - the observation sidecar handles background memory formation automatically.

Policy rules for explicit writes:

• explicit writes still inherit the current turn's audience and boundary
• explicit writes may narrow policy scope, but must never widen it
• raw secrets, credentials, tokens, and private keys are never durable memory

Automatic observation note:

• a non-empty adopted thread window still counts as adopted context for audit

and approval provenance

• automatic memory suppression only kicks in when the adopted window includes a

sender other than the current authorized author

• self-only adopted history does not suppress automatic memory formation by

itself

update_memory

Use only to correct or supersede an existing memory.

Use the memory ID exactly as shown by automatic recall, find_memories, or get_memories. For documents, prefer new_content when replacing a full hydrated memory. Use old_text + new_text only when making a precise find-and-replace edit. To delete a memory, pass delete: true.

Memory Classes

Policy Envelope