eliasoulkadi/shokunin/chromadb

chromadb · Memory Storage

ChromaDB is the vector database that powers Shokunin's persistent AI memory. It stores embeddings for semantic search across sessions. Built on SQLite with an ONNX embedding model for local-first, offline-capable operation.

Workflow

Step 1: Check storage status

python ~/.shokunin/scripts/chroma-helper.py count

Output: total entries, collection count, disk size on disk, last write timestamp.

python ~/.shokunin/scripts/chroma-helper.py stats

Output: entries per type (decision, file, command, checkpoint, session_end), per project, per date range.

Step 2: Search stored entries

python ~/.shokunin/scripts/chroma-helper.py search "query text" "project-name" 10

Searches via vector similarity + BM25 keyword hybrid. Results sorted by relevance with metadata attached.

python ~/.shokunin/scripts/chroma-helper.py search "error handling" "" 20 --type decision

Filter by entry type. Leave project empty to search all projects.

Step 3: List recent entries

python ~/.shokunin/scripts/chroma-helper.py recent 10

Shows the 10 most recently inserted entries across all projects. Use for quick audit.

python ~/.shokunin/scripts/chroma-helper.py recent 50 --project myproject

Filter recent entries to a specific project.

Step 4: Backup the database

$backupDir = "$env:USERPROFILE\.shokunin\backups\chroma-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
New-Item -ItemType Directory -Path $backupDir -Force | Out-Null
Compress-Archive -Path "$env:USERPROFILE\.shokunin\memory\chroma_db" -DestinationPath "$backupDir\chroma_db.zip" -Force

Also back up session logs:

Compress-Archive -Path "$env:USERPROFILE\.shokunin\memory\sessions" -DestinationPath "$backupDir\sessions.zip" -Force

Verify backup integrity:

Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::OpenRead("$backupDir\chroma_db.zip").Entries.Count

Step 5: Delete entries by session prefix

python ~/.shokunin/scripts/chroma-helper.py delete "session-prefix"

Deletes all entries whose session_id starts with the given prefix. Non-reversible. Backup first.

python ~/.shokunin/scripts/chroma-helper.py delete --project "old-project"

Delete all entries for a project. Useful when archiving finished projects.

Step 6: Full reset (requires confirmation)

Remove-Item -Recurse -Force "$env:USERPROFILE\.shokunin\memory\chroma_db"
# Restart MCP server to recreate the database

The database is auto-created on next MCP server startup. All sessions, embeddings, and metadata are permanently lost. Only do this when migrating or recovering from corruption.

Storage Details

Data Model

Session Management

Error Handling

Anti-Patterns

Automation

Weekly backup every Sunday via Windows Task Scheduler (pre-configured). Manual backup via Step 4 above. Logs auto-rotate at 5MB to prevent unbounded growth. The consolidate_memories function runs automatically when entry count exceeds 10,000 per project.

Recovery Procedures

Restore from backup

$backup = Get-ChildItem "$env:USERPROFILE\.shokunin\backups" -Directory | Sort-Object LastWriteTime -Descending | Select-Object -First 1
Expand-Archive -Path "$backup\chroma_db.zip" -DestinationPath "$env:USERPROFILE\.shokunin\memory\chroma_db" -Force

Rebuild from session logs

If backups are unavailable, session JSONL logs contain all raw text. Replay them:

python ~/.shokunin/scripts/chroma-helper.py rebuild --from-sessions

Verify database health

python ~/.shokunin/scripts/chroma-helper.py verify

Checks: SQLite integrity, embedding count vs document count, orphan metadata, collection schema version.

Checklist

• [ ] ChromaDB client initialized lazily (not at import time)
• [ ] Collection name sanitized (alphanumeric + hyphens only)
• [ ] Embedding dimension matches between writes and queries
• [ ] Backups configured before destructive operations (delete/reset)
• [ ] Search results validated — empty results handled gracefully

Sources

• ChromaDB documentation (docs.trychroma.com)
• OpenAI text-embedding-ada-002 (or local ONNX embedder)
• SQLite documentation (sqlite.org)
• DuckDB (ChromaDB's internal query engine)
• MCP Protocol (modelcontextprotocol.io)
• Shokunin Memory System Architecture (ARCHITECTURE.md)