maomao-2001/whispera/mem0-test-integration

mem0-test-integration

Verifies what /mem0-integrate produced. Runs in the same workspace, on the same feature branch. Loose coupling — fast, catches compile and runtime bugs, does not catch logical errors.

Canonical sources (use these, not ambient knowledge)

All static checks and smoke-test shapes validate against these URLs. WebFetch each before running step 3.

• Scope-tagged docs index: https://docs.mem0.ai/llms.txt
• OpenAPI (Platform REST): https://docs.mem0.ai/openapi.json
• Published SDK skill (canonical call patterns): https://raw.githubusercontent.com/mem0ai/mem0/main/skills/mem0/SKILL.md
• Vercel AI SDK skill (if the target repo uses @ai-sdk/*): https://raw.githubusercontent.com/mem0ai/mem0/main/skills/mem0-vercel-ai-sdk/SKILL.md
• SDK source (cross-check version against frontmatter mem0_tested_versions):
• Repo root: https://github.com/mem0ai/mem0
• Python: https://github.com/mem0ai/mem0/tree/main/mem0
• TypeScript: https://github.com/mem0ai/mem0/tree/main/mem0-ts

Read the Delegated skill: field in .mem0-integration/plan.md — if it names a skill URL, fetch that skill and use its example blocks as the reference for both static checks (step 3) and the smoke test (step 5).

Non-invasiveness contract

Every check in this skill assumes the integration is additive and feature-flagged (see /mem0-integrate "Integration principles"). Specifically:

• product.json must contain a feature_flag field.
• Steps 4–6 run in two passes:
• Pass A — flag unset. All pre-existing tests must pass, smoke/E2E

skip. The repo must behave like main. Any failure here is a hard fail — do not let the self-heal loop attempt a patch.

• Pass B — flag set. New tests must pass, smoke and E2E run.
• If Pass A fails, the scorecard marks non_invasive: false and sets

overall: fail with a distinct reason code the integrator's heal loop refuses to touch.

Preconditions

Refuse to start unless ALL of the following are true:

• .mem0-integration/ directory exists in the repo root.
• .mem0-integration/product.json, goal.md, and plan.md are readable

and internally consistent (JSON parses, docs non-empty).

• Current branch name begins with mem0-integrate/ (set by the companion

skill). Prevents accidental runs on unrelated branches.

• Working tree is clean. The skill never modifies source files; any dirty

state means the integration is mid-edit and not ready to verify.

• The same API key the integration used is available in the environment

(MEM0_API_KEY for Platform, OPENAI_API_KEY for OSS — read which from product.json). Interactive mode asks if missing; CI mode exits 2.

Exit with a written rationale on any precondition failure. Never attempt to "fix up" state.

Pipeline

1. Read the contract

Load:

• product.json → which language, which product (Platform vs OSS), which

mem0 version, write_site, read_site.

• plan.md → the mechanical contract (write pattern, read pattern,

preserved behavior).

• goal.md → the intent (displayed in the scorecard only; not tested).

2. Install dependencies

Route by language from product.json:

If install fails → exit code 2 with stderr tail. Never move to testing if dependencies don't resolve.

3. Static sanity checks (fast, local, no API calls)

• Import check: does the write-site file import the expected Mem0

surface? Authoritative list comes from ## Identify the User's Setup in https://docs.mem0.ai/llms.txt:

• Platform Python → from mem0 import MemoryClient
• Platform TS → import MemoryClient from "mem0ai"
• OSS Python → from mem0 import Memory
• OSS TS → import { Memory } from "mem0ai/oss"

If plan.md names a delegated skill (e.g., Vercel AI), use that skill's import signature instead of the list above. Mismatch → fail with line number.

• Version check: installed mem0ai version falls in the range from

this skill's mem0_tested_versions. Out of range → warn but continue.

• Type check (TS tracks only): run tsc --noEmit or tsup --dts.

Non-zero → fail.

• Lint (if the repo has a linter configured): run the repo's own

lint command. Lint failures from this skill's changes → fail; pre-existing lint failures → surface as a warning.

• Eager-init check: grep the write_site and read_site files (paths

from product.json) for MemoryClient( or Memory( at module scope — i.e., not inside a function, method, or class body. MemoryClient() validates the API key in __init__ (network call) and OSS Memory() can eagerly initialize embedding/LLM providers — module-level instantiation hits the wire on import and breaks Pass A's test collection whenever the key is unset. Hit → fail with file:line and the lazy-init guidance from /mem0-integrate step 8 constraint #7.

4. Run the repo's native test suite (two passes)

Pass A — `feature_flag` unset. Run the entire pre-existing suite (excluding the new test_mem0_* files). Must be 100% green. Any failure here marks non_invasive: false in the scorecard and is a hard fail — the integrator's self-heal loop refuses to touch it.

Pass B — `feature_flag` set (value from product.json). Run the full suite including the new tests. All must pass.

Isolate integration-introduced failures using git diff main..HEAD --name-only. A test file that exists on main and fails only under the integration branch (flag set or unset) counts against the scorecard regardless of pass. A test file that already failed on main is surfaced as pre_existing_unrelated and does not count — but is still reported so the user can clean it up.

Capture output to .mem0-integration/test-stdout-flag-off.log and .mem0-integration/test-stdout-flag-on.log. Scorecard reports pass/fail per pass.

5. Smoke test (real API call, shortest round-trip)

Scripted end-to-end flow tailored to product.json. The call shapes below are the minimal ones; if plan.md names a delegated skill, use that skill's minimal example verbatim instead — it is the canonical shape for the detected stack.

Platform (Python):

from mem0 import MemoryClient c = MemoryClient() # uses MEM0_API_KEY uid = f"mem0-test-integration-{os.urandom(4).hex()}" c.add([{"role": "user", "content": "I prefer aisle seats"}], user_id=uid) hits = c.search("seat preference", user_id=uid) assert any("aisle" in h.get("memory", "") for h in hits), hits c.delete_all(user_id=uid) # clean up

Platform (TS): same shape with MemoryClient from "mem0ai".

OSS (Python / TS): uses Memory() / new Memory() with default config (OpenAI LLM via OPENAI_API_KEY, local Qdrant). If the repo ships a docker-compose.yml with a Qdrant service, the skill starts it first and tears it down after. If no backing store is reachable → fail with a clear message naming the fix.

The smoke test always uses a disposable random user_id prefixed with mem0-test-integration- so a failed cleanup doesn't pollute the user's real data. A background tidy step deletes any prefix-matching entries older than 24 hours on the next run.

Capture output to .mem0-integration/smoke-stdout.log.

6. E2E integration test (run the app, exercise the flow)

Unit tests + smoke prove the SDK works in isolation. This step is the real signal: does memory actually appear in the app's user-visible output when the integration runs end-to-end?

Requires plan.md to contain an E2E recipe: section (authored by /mem0-integrate step 5). If absent → status skipped (not fail), note in scorecard that the repo has no runnable entry point.

Recipe fields the skill reads:

• start — shell command to launch the app using $PORT for any network

port. Run in background with stdout/stderr teed to .mem0-integration/e2e-app.log.

• ready_probe — how to detect readiness. url=... status=... polls an

HTTP endpoint; log="..." waits for a substring in e2e-app.log; sleep=N waits N seconds (last resort). 60-second hard timeout.

• compose_services — optional. If set, bring them up via

docker compose up -d <services> before start, tear them down with docker compose down at the end.

• write_call — triggers the Mem0 write path exactly once. Output is

captured and surfaced on failure. 60-second hard timeout.

• write_async_wait_ms — pause after write_call to let async memory

flushes land. Default 0.

• read_call — triggers the Mem0 read path. Typically a fresh session

or new request that should surface the stored memory.

• read_assert — substring, regex=..., or jsonpath=<expr>=<value>

that must appear in read_call's stdout. This is the E2E pass gate.

Execution order:

1. Allocate an ephemeral TCP port; export as PORT.
2. Set MEM0_USER_ID to a disposable mem0-test-integration-<rand> value

and export it, so the app can use the same scoping the smoke test does if the recipe wants cleanup.

1. Bring up compose_services if named.
2. Run start in the background.
3. Poll ready_probe until success or 60s timeout. Timeout → fail.
4. Run write_call. Non-zero exit → fail (but continue to cleanup).
5. Sleep write_async_wait_ms.
6. Run read_call.
7. Evaluate read_assert against read_call's stdout. Miss → fail.
8. Cleanup (always, even on failure): SIGTERM the app, SIGKILL after

5s, docker compose down if services were started, delete_all memories matching mem0-test-integration-* on Platform scenarios.

On any failure, the scorecard includes:

• Last 40 lines of e2e-app.log
• Full write_call output
• Full read_call output
• The expected vs actual for read_assert

7. Scorecard

Write .mem0-integration/scorecard.md and .mem0-integration/scorecard.json:

{ "timestamp": "2026-04-20T14:03:11Z", "branch": "mem0-integrate/remember-user-preferences", "product": "platform", "language": "python", "mem0_version": "2.0.0", "non_invasive": true, "feature_flag": "MEM0_ENABLED", "results": { "install": {"status": "pass", "duration_ms": 12043}, "static_checks":{"status": "pass", "duration_ms": 812}, "unit_tests_flag_off": {"status": "pass", "duration_ms": 3920, "count": 47, "reason": "all pre-existing tests green with flag unset"}, "unit_tests_flag_on": {"status": "pass", "duration_ms": 4321, "count": 49}, "smoke_test": {"status": "pass", "duration_ms": 2890, "memory_id": "mem_..."}, "e2e_test": {"status": "pass", "duration_ms": 14200, "ready_probe_ms": 3100, "write_exit": 0, "read_assert_matched": true} }, "friction": { "dependency_install_retries": 0, "pre_existing_test_failures": 0, "warnings": ["mem0ai 2.0.0 pinned; consider 2.0.1 for fix X"] }, "overall": "pass" }

The markdown version is human-readable and includes: