turnstonelabs/turnstone/import-conversation-history

Importing Conversation History into Turnstone

Overview

Source formats vary; the destination does not. Your job is to translate whatever the user hands you (JSON dump, ZIP export, scraped HTML, screenshot OCR, raw transcript) into Turnstone's internal shape: one workstream row plus an ordered sequence of conversation rows in OpenAI message format. This skill documents the destination so you can write a correct mapper for any source.

Two questions to settle with the user before writing anything:

1. Archive or resumable? An archive is left closed and is read-only history. A resumable import is also kept closed and unloaded while rows are written, then explicitly opened after validation; this only works cleanly when the source LLM matches a Turnstone-supported provider/model and tool definitions still resolve.
2. One workstream per source thread, or merge? Default to one-to-one unless the user explicitly asks to merge.

Default to archive when in doubt — resuming a foreign transcript with mismatched tool schemas or stale provider signatures will fail at the next turn.

Turnstone Data Model (the destination)

Two tables carry the conversation:

workstreams (one row per imported thread)

conversations (many rows per thread, ordered by id/timestamp)

The internal format is OpenAI-shaped, even when the source was Anthropic or Gemini. Providers translate at their own API boundary; storage stays uniform.

Identity & Routing (ws_id)

• ws_id is 32-char lowercase hex (i.e. secrets.token_hex(16)).
• Ordinary placement is rendezvous (Highest Random Weight, HRW) selection over

the full `ws_id` and the current live server set. For each node, Turnstone computes 32-bit FNV-1a over the node ID, a NUL separator, and the full workstream ID; it then applies the node weight and selects the highest score. A live per-workstream override takes precedence.

• The live set comes from recent services heartbeats. Placement can therefore

change when nodes join, leave, change weight, or an override changes. There is no stable prefix-derived placement to pre-compute or persist.

• workstreams.node_id is stamped at creation and is not updated as HRW

placement changes. It is display metadata: cleanup keys on the owner lease, and the console router prefers the node holding a live lease before ordinary placement.

• For multi-node imports, create through the console routing proxy when the

lifecycle must be published, or write the history once through the cluster's configured shared storage backend. Never partition rows across node-local databases by ID prefix or by a one-time HRW result: a later membership change can route the same full ID to another node.

• For single-node imports, HRW placement is degenerate; any valid ws_id works.
• Do not reuse the source platform's IDs as `ws_id` unless they happen to be 32-char hex. Generate fresh; if you need the old ID for traceability, store it in workstream_config under a key like import.source_id.

Recommended Import Path

Three options, in order of preference:

1. Quiesced storage import (recommended for full history)

Use the current turnstone.core.storage.StorageBackend protocol against the same shared backend as the cluster. The destination must remain absent from all in-memory session managers while rows are changing: a loaded ChatSession holds its own trajectory and will not observe conversation rows inserted behind it.

The safe sequence is:

1. Normalize and validate the complete source transcript before writing.
2. Call register_workstream(..., state="closed") and require a True return;

False means the caller-selected ID already exists, so abort rather than appending to an unrelated workstream.

1. Insert the ordered conversation rows and attachment references.
2. Load the saved rows back and run the validation checklist below.
3. Leave an archive closed. For a resumable import, only now invoke the normal

POST /v1/api/workstreams/{ws_id}/open endpoint on the currently routed node so the session hydrates from the complete transcript.

Do not create the destination through the web/SDK create endpoint before a direct bulk import. Create publishes an empty live session. If that already happened, close the workstream and confirm the manager-authoritative live probe returns false before writing, then explicitly open it again after validation. Storage enforces this: while any server holds the workstream's owner lease, the import's writes are refused with WorkstreamLeaseHeldError.

For attachment-free history, save_messages_bulk(rows) is the canonical single-transaction insert primitive and bypasses the LLM round-trip entirely. New attachment bytes require the per-row path described under Attachments.

from turnstone.core.storage import get_storage  # initialized by the host/import entry point
storage = get_storage()
inserted = storage.register_workstream(
    ws_id=ws_id,
    user_id=user_id,
    name=name,
    state="closed",
    kind="interactive",
    ...
)
if not inserted:
    raise RuntimeError(f"destination already exists: {ws_id}")
storage.save_messages_bulk([
    {"ws_id": ws_id, "role": "user", "content": "Hello"},
    {"ws_id": ws_id, "role": "assistant", "content": "Hi! What can I help with?"},
    {"ws_id": ws_id, "role": "assistant", "content": None,
     "tool_calls": json.dumps([{"id": "call_1", "type": "function",
                                "function": {"name": "search", "arguments": "{\"q\":\"x\"}"}}])},
    {"ws_id": ws_id, "role": "tool", "tool_name": "search", "tool_call_id": "call_1",
     "content": "result text"},
    # ...
])

save_messages_bulk handles timestamp and the workstream's updated column internally, so you don't need to compute them per row. Verify the exact register_workstream and message signatures in turnstone/core/storage/_protocol.py; the Storage protocol, not the physical table layout, is the source of truth.

Multi-node note: this path assumes get_storage() is connected to the cluster's shared backend. Do not open a node-local database selected from the current HRW result, and do not pre-create a live session through the console routing proxy. After the shared-storage import commits, resolve the current route and open the closed workstream on that node. Any stored node_id describes creation-time placement, not a permanent shard that should receive a separate copy.

2. SDK create_workstream(resume_ws=...) (when the source is already a Turnstone workstream)

Only useful for Turnstone → Turnstone re-parenting. Not relevant for foreign sources.

3. SDK create_workstream(initial_message=...) + send() per turn (last resort)

Only fits archives where the source had no tool calls and you don't care about preserving assistant turns verbatim. Each send() triggers a real LLM round-trip, which is expensive and rewrites assistant content. Don't use this for full history.

Role Mapping

Common source-role conventions and how they map to Turnstone:

Tool Calls (the most error-prone part)

Turnstone stores tool calls in OpenAI's nested-function shape on the assistant row, and matches them with role="tool" result rows by tool_call_id.

Assistant row with tool calls

{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "search_web",
        "arguments": "{\"query\":\"turnstone import\"}"
      }
    }
  ]
}

tool_calls[].function.arguments is a JSON-encoded string, not an object. Source formats commonly get this wrong — Anthropic stores arguments as a parsed object, Gemini as a struct. Always re-serialize to a string.

Tool result row

{
  "role": "tool",
  "tool_name": "search_web",
  "tool_call_id": "call_abc123",
  "content": "..."
}

Pairing rules:

• Every assistant tool_calls[].id MUST be followed by exactly one role="tool" row with the matching tool_call_id, before the next user/assistant turn.
• If the source dropped the tool result (cut-off transcript), insert a synthetic role="tool" row with content="[tool result missing in source]" to keep the chain valid. An assistant row with an unanswered tool_calls[].id will break replay and any LLM round-trip.
• Multi-tool assistant turns: one role="tool" row per call, in any order, all before the next non-tool row.

Tool ID generation