osu-nlp-group/earlyexperience/skill

Early Experience Data Generation

This skill governs how data generation for the early-experience paradigm is done. Read this entire file before starting work on a new env.

What this skill is for

This skill produces SFT-ready training data for the two methods defined in the paper (https://arxiv.org/abs/2510.08558) and recapped in METHOD.md:

• Implicit World Modeling (IWM) — train the policy to predict the next state given the current state and an action.
• Self-Reflection (SR) — train the policy to produce a chain-of-thought reasoning over expert vs alternative actions, then the expert action.

The output of the workflow is JSONL files in three categories: expert, IWM, reflection. Where you put them and how you organize the surrounding pipeline scripts is up to your project — the skill is agnostic about layout. Training itself is out of scope. If a task requires running SFT, evaluating checkpoints, or tuning training hyperparameters, that belongs elsewhere — surface it to the user rather than acting on it inside this skill.

Required reading order

Before starting a new env, read in this order:

1. METHOD.md — what IWM and SR actually are, and the reflection prompt template. This is the source of truth for the method.
2. method_recap.md — short list of decisions where the right answer is non-obvious. Use as a runbook, not as a tutorial.
3. pitfalls.md — accumulated gotchas from previous envs. Skim before each new env in case something carries over.
4. This env's own NOTES.md (if you or a previous agent already started one) — env-specific decisions already recorded. NOTES_TEMPLATE.md is the skeleton to copy when you're starting fresh.

If any of these files is missing or empty (aside from NOTES.md on a fresh env), surface it as a question rather than guessing.

On conflicts between layers. For skill-wide mechanics (gates, output shape), this skill wins. For env-specific content (state representation, K, sampling strategy, prompt extensions, filtering rationale, anything tied to one env's interface or paper section), the env's NOTES.md is closer to truth than the generic guidance here. But never silently resolve a conflict. If NOTES.md says one thing and SKILL.md / METHOD.md says another, stop and surface the conflict to the user before proceeding — the usual outcome is that NOTES.md needs a correction, and the user will edit it.

Step 0: Ask the user which generator model to use

Before writing any pipeline code or making any LLM API call, ask the user which model to use for the generation stages of this pipeline — alternative-action proposal (when the action space isn't fully enumerable), state/observation summarization (when the raw next-state text is too noisy), and self-reflection CoT generation.

Explain the choice to the user:

• The paper's method uses the base policy model itself as the generator — the same model that will be trained. This is what "self" in Self-Reflection literally refers to: the model reflects on its own alternative actions using its own capacity. The paper's core claim is that base-model exploration and self-reflection alone can improve over training on expert trajectories, without any external stronger model. This matters especially in environments without verifiable rewards where RL is impractical.
• Another model — stronger, or just different — is also a valid choice. The pipeline works with any capable instruction-following LLM. Use a stronger one if you care more about data quality than paper-faithful reproduction, or a smaller / faster one if cost is the constraint.

Neither is more "correct" — the choice is up to the user. Record the decision in the env's NOTES.md under a "Model choice for data generation" line, then use that model consistently for all three generation stages in this env. If later stages want to switch, surface the change to the user.

Hard rules (non-negotiable)

These override any user request short of an explicit override.

1. No LLM API call without user confirmation. Every batch — smoke or full, 5 samples or 50k — must be preceded by a message stating what will run, how much data, and approximate token / cost volume. Wait for explicit go-ahead. Silence is not consent.

1. Default to not editing inside the upstream env repo. Prefer external wrappers or config overrides from outside it. If touching upstream internals is genuinely the simpler path, explain the reason to the user, get approval, then make the change.

1. Env runs end-to-end before pipeline code is written. For any new env: install per its README, run its own tests or a minimal reset → step → done loop, confirm it works. Pipeline code (rollout, reflection generation, SFT writers) does not get written until this gate passes. The point is to surface install / dependency / runtime issues before they get tangled with your own code.

1. Smoke before scale. No full-scale rollout or reflection generation before a smoke run has produced data that passes hand inspection of at least 5 samples.

1. Filter rules require explicit user approval. Default position for any new env is no filters. Any filter applied to SFT data must first be proposed to the user with rationale and quantified impact (how many samples it drops, what kind); wait for approval, record the rule and its rationale in the env's NOTES.md, then apply.

1. Before scaling from smoke to full, summarize and confirm. Send a single message covering all the per-env decisions locked in during smoke (K, sampling regime, prompt structure, filter rules if any, projected token / time / cost). Wait for explicit user go-ahead before launching the full run.

1. Final SFT output shape is fixed. See "Output shape" below. The three SFT categories (expert / IWM / reflection), the filename prefix per category, and the outer JSONL schema are not negotiable. Default is one file per category; produce more only when the env's pipeline (as recorded in its NOTES.md) prescribes multiple variants.

1. This skill's scope stops at SFT-ready files. If a task requires running SFT, evaluating checkpoints, or tuning training hyperparameters, that is out of scope for the skill — surface it to the user rather than acting on it silently.

1. SR reflection length is a per-env decision, confirmed with the user before any LLM batch. Unconstrained, modern LLMs emit ~500–1000 word reflections, with the longest cases also being the most pathological (model wrestling with itself, drifting to a different action). Recommended default: target 200–400 words with a soft cap around 500 words, enforced only through prompt instruction — not via a max_tokens API limit. Hard token caps truncate mid-sentence and produce worse training data than overlong-but-complete reflections; rely on the prompt to nudge length, and accept the occasional outlier. Each env must surface its own length choice in the env's NOTES.md and have the user explicitly confirm it — sometimes a short-task env wants tighter (e.g. 200 target); sometimes a long-horizon complex-reasoning env wants more headroom.

Note for smaller base policy models: consider shortening the SR target further. The action tokens the model must actually emit at inference are a small fraction of any single reflection example, so long reflections dilute the imitation signal in the loss. Smaller models have less capacity to spare on background reasoning — a tighter reflection concentrates supervision on the action itself and often improves the downstream gain.

1. The prediction stage (IWM) must be format-separated from the imitation stages (expert / SR). IWM trains the model to predict a next state; expert and SR train it to produce an action. These are opposite behaviors and must be distinguishable by the model itself — otherwise a checkpoint that inherits IWM weights (the paper's two-stage IWM continues imitation from the IWM stage, and any joint mix trains them together) learns to emit next-state / observation text where an action is expected. Give IWM its own system prompt (framed as world-modeling / next-state prediction, not "you are an agent who acts") and a delimited next-state target (e.g. Observation:\n<state>); keep the agent system prompt and the action-output format (Thought:/Action: or the env's equivalent) for expert and SR. Never let the prediction stage and the action stages share one system prompt with an undelimited target. See "Output shape" below.

1. The SFT message format must match the env's native evaluator's input structure. Training and evaluation are out of scope, but the SFT message shape is this skill's output, and a train/eval format mismatch degrades the trained model no matter how clean the data is. Before finalizing an env's SFT files, read how the env's native eval harness builds the policy's input — multi-turn vs single-turn, whether the task instruction sits in a system message or the first user turn, any fixed acknowledgement turns (e.g. an "OK, I'll follow…" turn), the exact instruction text, and how the state/task string is rendered — and match it in expert_sft and reflection_sft (the categories the policy is actually run in at eval). Record the eval-format reference (which file/function builds the input) in the env's NOTES.md.

Output shape

Only the final SFT outputs have a fixed shape. Everything else — directory layout, intermediate artifact naming, pipeline code structure — is the project's choice.

Required final outputs

Each env produces files in three categories — expert, IWM, reflection — all in OpenAI chat-messages JSONL format:

{"messages": [{"role": "...", "content": "..."}, ...]}

Default is one file per category, using the canonical names below. Some envs' pipeline (as captured in the env's NOTES.md) prescribes more than one variant in a category — typically when the paper itself specifies different settings for different policy model sizes. In those cases produce each variant as its own file with a descriptive suffix and list them in the NOTES.md.

• `expert_sft.jsonl` — imitation learning baseline.

user content = state representation; assistant content = expert action.

• `iwm_sft.jsonl` — implicit world modeling stage.

user content = state representation + the chosen action (expert or alternative); assistant content = next state (raw or summarized, env's choice).

• `reflection_sft.jsonl` — self-reflection stage.

user content = state representation; assistant content = reflection chain-of-thought, followed by the expert action.

The string format inside each content field — how a "state" is serialized, how an "action" is encoded — is the env's choice. The serialization must be consistent across all three categories for the same env: a given s_i is rendered the same way in expert, IWM, and reflection files. Document the choice in the env's NOTES.md.

Each file is single-purpose within its category. Do not mix reflection samples into an expert file, or expert-only samples into a reflection file. Cross-category mixing at training time is the trainer's job.