ippclub/dora-ssr/dora-agent-development-validation

Dora Agent Development Validation

Treat Dora Agent as an interactive product, not only a code path. Validate the implementation, deterministic runtime behavior, actual LLM sessions, and what a human user can see and operate before calling an iteration effective.

For exact engine build and game-run commands, also follow the project skills dora-engine-development and dora-cli-game-development. Use this skill for experiment design, evidence standards, and the iteration loop around those commands.

For end-to-end game creation through Dora Agent, read references/game-iteration-playbook.md before starting. It defines how to turn a user-level brief into a plan, conduct multiple development sessions, play the result as a user, and convert observations into the next bounded prompt.

1. Establish The Real Baseline

1. Inspect the current checkout. Record the branch, commit, working-tree state, Agent configuration, model configuration, and Web IDE build.
2. Read the implementation path that owns the behavior. Distinguish prompt guidance from enforced runtime behavior.
3. Inspect relevant real records under the tested project's .agent/ directory, especially .agent/main/PROJECT_MEMORY.md, .agent/main/SESSION_SUMMARY.md, task/session histories, and test-result artifacts.
4. Verify summary claims against source, logs, diffs, and runnable artifacts. A summary saying “passed” is not proof of visual correctness or user acceptance.
5. Preserve unrelated work. Use separate worktrees, copies, or explicit commits for baseline and improved variants; never obtain a baseline by destructively resetting user changes.

Trace these owners first when relevant:

• Assets/Script/Lib/Agent/AgentConfig.ts: central limits and ratios.
• AgentToolRegistry.ts: tool schemas, role availability, and parallel safety.
• CodingAgent.ts: decision loop, prompts, execution guidance, completion checks, and streamed responses.
• AgentSession.ts: task/session lifecycle, cancellation, sub-agent handoff, and persistence.
• Memory.ts: consolidation and compression behavior.
• Tools.ts: file, command, checkpoint, and recovery operations.
• Utils.ts: LLM transport, stream aggregation, usage extraction, and parsing.
• Assets/Script/Dev/WebServer.yue: Agent and run/build routes.
• Tools/dora-dora/src/AgentPanel.tsx: user-visible Agent state and controls.
• Assets/Doc/skills/dora-engine-coding/SKILL.md: instructions injected for Dora runtime work.

Patch authored TypeScript/YueScript first. Regenerate matching Lua through the normal build path when generated Lua is tracked.

2. State A Testable Hypothesis

Express each iteration as one behavioral claim, for example:

• “A truncated whole-file overwrite keeps the recoverable prefix, while a truncated replacement leaves the target unchanged.”
• “Spawning a sub-agent does not block the main user session or force a join.”
• “Model-aware compression thresholds prevent premature compaction without overflow.”
• “The improved game-generation flow increases first-pass playable results under the same prompt and time limit.”

Define before editing:

• The observable success condition.
• The regression condition.
• Which evidence layer can prove each condition.
• The smallest implementation change expected to affect it.

Avoid bundling unrelated optimizations into one comparison. If several changes ship together, retain per-feature tests so later regressions can be localized.

3. Validate In Layers

Use every relevant layer. Passing an earlier layer never substitutes for a later one.

Layer A: Static And Build Integrity

• Run git diff --check for the intended scope.
• Build only the Agent subtree first:

env -u http_proxy -u https_proxy -u HTTP_PROXY -u HTTPS_PROXY \
  -u ALL_PROXY -u all_proxy -u NO_PROXY -u no_proxy \
  dora ts build -f /path/to/Dora-SSR/Assets/Script/Lib/Agent

• Check generated Lua with luac -p when Lua artifacts changed.
• Use the explicit Agent project root for /ts/build. An unrelated active Web IDE tab must not determine TSTL module lookup.
• Inspect per-file messages; a successful HTTP request can still contain failed compilation items.

Layer B: Deterministic Unit And Policy Tests

Exercise success and refusal paths without an LLM where possible. Include malformed, boundary, interruption, and repeated-call cases.

Examples:

• A tool visible in schema is executable by the same role.
• A provider omits or rejects forced tool_choice.
• Empty old_str plus truncated new_str overwrites with exactly the recoverable prefix.
• Non-empty old_str plus truncated new_str fails and leaves the target byte-for-byte unchanged.
• Cancellation reaches STOPPED and remains STOPPED in final task state.
• “End and hand off” finishes an interrupted sub-agent with a coherent handoff.
• Compression preserves every active work item and its next checkpoint.

Layer C: Dora In-Engine Validation

Use the Agent's execute_command path plus the Dora engine coding skill for runtime tests. Inject or reuse Entry.yue helpers such as enterEntryAsync and stop when supported.

Confirm:

• The authored entry builds.
• The actual project loads inside Dora.
• The expected state transition occurs.
• A deterministic marker or test result reports passed.
• The project stops and cleans up.

Treat lifecycle success as lifecycle evidence only. running=true, a clean stop, or a marker test cannot prove that the screen is understandable, buttons are aligned, hit areas match visuals, or gameplay feels usable.

Layer D: Human-Visible And Interactive Acceptance

For games, rendering, and Web IDE UI, inspect at minimum:

1. Initial frame before input.
2. First actionable state after starting.
3. A representative primary interaction.
4. Success/failure/retry or next-level state.
5. A resized or target-resolution state when layout adapts.

Check object visibility, hierarchy, z-order, labels, contrast, button background/text order, hit regions, overlays, selected state, instructions, and whether the user can infer what to do.

Actually play enough of the game to exercise its core rule. A scripted logic test does not replace this. When reporting as a human acceptance tester, describe what a user can see and do; keep instrumentation details in the technical evidence section.

If no visual observation path exists, mark visual validation not_run and keep the task in “needs visual acceptance.” Do not finish it as fully verified. execute_command by itself does not give the model visual perception.

Layer E: Real LLM Agent Sessions

Run the real Agent with the configured provider and the same tools a user receives. Record raw task/session artifacts rather than reconstructing behavior from the final answer.

Exercise:

• A fresh or nearly empty project.
• An existing project requiring a narrow edit.
• A compile failure followed by repair.
• A long edit that reaches output limits.
• A task that benefits from multiple asynchronous sub-agents.
• A user interruption, cancellation, resume, and handoff.
• A long conversation that crosses memory compression thresholds.
• A game task requiring deterministic and visual acceptance.

Keep session roles distinct during game work:

• Use Plan mode to settle the player fantasy, core loop, controls, progression, visual direction, scope, and acceptance criteria. Do not prescribe source files or engine APIs unless the user did.
• Use implementation sessions to deliver one playable milestone at a time. Preserve the proven core instead of repeatedly rewriting the whole game.
• Use acceptance sessions only after personally exercising the current build. Report player-observable problems and desired outcomes; do not leak evaluator instrumentation or dictate a patch.
• Start a repair session when a visible or playable defect remains. Do not accept an Agent's own completion statement as the final verdict.

4. Run Controlled Baseline/Improved Comparisons

Keep these fixed within one comparison group:

• User-level prompt.
• Model and provider configuration.
• Context-window and token limits.
• Tool availability.
• Starting project and assets.
• Time limit.
• Evaluator-only acceptance criteria.

Write prompts as normal user requirements. Do not leak implementation details, file names, API choices, test mechanics, or the expected fix into the generation prompt.

Use at least three trials per variant for a quick directional result and five or more for a stronger claim. Treat a model switch as a new comparison group; do not mix GLM and DeepSeek results into one baseline.

Alternate or randomize baseline/improved run order when practical. Use isolated copies or worktrees and clean derived artifacts between trials. Do not let later trials read earlier solutions, summaries, or test results.

Apply the same strict time limit to both variants. Ten minutes per trial is useful for fast game-generation comparisons, but choose a limit appropriate to the task.

Read references/evaluation-template.md before running or reporting a controlled comparison.

5. Measure Product Outcomes And Cost

Record at least:

• First-pass acceptance without user-requested repair.
• Build and deterministic-test success.
• Playable/usable result count.
• Visual and interaction acceptance.
• Agent loop steps and total LLM calls.
• Main-session starts and sub-agent session starts.
• Wall-clock time to first build, first runnable result, and accepted result.
• Input, output, cached-input, and total token usage when available.
• Local token estimate used by runtime policy, separately from provider-reported usage.
• Tool failures, invalid calls, retries, compression rounds, and interrupted outputs.
• User-found regressions after the Agent declared completion.

Do not optimize only for fewer turns. A shorter run that produces an unusable game is worse. Prioritize first-pass acceptance, correctness, repair burden, and stable interaction; use turns, time, and tokens as secondary efficiency measures.

6. Test High-Risk Agent Semantics Explicitly

Tool Availability

• Ensure every tool shown in schema and prompt can execute for that role.
• Put recommendations in prompt/result guidance unless a true safety or capability boundary requires enforcement.
• Test providers that do not support forced tool_choice.
• Measure cache effects, but never sacrifice correct tool availability only to improve cache hit rate.

Asynchronous Sub-Agents

• Verify the main Agent can dispatch several independent sub-agents without ending the turn after each spawn.
• Verify it can finish the user-facing turn without joining or polling background work.
• Verify the user can start another session while sub-agents continue.
• Verify handoffs arrive asynchronously and do not overwrite newer user instructions.
• Treat foreground-work limits as guidance unless the product deliberately defines a hard limit.
• Keep one task per sub-agent and measure duplicated discovery, conflicting edits, and abandoned interrupted agents.

Memory Compression

• Derive thresholds from the configured model context window.
• Use local estimates for runtime decisions when that is the contract; show provider usage only as supplementary evidence.
• Test high message-count conversations separately from high token-count conversations.
• Verify compression retains active instructions, completed items, open items, paths, test failures, next action, and sub-agent state.
• Confirm the next turn resumes useful work instead of restarting broad discovery.

Long Tool Output