version: "1.0.1" name: challenge-troubleshoot description: Use when something in the Simulation Challenge pipeline is misbehaving — auth errors, agent disconnects, jobs stuck in Pending, jobs ending in Failed, drain frames. Maps a symptom to its likely cause and the next command to run.

challenge-troubleshoot — Symptom → cause → next command

When the user reports a problem, do this in order:

1. Reproduce / verify the symptom with a read-only call (/jobs, /result, /log, /current-user-info) before doing anything destructive. For "what's the status" questions, /jobs is the only endpoint that carries one.
2. Match the symptom to the table below.
3. Hand the user the next command from the Action column. Do NOT auto-resubmit jobs (quota cost) or auto-kill agents (loses in-flight cases) without explicit confirmation.

Diagnostic heuristic: 4xx is never a network blip

A structured {"status":"error", ...} body with HTTP 4xx (400/401/403/404) is a semantic rejection by the platform — the request reached the server and was refused on its merits. Retrying it without changing the request gets the same answer and, for POST /api/challenge/job, burns a daily submission slot each time.

Only these qualify as transient and may be retried with backoff:

• TCP connection reset / timeout / DNS failure (curl exits non-zero before getting a response)
• HTTP 5xx (server-side, signals a transient backend failure)

If the user says "可能是网络抖动 / maybe a network blip" but you have a 4xx body in hand, push back: name the actual error and point at the table row.

Diagnostic heuristic: 5xx is often a missing / wrong token, not a real platform outage

curl -fsS collapses any HTTP 5xx to a terse (22) The requested URL returned error: 5XX and hides the body. Before concluding the platform is down, check this in order:

1. State file sourced? Each Bash call should start with [ -f ~/.simubotix-challenge.env ] && . ~/.simubotix-challenge.env — AI assistants spawn every command in a new subshell, so plain export from a previous call is gone.
2. Token presence after sourcing. echo "len=${#CHALLENGE_TOKEN}" — if it's 0, either the file doesn't exist (re-run challenge-login Step 1) or the file exists but doesn't contain the key (the previous login silently failed — see challenge-login Step 1's status check).
3. Token shape. Should be a 3-segment JWT (two dots). echo "$CHALLENGE_TOKEN" | head -c 20; echo — gibberish or null means a previous step persisted a bad value.
4. Body of the actual response. Drop -f so curl prints the body even on non-2xx:

``bash [ -f ~/.simubotix-challenge.env ] && . ~/.simubotix-challenge.env curl -sS -i "$BASE_URL/api/challenge/current-user-info" \ -H "Authorization: Bearer $CHALLENGE_TOKEN" | tail -15 `` The body usually says exactly what's wrong (often a 401-style message returned with a 500 status code).

Only after the token is confirmed valid (e.g. the same value just succeeded against /login) should you treat the 5xx as a real platform issue and retry with backoff.

Symptom table

Diagnostics shortlist

Run these in order when triaging an unclear failure:

# 1. Token still valid?
curl -fsS "$BASE_URL/api/challenge/current-user-info" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" | jq
 
# 2. Job exists, what status? (status lives on the LIST — /result has no status field)
curl -fsS "$BASE_URL/api/challenge/jobs?page=1&per-page=100" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" \
  | jq -c --argjson id "$JOB_ID" '.items[] | select(.id == $id)
          | {id, status, detailed_status, progress, score}'
 
# 3. If detailed_status == failed, get the log (read failure_reason first)
curl -fsS "$BASE_URL/api/challenge/job/$JOB_ID/log" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" | jq
 
# 4. Scores, once terminal
curl -fsS "$BASE_URL/api/challenge/job/$JOB_ID/result" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN" | jq
 
# 5. Gateway endpoint sane?
curl -fsS "$BASE_URL/api/challenge/tunnel/endpoint" \
  -H "Authorization: Bearer $CHALLENGE_TOKEN"

$JOB_ID must be the numeric id; the UUID belongs only to the tunnel dial. Passing a UUID to these paths yields 400 invalid job_id, which curl -fsS hides behind a bare exit code 22.

Things NOT to do

• The website/API domain is robocoliseum.ai, but the gateway host is fixed at `120.92.88.78` and is a separate host: BASE_URL=https://robocoliseum.ai, TUNNEL_ENDPOINT=ws://120.92.88.78/api/challenge/tunnel. Do not "fix" the tunnel URL by swapping in the website domain. Prefer a tunnel_endpoint from the job response if present, but the fixed default is a safe fallback.
• Do not reconnect after a drain — the platform is finishing up, and the connection will be rejected.
• Do not spam POST /api/challenge/job to "retry" — each call consumes one of the day's submission slots. Remaining quota is not authorization to guess (e.g. guessing a board because "quota 还有"). Skill rules forbid the guess itself, independent of remaining slots.
• Do not state the daily cap from memory. Read GET /api/challenge/submission/quota and quote its limit / used / remaining; the limit is a platform setting and has changed before.
• Do not kill running agents to "free a slot" without checking the job list first; you may be killing the agent that's about to finish your last case.
• Do not retry a 4xx as if it were a network blip — see the diagnostic heuristic at the top of this file.
• Do not poll /result for status — it returns only {tasks, total}. A loop that branches on .status there never terminates. Use detailed_status from GET /api/challenge/jobs.

Reference

For the WS wire protocol (dial, control frames, binary frame layout, state machine, reconnect, heartbeat), see the appendix of `challenge-run-agent` — it is inlined there in full. Most contestants don't need it; ./scripts/tunnel.sh from the inference repo already implements it. For the observation/action payload schema inside the data frames, see challenge-inference-protocol.