nguyenluan213/auto_skyvern/testing

Testing

Smoke-test a Skyvern deployment to verify the backend API responds, the frontend renders correctly, browser sessions can be provisioned, and workflows can execute end to end.

Checks

Run these three checks sequentially. Stop on the first failure — later checks depend on earlier ones passing.

1. Backend API Health + Frontend Renders

Verify the API server is reachable and the frontend renders correctly.

skyvern_browser_session_create(timeout=5)
skyvern_navigate(url="{{base_url}}")
skyvern_evaluate(expression="fetch('/api/v1/workflows?page=1&page_size=1', {credentials: 'include'}).then(r => ({status: r.status, ok: r.ok, reachable: r.status > 0})).catch(e => ({status: 0, ok: false, reachable: false, error: e.message}))")

Pass: fetch returns a response (any HTTP status confirms the backend is reachable). A 2xx means fully healthy; 401/403 means the backend is running but requires authentication. Only status: 0 or a network error means the backend is actually down.

skyvern_navigate(url="{{base_url}}/discover")
skyvern_validate(prompt="The page does NOT show any error messages, error toasts, 'Something went wrong', a persistent loading spinner, a blank white screen, or a connection refused message")
skyvern_validate(prompt="The page shows 'What task would you like to accomplish?' as a heading, a text input area with 'Enter your prompt...' placeholder, an engine version selector, and a send/submit button icon")
skyvern_screenshot()
skyvern_browser_session_close()

Pass: backend returned an HTTP response (not a network error) AND both frontend validations return valid: true.

2. Browser Session Provisioning

Verify the system can create, use, and close browser sessions end to end. This tests the critical path — Skyvern's ability to provision cloud browsers.

skyvern_browser_session_create(timeout=5)
skyvern_navigate(url="https://example.com")
skyvern_validate(prompt="The page shows 'Example Domain' as a heading and contains a link to 'More information'")
skyvern_browser_session_close()

Pass: session creation succeeds, navigation works, and the external page loads correctly. If this fails, browser provisioning infrastructure is broken.

3. Workflow Execution (Smoke)

Verify a minimal workflow can be created and executed to completion.

skyvern_workflow_create(definition='{"title":"Deployment Smoke Test","workflow_definition":{"parameters":[],"blocks":[{"block_type":"goto_url","label":"visit","url":"https://example.com"}]}}', format="json")
skyvern_workflow_run(workflow_id="<id from above>", wait=true, timeout_seconds=60)
skyvern_workflow_status(run_id="<run_id from above>")

Pass: workflow run completes with status completed. If this fails, the execution pipeline (Temporal workers, browser provisioning, or task orchestration) is broken.

Always clean up the smoke test workflow, regardless of pass or fail:

skyvern_workflow_delete(workflow_id="<id>", force=true)

Parameters

Pass Criteria

All three checks pass in order. If any check fails:

1. Capture a screenshot with skyvern_screenshot()
2. Report which check failed and why
3. Skip remaining checks (they depend on earlier ones)

Retry Protocol

When a validation returns valid: false:

1. Wait 3 seconds: skyvern_wait(time_ms=3000)
2. Take a screenshot for evidence: skyvern_screenshot()
3. Retry the same validation once
4. If still false, mark as FAILED with both screenshots attached

Session Cleanup

ALWAYS close the session, even if earlier steps fail. If any step errors out:

1. Capture a failure screenshot: skyvern_screenshot()
2. Record the failure reason
3. Call skyvern_browser_session_close() before moving to the next check

Troubleshooting