speakeasy-api/agent-skills/setup-sdk-testing

setup-sdk-testing

Set up and run tests for Speakeasy-generated SDKs using contract testing, custom Arazzo workflows, or integration tests against live APIs.

Content Guides

The Arazzo reference provides complete syntax for workflows, steps, success criteria, environment variables, and chaining operations.

When to Use

• Setting up automated testing for a generated SDK
• Enabling contract test generation via gen.yaml
• Writing custom multi-step API workflow tests with Arazzo
• Configuring integration tests against a live API
• Debugging ResponseValidationError or test failures
• User says: "test SDK", "contract testing", "Arazzo tests", "speakeasy test", "mock server", "test generation"

Inputs

Outputs

Prerequisites

• Speakeasy CLI installed and authenticated (speakeasy auth login)
• A working SDK generation setup (gen.yaml + OpenAPI spec)
• For integration tests: API credentials as environment variables

Decision Framework

Pick the right testing approach based on what you need:

Start with contract testing. It is auto-generated and catches the most common issues. Add custom Arazzo tests for workflow coverage, and integration tests only when live API validation is required.

Command

Enable Contract Testing

Add to gen.yaml:

generation:
  tests:
    generateTests: true

Then regenerate the SDK:

speakeasy run --output console

Run Tests

# Run all Arazzo-defined tests (contract + custom)
speakeasy test
 
# Run tests for a specific target
speakeasy test --target my-typescript-sdk
 
# Run with verbose output for debugging
speakeasy test --verbose

Run via CI

Contract tests run automatically in the Speakeasy GitHub Actions workflow when test generation is enabled. No additional CI configuration is needed for contract tests.

Example

1. Contract Testing (Quick Start)

Enable test generation in gen.yaml:

generation:
  tests:
    generateTests: true

Regenerate the SDK:

speakeasy run --output console

Run the generated tests:

speakeasy test

The CLI generates tests from your OpenAPI spec, creates a mock server that returns spec-compliant responses, and validates that the SDK correctly handles requests and responses.

2. Custom Arazzo Tests

Create or edit .speakeasy/tests.arazzo.yaml:

arazzo: 1.0.0
info:
  title: Custom SDK Tests
  version: 1.0.0
 
sourceDescriptions:
  - name: my-api
    type: openapi
    url: ./openapi.yaml
 
workflows:
  - workflowId: create-and-verify-resource
    steps:
      - stepId: create-resource
        operationId: createResource
        requestBody:
          payload:
            name: "test-resource"
            type: "example"
        successCriteria:
          - condition: $statusCode == 201
        outputs:
          resourceId: $response.body#/id
 
      - stepId: get-resource
        operationId: getResource
        parameters:
          - name: id
            in: path
            value: $steps.create-resource.outputs.resourceId
        successCriteria:
          - condition: $statusCode == 200
          - condition: $response.body#/name == "test-resource"
 
  - workflowId: list-and-filter
    steps:
      - stepId: list-resources
        operationId: listResources
        parameters:
          - name: limit
            in: query
            value: 10
        successCriteria:
          - condition: $statusCode == 200

Run the custom tests:

speakeasy test

Using Environment Variables in Arazzo Tests

Reference environment variables for sensitive values:

steps:
  - stepId: authenticated-request
    operationId: getProtectedResource
    parameters:
      - name: Authorization
        in: header
        value: Bearer $env.API_TOKEN
    successCriteria:
      - condition: $statusCode == 200

Disabling a Specific Test

Use an overlay to disable a generated test without deleting it:

overlay: 1.0.0
info:
  title: Disable flaky test
actions:
  - target: $["workflows"][?(@.workflowId=="flaky-test")]
    update:
      x-speakeasy-test:
        disabled: true

3. Integration Testing

For live API testing, use the SDK factory pattern:

TypeScript example:

import { SDK } from "./src";
function createTestClient(): SDK {
  return new SDK({
    apiKey: process.env.TEST_API_KEY,
    serverURL: process.env.TEST_API_URL ?? "https://api.example.com",
  });
}
describe("Integration Tests", () => {
  const client = createTestClient();
  it("should list resources", async () => {
    const result = await client.resources.list({ limit: 5 });
    expect(result.statusCode).toBe(200);
    expect(result.data).toBeDefined();
  });
  it("should create and delete resource", async () => {
    // Create
    const created = await client.resources.create({ name: "integration-test" });
    expect(created.statusCode).toBe(201);
    // Cleanup
    const deleted = await client.resources.delete({ id: created.data.id });
    expect(deleted.statusCode).toBe(204);
  });
});

Python example:

import os
import pytest
from my_sdk import SDK
@pytest.fixture
def client():
    return SDK(
        api_key=os.environ["TEST_API_KEY"],
        server_url=os.environ.get("TEST_API_URL", "https://api.example.com"),
    )
def test_list_resources(client):
    result = client.resources.list(limit=5)
    assert result.status_code == 200
    assert result.data is not None
@pytest.mark.cleanup
def test_create_and_delete(client):
    created = client.resources.create(name="integration-test")
    assert created.status_code == 201
    try:
        fetched = client.resources.get(id=created.data.id)
        assert fetched.data.name == "integration-test"
    finally:
        client.resources.delete(id=created.data.id)

GitHub Actions CI for integration tests:

name: Integration Tests
on:
  schedule:
    - cron: "0 6 * * 1-5"
  workflow_dispatch:
 
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: speakeasy-api/sdk-generation-action@v15
        with:
          speakeasy_version: latest
      - run: speakeasy test
        env:
          SPEAKEASY_API_KEY: ${{ secrets.SPEAKEASY_API_KEY }}
      - name: Run integration tests
        run: npm test -- --grep "integration"
        env:
          TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
          TEST_API_URL: ${{ secrets.TEST_API_URL }}

What NOT to Do

• Do NOT skip contract testing and jump straight to integration tests -- contract tests are free and catch most issues
• Do NOT hardcode API keys or secrets in Arazzo test files -- use $env.VARIABLE_NAME syntax
• Do NOT modify auto-generated test files directly -- they are overwritten on regeneration; use custom Arazzo tests instead
• Do NOT disable failing contract tests without investigating -- a ResponseValidationError usually means the spec and API are out of sync
• Do NOT run integration tests with destructive operations in production -- always use a test/staging environment
• Do NOT assume all languages support contract testing -- currently supported: TypeScript, Python, Go, Java

Troubleshooting

ResponseValidationError in contract tests

The SDK response does not match the OpenAPI spec. Common causes:

1. Spec is outdated: Regenerate from the latest API spec
2. Missing required fields: Check your spec's required arrays match actual API responses
3. Type mismatches: Verify type and format fields in schema definitions

# Regenerate with latest spec and re-run tests
speakeasy run --output console && speakeasy test --verbose

Tests pass locally but fail in CI

1. Check that all environment variables are set in CI secrets
2. Verify the CI runner has network access to mock server ports
3. Ensure the Speakeasy CLI version matches between local and CI

speakeasy test command not found

Update the Speakeasy CLI:

speakeasy update

Mock server fails to start

1. Check for port conflicts on the default mock server port
2. Ensure the OpenAPI spec is valid: speakeasy lint openapi -s spec.yaml
3. Verify generateTests: true is set in gen.yaml and the SDK has been regenerated

Custom Arazzo test not running

1. Verify the file is at .speakeasy/tests.arazzo.yaml
2. Check that operationId values match those in your OpenAPI spec exactly
3. Validate YAML syntax -- indentation errors are the most common cause

Integration tests intermittently fail

1. Add retry logic for network-dependent tests
2. Use unique resource names with timestamps to avoid collisions
3. Ensure cleanup runs even on test failure (use finally or afterEach)