alignbase/sunpeak/test-mcp-server

Test MCP Server

sunpeak includes a testing framework that works with any MCP server, regardless of language or framework. It provides four testing layers: e2e tests (inspector-based), visual regression, live tests (against real ChatGPT), and evals (multi-model tool calling).

For sunpeak app projects, testing integrates automatically. For non-sunpeak MCP servers (Python, Go, Rust, etc.), sunpeak test init scaffolds a self-contained test directory.

Getting Started

sunpeak test init                        # Interactive setup (detects project type)
sunpeak test init --server http://localhost:8000/mcp  # URL-based server
sunpeak test init --server "python server.py"         # Command-based server

sunpeak test init detects three project types:

• sunpeak projects -- Adds defineConfig() and eval boilerplate
• JS/TS projects -- Adds Playwright config, smoke test, and evals at project root
• Non-JS projects -- Creates a self-contained tests/sunpeak/ directory with its own package.json

Getting Reference Code

Clone the sunpeak repo for working test examples:

git clone --depth 1 https://github.com/Alignbase/sunpeak /tmp/sunpeak

Test examples live at /tmp/sunpeak/packages/sunpeak/template/tests/. This includes e2e tests, simulations, evals, and live tests.

Commands

sunpeak inspect              # Inspect any MCP server in the inspector (standalone)
sunpeak test                 # Run unit (if configured) + e2e tests
sunpeak test --e2e           # Run e2e tests only (Playwright)
sunpeak test --visual        # Run e2e tests with visual regression comparison
sunpeak test --visual --update  # Update visual regression baselines
sunpeak test init            # Scaffold test infrastructure into a project
sunpeak test --live          # Run live tests against real ChatGPT (requires tunnel + browser session)
sunpeak test --eval          # Run evals against multiple LLM models (requires API keys)
sunpeak test --unit          # Run unit tests (sunpeak app framework only, not standalone)

Flags are additive: --e2e --live --eval runs all three. --update implies --visual. --eval and --live are never included in the default run (they cost money). --unit is only for sunpeak app framework projects (created with sunpeak new), not standalone testing.

E2E Tests with the mcp and inspector Fixtures

Import test and expect from sunpeak/test. The mcp fixture provides protocol-level methods, and the inspector fixture handles rendering, double-iframe traversal, URL construction, and host selection. Tests run automatically across ChatGPT and Claude hosts via Playwright projects.

import { test, expect } from 'sunpeak/test';
test('renders weather card', async ({ inspector }) => {
  const result = await inspector.renderTool('show-weather');
  const app = result.app();
  await expect(app.locator('h1')).toHaveText('Austin');
});
test('renders in dark mode', async ({ inspector }) => {
  const result = await inspector.renderTool('show-weather', {}, { theme: 'dark' });
  const app = result.app();
  await expect(app.locator('h1')).toBeVisible();
});
test('loads without console errors', async ({ inspector }) => {
  const errors: string[] = [];
  inspector.page.on('console', (msg) => {
    if (msg.type() === 'error') errors.push(msg.text());
  });
  const result = await inspector.renderTool('show-weather', {}, { theme: 'dark' });
  const app = result.app();
  await expect(app.locator('h1')).toBeVisible();
  const unexpectedErrors = errors.filter(
    (e) =>
      !e.includes('[IframeResource]') &&
      !e.includes('mcp') &&
      !e.includes('PostMessage') &&
      !e.includes('connect')
  );
  expect(unexpectedErrors).toHaveLength(0);
});
test('pip mode (skip on Claude)', async ({ inspector }) => {
  test.skip(inspector.host === 'claude', 'Claude does not support PiP');
  const result = await inspector.renderTool('show-weather', {}, { displayMode: 'pip' });
  await expect(result.app().locator('h1')).toBeVisible({ timeout: 5000 });
});

When testing named object output schemas with MCP SDK 2.4, assert against the object in structuredContent directly, including on 2025-11-25 connections. Non-object legacy results still use { result: value }. Invalid URI syntax in $id or $ref can prevent a tool schema from compiling; use valid IDs before diagnosing a missing tool as an inspector failure.

mcp Fixture API

Protocol-level methods (raw MCP data, no rendering):

inspector Fixture API

Rendering methods and properties:

InspectorResult API

MCP-Native Matchers

renderTool

inspector.renderTool renders the tool result in the inspector and returns an InspectorResult. With input, the tool is called on the real server with those arguments. Without input, simulation fixture data is used when available, or the real server is called with empty args.

// Calls the real server with arguments and renders
const result = await inspector.renderTool('search', { query: 'headphones' });
// Uses simulation fixture data, or calls server with empty args
const result = await inspector.renderTool('show-albums');

Configuring Timeouts

Set default timeouts in Playwright config for servers that need more time:

export default defineConfig({
  server: { command: 'python', args: ['server.py'] },
  timeout: 120_000,  // Server startup timeout (default: 60s)
  use: {
    mcpTimeout: 30_000,    // renderTool iframe timeout (default: 15s)
  },
});

Visual Regression Testing

Use result.screenshot() to capture and compare screenshots against saved baselines. Comparisons only run with sunpeak test --visual. Without it, screenshot() silently skips, so you can include it in regular e2e tests.

By default, result.screenshot() captures the rendered app inside the double-iframe, not the inspector chrome. Keep visual baselines scoped to the app unless the test is explicitly about the inspector UI.

import { test, expect } from 'sunpeak/test';
test('albums renders correctly', async ({ inspector }) => {
  const result = await inspector.renderTool('show-albums', {}, { theme: 'light' });
  const app = result.app();
  await expect(app.locator('button:has-text("Summer Slice")')).toBeVisible();
  await result.screenshot('albums-light');
});

screenshot() options:

All Playwright toHaveScreenshot options are passed through.

Configure project-wide visual defaults:

import { defineConfig } from 'sunpeak/test/config';
export default defineConfig({
  visual: {
    threshold: 0.2,
    maxDiffPixelRatio: 0.05,
  },
});

Playwright Config

// playwright.config.ts
import { defineConfig } from 'sunpeak/test/config';
export default defineConfig();
// Creates per-host projects (chatgpt, claude). Tests run once per host automatically.

For non-sunpeak MCP servers, pass a server option:

import { defineConfig } from 'sunpeak/test/config';
export default defineConfig({
  server: {
    url: 'http://localhost:8000/mcp',
  },
});
// Or with a command (stdio server):
export default defineConfig({
  server: {
    command: 'python',
    args: ['server.py'],
    env: { API_KEY: 'test-key' },  // Extra environment variables
    cwd: './backend',               // Working directory
  },
  timeout: 120_000,  // Server startup timeout in ms (default: 60s)
});

Locator Rules

Resource content renders inside a double-iframe (outer sandbox proxy + inner app iframe). In e2e tests:

• Use result.app().locator(...) (from inspector.renderTool()) for resource content. This handles the double-iframe sandbox architecture.
• Use inspector.page.locator(...) only for inspector chrome elements (header, #root, sidebar controls).

Simulations

E2e tests consume simulation fixtures defined in tests/simulations/*.json. For sunpeak projects, simulations are part of the app project structure (see the create-sunpeak-app skill for the simulation file format). For non-sunpeak servers, renderTool connects to the live server via the configured server option.

Live Testing (against real ChatGPT)

Live tests validate MCP Apps inside real ChatGPT. They use Playwright to open the user's browser, send messages that trigger tool calls, and assert on the rendered app iframe.

Before the first run, make sure Developer mode is enabled. If needed, click the bottom-left user menu and select Settings > Security and login > Developer mode. Then return to the ChatGPT homepage, click Plugins in the sidebar, and open the app if it is already there or select + to create it with the tunneled /mcp URL. Set the app name to the package.json name because live tests use that name when invoking the app.

Live Test Pattern

One spec file per resource. Import test and expect from sunpeak/test/live -- the live fixture handles login, MCP refresh, and host-specific message formatting.

// tests/live/weather.spec.ts
import { test, expect } from 'sunpeak/test/live';
test('weather tool renders forecast', async ({ live }) => {
  const app = await live.invoke('show me the weather in Austin');
  await expect(app.locator('h1')).toBeVisible();
});

Config is a one-liner:

// tests/live/playwright.config.ts
import { defineLiveConfig } from 'sunpeak/test/live/config';
export default defineLiveConfig();
// Add hosts: defineLiveConfig({ hosts: ['chatgpt', 'claude'] })
// Generates one Playwright project per host. Tests switch themes internally via live.setColorScheme().

live Fixture API