gatewai-dev/framefields/framefields-render

framefields-render

Setup

Import everything from the `framefields` npm package. If the project doesn't have framefields set up yet, follow the framefields skill first (install, project scaffold, first render).

Overview

framefields renders headlessly in Node.js on top of native WebGPU (Dawn), encoding through FFmpeg with a hardware encoder when one works (NVENC, VideoToolbox) and a software one otherwise. It does not run a browser, Puppeteer, or Chromium.

Because video rendering is resource-intensive and native GPU pipelines run asynchronously in worker pools, framefields workflows enforce a strict verification ladder: inspect inexpensive intermediate artifacts first before committing to a full video render.

The 5-Step Verification Ladder

Always proceed through the ladder in order. Catching a layout or timing bug at step 1 or 2 takes seconds; catching it after a 30-second 4K video render wastes minutes.

Step 1: Static Typecheck & Schema Validation

Before touching the GPU, verify TypeScript types and parse the composition against the strict runtime schema:

import { CompositorProgramSchema } from "framefields";
// 1. Compile spec
const spec = comp.toSpec();
// 2. Validate against strict schema
const parsed = CompositorProgramSchema.safeParse(spec);
if (!parsed.success) {
  console.error("Schema validation failed:", parsed.error.format());
  process.exit(1);
}
// 3. Lint: things that render without error but not as intended
for (const d of comp.lint()) console.log(`${d.severity} ${d.code}: ${d.message}`);

Typecheck the project:

npx tsc --noEmit -p .

In projects initialized with the current CLI, pnpm check runs typechecking and document preflight. For another entry, run pnpm exec framefields check src/film.ts --export buildFilm --json. The export is a Composition or a builder returning one; importing it executes project code, so keep rendering, generation and writes out of the builder. The CLI runs it in an isolated process, routes builder logs to stderr and writes one versioned JSON result to stdout. Errors fail the command; warnings fail with --warnings-as-errors. --timeout 60000 bounds the worker lifetime.

pnpm check:render adds GPU verification, a frame and a three-point grid under output/check; --output <dir> changes the location. This smoke check does not replace targeted pixel assertions or video QA. A failed or unavailable GPU check is reported as a failure. Use pnpm exec framefields api for timing, grid and preflight type declarations generated from this SDK version.

Step 2: Single Frame Inspection (renderFrame)

Render isolated keyframe points (e.g. entry, peak action, exit) directly to PNG files on disk to inspect alignment, typography, colors, and opacity:

import fs from "node:fs/promises";
import path from "node:path";
import { HeadlessMediaRenderer } from "framefields";
const renderer = new HeadlessMediaRenderer();
// Render frame 45 (1.5s at 30fps)
const pngBuffer = await comp.renderFrame({
  frame: 45,
  renderer,
});
await fs.writeFile("output/frame-45.png", pngBuffer);

atMs takes precedence when both frame and atMs are supplied, including for frame hooks. Frame indices must be nonnegative safe integers; fractional timestamps and FPS are supported. A composition's FPS is authoritative: a different render/grid FPS override throws. Media pipelines can still select their rendering FPS.

Step 3: Motion & Layout Stability Contact Sheets (renderFrameGrid)

To verify easing curves, cut points, kinetic sweeps, or text wrapping without encoding a full video, generate a sequential frame grid image:

import fs from "node:fs/promises";
import { renderFrameGrid } from "framefields";
// Samples frames across the intro sweep (frames 0 to 60)
const gridPng = await comp.renderFrameGrid({
  frames: [0, 6, 12, 18, 24, 30, 36, 42, 48, 54, 60],
  columns: 4,
  cellWidth: 480, // Downscaled thumbnail size for rapid visual scanning
  showLabels: true, // frame number and timestamp under each cell
});
await fs.writeFile("output/intro-grid.png", gridPng);

Contact sheets immediately reveal:

• Motion hitches and non-smooth easing deceleration.
• Text clipping or wrapping jumps across keyframe boundaries.
• Bad layer z-indexing or missing cut transitions.

Grid frame ranges are inclusive, but the composition's duration is exclusive. Automatic sampling stops before the exported frame count (max(1, round(durationMs * fps / 1000))). Explicit points may sample beyond the export window for diagnostics. Sample order and duplicates are preserved. Frame steps must be positive safe integers, timestamp steps finite and positive. The sample limit is checked while generating points (64 by default, at most 120); invalid or excessive requests fail before creating a large list. Labels describe the actual selected frame, with timestamps formatted for display only.

Step 4: Programmatic Conformance Probes in Vitest

Write automated headless vitest tests that assert pixel colors, transparency, or layout bounding boxes:

import { describe, it, expect } from "vitest";
import { HeadlessMediaRenderer } from "framefields";
import { buildFilm } from "../film.js";
describe("Hero Scene Conformance", () => {
  it("renders non-black title text on frame 30", async () => {
    const comp = await buildFilm();
    const renderer = new HeadlessMediaRenderer();
    const png = await comp.renderFrame({ frame: 30, renderer });
    expect(png.length).toBeGreaterThan(1000);
    // Use skia-canvas or pixel probes to assert color at target coordinates
  });
});

Step 4.5: Show the User Before Exporting (preview)

When a person is in the loop, let them watch the composition before you spend time on the final render. startPreview serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code, the project's files, and the soundtrack mixed by the export engine. Point it at the module that builds the composition:

import { startPreview } from "framefields";
const session = await startPreview(
  { entry: new URL("./film.ts", import.meta.url), export: "buildFilm" },
  { title: "Launch film" },
);
console.log(`Preview at ${session.url}`);
await session.closed; // keep the process alive while the user watches

• The export is a function returning the `Composition` (it may be async), or a Composition itself. The browser calls it, so build-time Node code in it must stay simple: node:path, fileURLToPath(import.meta.url) and synchronous fs reads (JSON, text) work through shims; writes are skipped. Assets referenced by absolute path load from the server, limited to the project that holds the entry (its nearest .git or package.json); pass root with one or more paths for assets kept outside it (e.g. a shared fonts folder).
• Run it in the background, and read its output. Start the script as a background command with the longest timeout your harness allows, tell the user the URL, and ask for feedback in the chat. The command's output (stderr, prefixed [framefields preview]) says when the page opened, reports each distinct error from the page once (a build error, a frame that threw, a font or renderer that failed to load, no WebGPU), and says why it stopped. Check it after the page opens instead of assuming the film drew. log redirects or silences these lines, and session.closed resolves with the reason: "idle", "replaced" or "closed".
• One tab for the whole loop. The port is fixed per project directory. Running the preview again (after you've made changes) takes over from the one already running, and a tab left open reconnects and reloads into the new version by itself, at the moment it was showing. Don't ask the user to reopen anything. When the user closes the tab, the preview stops by itself about 5 s later (a reload doesn't stop it; idleCloseMs changes the delay). If the preview was stopped (tab closed, time limit, or you killed it), just start it again.
• Notes pinned on the picture are the user's precise feedback. Paused, the user clicks the frame (or drags an area) and writes what should change; the note keeps the frame, the spot as fractions of the frame (x, y, and w, h for an area) and the text. Each note prints in the output (note 3 at 0:12.50 (frame 750), spot 42%,31%: …) with the path of note-<id>.jpg, the frame with the spot marked, and all of them are in .framefields/preview-notes/notes.json (.framefields ignores itself in git; notesDir moves it, false turns notes off). To act on them: read notes.json, open each open note's picture to see exactly what it points at, find the layer in the code from the moment and the spot, make the change, and run the preview again. The tab comes back with the notes, so the user can check them and tick them off. You can also set "done": true in notes.json yourself. A chat that can't read files gets the same list from the Copy button in the Notes panel.
• Point at a moment. #t=12.5 in the URL opens the player at 12.5 s (#frame=750 by frame), and the address follows the playhead when it stops, so a link you give the user lands where you mean.
• Drive it from a script, not with clicks. If you can run JavaScript in the page (an agent browser), use window.framefieldsPreview: await seek(seconds) or await seekFrame(n) resolve once that frame is drawn (take the screenshot then), play() and pause() control playback, notes() lists the notes, and state() returns { state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, playbackFps, renderMs, meta }. <html data-preview="ready"> marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.
• Show it where the user is. startPreview only serves the page and returns the URL. If your app has a built-in browser, open the URL there. Otherwise give the user the link, or pass open: true to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).
• Playback runs in real time, on any device. Play draws the frame due on the clock and skips the ones it missed, so it starts at once and the sound keeps its speed; a slow device shows fewer frames. Next to the timecode the page shows the frame rate playback reaches, apart from the film's own fps; state() reports it as playbackFps, and renderMs is how long the last frame took to draw. Paused, the page draws the exact frame under the playhead. Vision effects analyse each frame from the one drawn before it: while paused, the page analyses up to 20 s ahead so that stretch plays exactly, but past it a device that skips frames shows cutouts and tracking from the last frame it drew. Check them paused, or in the export.
• Audio is the same mix renderVideo muxes (audio: false skips it). Playback uses the soundtrack as its clock; when a frame takes long (a video seek, a heavy shot), the picture skips ahead to catch up with the sound.

Step 5: Full Video Export (renderVideo)

Once static frames and motion grids are verified, encode the final MP4/WebM video:

import path from "node:path";
const result = await comp.renderVideo({
  outputPath: "output/final-video.mp4",
  quality: "high", // "draft" | "medium" | "high" | "lossless"
  codec: "avc",    // "avc" (H.264) | "hevc" (H.265) | "vp9" | "av1"
  audioCodec: "aac",
  concurrency: 4,  // Parallel frame encoding threads
});