calesthio/generative-media-skills/assemblyai-transcription

AssemblyAI transcription for media-production agents

Use AssemblyAI when the production job needs reliable transcript artifacts from recorded or live speech, especially when the transcript must drive captions, clip selection, speaker turns, show notes, chaptering, translation, moderation, or compliance review.

Treat this as a provider skill, not a generic ASR primer. Check the live AssemblyAI docs before implementing code in production because model names, pricing, add-ons, rate limits, and data-retention settings are volatile. The facts below were verified against official AssemblyAI documentation on 2026-07-10.

Pick the product surface first

AssemblyAI exposes several surfaces that look similar but serve different production needs.

Documented facts:

• Pre-recorded STT is the main batch/async API for media files. Submit audio_url or upload a local file first, then poll GET /v2/transcript/{id} or use webhooks until the transcript is completed or error. The returned transcript includes text, metadata, words with millisecond timestamps, and optional feature outputs. Source: https://www.assemblyai.com/docs/pre-recorded-audio/getting-started/transcribe-an-audio-file
• Sync STT is for short clips in a single HTTP request, with no polling or session management. Official docs describe it as accepting audio clips from 80 ms to 120 s. Use it for short voice-agent turns, quick QC snippets, or small app interactions, not long podcasts or interviews. Sources: https://www.assemblyai.com/docs/sync-stt/getting-started/transcribe-a-short-audio-file and https://www.assemblyai.com/docs/sync-stt/audio-requirements
• Real-time STT streams audio over WebSocket (wss://streaming.assemblyai.com/v3/ws) and emits Begin, repeated Turn, and Termination events. Use it for live captioning, voice agents, live meeting assistants, or real-time production monitors. Word timings are in milliseconds. Source: https://www.assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio
• Streaming is billed for the duration the WebSocket is open, not the amount of audio sent. Unclosed sessions auto-close after 3 hours and can bill for the full session duration. Source: https://www.assemblyai.com/docs/billing-and-pricing
• Use temporary streaming tokens for browser clients; never expose the account API key in client-side code. Source: https://www.assemblyai.com/docs/streaming/getting-started/transcribe-streaming-audio

Production heuristic:

• For a finished media asset, prefer pre-recorded STT even if the recording started as a live session. Async models can use the full file context, which is better for final captions, transcript cleanup, chaptering, and quote extraction.
• For live captions or voice agents, use real-time STT during the event and optionally re-transcribe the final recording asynchronously afterward for the publication-grade transcript.
• Use Sync STT only when the clip is deliberately short and you need immediate response without job lifecycle overhead.

Model and feature selection

Documented facts verified 2026-07-10:

• Pre-recorded models include Universal-3.5 Pro and Universal-2. AssemblyAI documents Universal-3.5 Pro as the highest-accuracy/fastest pre-recorded model with 18 languages, native code switching, contextual prompting, and up to 1,000 keyterms; Universal-2 supports 99 languages and up to 200 keyterms. Source: https://www.assemblyai.com/docs/getting-started/models
• Streaming models include Universal-3.5 Pro Streaming, Universal-Streaming Multilingual, and Universal-Streaming English. Universal-3.5 Pro Streaming supports 18 languages and up to 100 keyterms; Universal-Streaming Multilingual supports English, Spanish, Portuguese, German, French, and Italian; Universal-Streaming English is English-only. Source: https://www.assemblyai.com/docs/getting-started/models
• Medical Mode is an add-on (domain: "medical-v1") for medical terminology and is billed separately. AssemblyAI documents it for pre-recorded and streaming models in English, Spanish, German, and French. Source: https://www.assemblyai.com/docs/getting-started/models
• Current posted model prices in the docs on 2026-07-10 were: pre-recorded Universal-3.5 Pro at $0.21/hr, Universal-2 at $0.15/hr; streaming Universal-3.5 Pro Streaming at $0.45/hr, Universal-Streaming Multilingual at $0.15/hr, Universal-Streaming English at $0.15/hr. Always re-check pricing before estimating costs. Sources: https://www.assemblyai.com/docs/getting-started/models and https://www.assemblyai.com/pricing

Decision rules:

• Use Universal-3.5 Pro for final English or supported-language interviews, podcasts, social clips, and captions when accuracy and diarization quality matter.
• Use Universal-2 when broad language coverage or lower cost matters more than the newest model.
• Use streaming Universal-3.5 Pro for real-time agent assist, live captions with difficult vocabulary, multilingual code-switching, or speaker-diarized live meetings.
• Use the cheaper streaming models for cost-sensitive live captions where the language set is simple and latency matters more than post-production accuracy.
• Add keyterms or prompting for names, brand vocabulary, show titles, technical terms, product names, spellings, and recurring phrases. Keep keyterms specific; do not stuff generic vocabulary.

Inputs and artifact custody

Documented facts:

• Local files are uploaded to /v2/upload, and the returned upload_url is submitted as audio_url. For cURL uploads, stream raw bytes with --data-binary @file; sending JSON or a path string can create downstream transcoding failures. Source: https://www.assemblyai.com/docs/pre-recorded-audio/getting-started/transcribe-an-audio-file
• audio_start_from and audio_end_at can constrain a transcript to part of a file in pre-recorded jobs. Source: https://www.assemblyai.com/docs/api-reference/transcripts/submit
• Pre-recorded transcript statuses include queued, processing, completed, and error. A queued status can indicate waiting for a processing slot. Source: https://www.assemblyai.com/docs/pre-recorded-audio/check-transcript-status
• Delete transcripts with the transcript delete endpoint; store the transcript_id with every production artifact so deletion, audit, support, and retries are possible. Source: https://www.assemblyai.com/docs/api-reference/transcripts/delete

Production heuristic:

• Before upload, normalize production media to an audio-only mezzanine for repeatability: WAV or FLAC for master archival, 16 kHz mono WAV for telephony-like speech, or a high-quality compressed copy when transfer size matters. Preserve the original media separately.
• Keep a job ledger with: source asset path/hash, speaker/channel notes, consent status, request JSON, region endpoint, model, feature flags, transcript ID, submission time, completion time, cost estimate, output file paths, and deletion deadline.
• Do not use ephemeral AssemblyAI URLs as your only source of truth. Download subtitles, redacted audio, transcript JSON, and any derived edit artifacts into your production storage.

Speaker separation and identity

Documented facts:

• Pre-recorded speaker diarization is enabled with speaker_labels: true. Results appear as utterances, each with speaker label, text, start/end in milliseconds, confidence, and word-level details. Source: https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers
• Diarization labels are generic sequential labels such as A, B, C. Speaker Identification can map labels to known names or roles through Speech Understanding when you provide known values. Source: https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers
• AssemblyAI documents speakers_expected, speaker_options.min_speakers_expected, and speaker_options.max_speakers_expected as hard boundaries, not soft hints. Over-tight caps merge speakers; overly high caps can over-split. Source: https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers
• AssemblyAI recommends at least 30 seconds of continuous speech per speaker for best diarization results and warns that crosstalk, noise, similar voices, and short backchannels reduce accuracy. Source: https://www.assemblyai.com/docs/pre-recorded-audio/label-speakers
• Streaming supports diarization and multichannel workflows, but these are live session features with real-time constraints, not replacements for final editorial speaker QA. Source: https://www.assemblyai.com/docs/streaming/label-speakers-and-separate-channels
• Multichannel pre-recorded transcription bills each channel separately. Source: https://www.assemblyai.com/docs/billing-and-pricing

Production heuristic:

• For podcast/interview post-production, prefer isolated channels when available. Use multichannel when each speaker is reliably on a separate channel; use diarization when the recording is mono or channels are mixed.
• If exact speaker count is known from production notes, set it; otherwise use a plausible min/max range with slight headroom.
• Always perform human speaker-label QA before publishing captions, pull quotes, clips, or legal/medical notes.

Captions, subtitles, timestamps, and edit data

Documented facts:

• Pre-recorded transcript responses include word-level start/end timestamps by default in the words array; utterances also include start/end when diarization is enabled. Source: https://www.assemblyai.com/docs/pre-recorded-audio/getting-started/transcribe-an-audio-file
• Subtitles are exported from completed transcripts as SRT or VTT through GET /v2/transcript/{transcript_id}/{subtitle_format}. Source: https://www.assemblyai.com/docs/api-reference/transcripts/get-subtitles
• Sentence and paragraph endpoints are available for completed transcripts. Source: https://www.assemblyai.com/docs/api-reference/transcripts/get-sentences and https://www.assemblyai.com/docs/api-reference/transcripts/get-paragraphs
• AssemblyAI has guides for custom-length subtitles and speaker-labeled subtitles. Source index: https://www.assemblyai.com/docs/llms.txt
• Sync STT returns word-level details in the words array with millisecond start/end timestamps and per-word confidence in the response for short-file requests. Source: https://www.assemblyai.com/docs/sync-stt/getting-started/transcribe-a-short-audio-file

Production heuristic:

• For broadcast-style captions, treat AssemblyAI’s SRT/VTT as a first timed draft, then enforce platform style: reading speed, line length, sentence segmentation, speaker labels, profanity policy, and house capitalization.
• For social burned-in captions, use word timestamps for karaoke/highlight animation but smooth micro-timing in the compositor. Do not animate every disfluency unless the style intentionally preserves verbatim speech.
• For edit decisions, export a durable transcript JSON plus EDL-like spans: speaker, start_ms, end_ms, text, confidence, source_transcript_id, and reviewer notes.

Languages, code-switching, and translation

Documented facts: