calesthio/generative-media-skills/amazon-transcribe

Amazon Transcribe

Use this skill when an agent needs to plan, run, review, or troubleshoot Amazon Transcribe as a speech-to-text provider inside an AWS-controlled workflow. Treat Amazon Transcribe as an AWS custody option first: it is strongest when the media already belongs in S3, the organization needs IAM/KMS/CloudTrail governance, captions must be generated from batch files, or real-time transcripts must flow through AWS streaming endpoints.

Do not use this skill as a generic "best transcription model" ranking. Compare accuracy, latency, cost, language coverage, and custody requirements against the actual job. For healthcare, call-center analytics, and sensitive audio, separate the general Transcribe path from the Medical and Call Analytics products before choosing.

Facts below were verified against AWS documentation on 2026-07-10 unless a line says otherwise.

Choose the right Amazon Transcribe surface

Documented fact: Amazon Transcribe is an automatic speech recognition service that converts audio to text. It can be used through asynchronous batch jobs for media files or real-time streaming for live audio. Sources: What is Amazon Transcribe, Streaming audio, StartTranscriptionJob API.

Use this decision map:

• Batch transcription: use for files in Amazon S3, offline transcripts, subtitles, captions, podcast/video backlogs, archives, QA review, and workflows that can wait for job completion. Batch requires the media file to be in S3 before StartTranscriptionJob.
• Streaming transcription: use for live captions, call assistants, meetings, voice input, or low-latency partial/final transcripts. Streaming uses bidirectional HTTP/2 or WebSockets and requires a language choice or language identification, media encoding, and sample rate. Source: StartStreamTranscription API.
• Call Analytics: use only when the media is a customer-agent or sales/support call and the desired output includes call-specific insights such as turn-based output, sentiment, interruptions, non-talk time, talk speed, categories, summaries, or action items. Do not use it as a generic captioning path. Sources: Call Analytics, post-call output.
• Medical: use only for medical-related speech such as clinician dictation, telemedicine, or clinician-patient conversations. Amazon Transcribe Medical is available for batch and streaming, is US English (en-US) only, and AWS says it is not a substitute for professional medical advice, diagnosis, or treatment; patient-care uses require trained human review. Source: Amazon Transcribe Medical.

Production heuristic: for edited media and caption deliverables, prefer batch unless the user explicitly needs live text. For two-person recordings with each speaker isolated on a channel, prefer channel identification over diarization. For unknown single-channel speakers, use diarization and plan a human speaker-name pass.

Input and output contract

Documented facts:

• Batch input formats include AMR, FLAC, M4A, MP3, MP4, Ogg, WebM, and WAV; streaming formats include FLAC, Ogg Opus, and PCM encoding. AWS recommends lossless formats such as FLAC or WAV with PCM 16-bit encoding where possible. Source: Data input and output.
• Amazon Transcribe supports single-channel and dual-channel media. Media with more than two channels is not supported for standard Transcribe. Sources: Data input and output, channel identification.
• Batch transcript output is JSON. The JSON includes a paragraph transcript and item-level data; at minimum, word items include start time, end time, and confidence score. Source: Data input and output.
• Batch video subtitle output supports WebVTT (.vtt) and SubRip (.srt). When subtitles are requested, AWS produces both the subtitle file(s) and the regular JSON transcript in the same destination. Source: Creating video subtitles.

Production heuristic:

• Normalize production audio before upload: extract the final mix intended for captions, downmix only if you do not need channel identification, remove long silence where editorially safe, and keep a copy of the exact audio used to generate captions.
• Use the JSON transcript as the source of truth for QA and downstream alignment. Treat AWS-generated SRT/VTT as a starting point; review line breaks, reading speed, punctuation, brand terms, speaker changes, and shot-change collisions before delivery.
• For subtitle accessibility, plan a separate editorial pass for non-speech information such as [music], [laughter], or meaningful sound effects. Amazon Transcribe's transcript is speech-to-text, not a full human caption authoring system.

Feature boundaries that matter in production

Speaker labels versus channel identification

Documented facts:

• Speaker diarization partitions speakers and can label up to 30 unique speakers as spk_0 through spk_29; it is available for batch and streaming. Source: Partitioning speakers.
• Channel identification transcribes two-channel audio separately and labels channels ch_0 and ch_1; Amazon Transcribe does not support more than two channels. Source: Transcribing multi-channel audio.

Production heuristic:

• Use channel identification for call-center stereo, interviewer/interviewee split tracks, or any file where each speaker side has a stable separate channel.
• Use diarization for one-channel meetings, panels, podcasts, or camera audio, but do not promise human names. Build a mapping pass: spk_0 = host, spk_1 = guest, and verify the mapping after speaker turns change.
• Do not enable both channel identification and diarization unless the current API documentation explicitly supports the exact combination for the chosen request type. If you need both "which channel?" and "which person?", create separated workflows or post-process manually.

Custom vocabulary, custom language models, and vocabulary filters

Documented facts:

• Custom vocabularies improve recognition of specific words such as brand names, acronyms, proper nouns, and domain terms. They can be used with all supported languages, but only characters in the language's supported character set are allowed. Source: Custom vocabularies.
• AWS says table format is the preferred custom vocabulary format. Tables contain Phrase, SoundsLike, IPA, and DisplayAs; Phrase is required and must not contain spaces. Source: Creating a custom vocabulary using a table.
• Custom vocabulary filters modify exact word matches using a selected method and can be applied to batch and streaming requests. Source: Vocabulary filtering.
• When language identification is combined with custom vocabularies, custom language models, or vocabulary filters, LanguageIdSettings is required for each language; multi-language identification does not support custom language models. Source: Batch language identification.

Production heuristic:

• Use a custom vocabulary for "please recognize this correctly"; use a vocabulary filter for "mask, remove, or flag this exact word if it appears."
• For brand-safe captions, custom vocabulary and vocabulary filters solve different problems. A product name belongs in vocabulary; profanity or embargoed terms belong in a vocabulary filter.
• Build a vocabulary from the script, guest names, product names, acronyms, unusual spellings, and campaign terms before the first production job. Re-run a short representative audio sample to verify it helps rather than hurts.
• For multilingual or dialect-ambiguous media, constrain LanguageOptions when possible. AWS warns that mismatched language options can produce inaccurate transcripts, and dialect mismatch can prevent language-specific customizations from applying.

Language identification

Documented facts:

• Batch supports automatic language identification and automatic multiple-language identification. Source: Batch language identification.
• Streaming supports automatic language identification through the console, HTTP/2, or WebSockets. Source: Streaming language identification.
• Supported languages and language-specific features vary; AWS tells users to verify that the feature they need is supported for the language before proceeding. Source: Supported languages and language-specific features.

Production heuristic:

• If the user knows the language, specify it. Use language identification when language is genuinely unknown or the workflow must accept multiple languages.
• For regulated redaction, do not rely on broad auto language detection unless the redaction feature supports every likely language. AWS documents that when language identification is combined with content redaction, only supported redaction languages are redacted and unsupported languages may not generate warnings or failures.

PII, PHI, redaction, and toxicity

Documented facts:

• Streaming PII redaction replaces each identified PII instance with [PII]. Source: PII redaction in real-time streams.
• Batch language identification plus content redaction has a critical boundary: if the audio contains languages other than supported redaction languages, only supported-language content is redacted and other languages are not redacted without warnings or job failure. Source: Batch language identification.
• Toxicity Detection identifies and classifies voice-based toxic content across categories such as sexual harassment, hate speech, threat, abuse, profanity, insult, and graphic content; AWS says it uses audio and text cues. It is available only for batch transcriptions in US English (en-US). AWS's console workflow also notes that PII redaction can be enabled with Toxicity Detection, but other listed options are not supported with it. Sources: Detecting toxic speech, Using toxic speech detection.
• Amazon Transcribe Medical is HIPAA eligible according to AWS documentation, but Medical output intended for patient care must be reviewed by trained medical professionals. Sources: Amazon Transcribe Medical, Amazon Transcribe FAQ.

Production heuristic: