calesthio/generative-media-skills/azure-speech

Azure Speech for media production agents

Use this skill when Azure Speech in Foundry Tools is a candidate provider for transcription, captions, subtitles, narration, synthetic voices, avatar speech, speech translation, video localization, or production speech QA.

Treat Azure Speech as a family of related services, not one model. Select the narrowest official capability that matches the job, then verify region, quota, language, pricing, and access status before committing.

Facts below were verified against Microsoft/Azure documentation on 2026-07-10. Volatile items such as supported locales, regions, pricing, quota values, API versions, and preview/limited-access status must be rechecked before paid or production use.

First decision: what job is the user actually asking for?

Documented facts:

• Azure Speech exposes speech to text, text to speech, speech translation, custom speech, custom voice, personal voice, text-to-speech avatar, video translation, LLM Speech, Speech SDK, Speech CLI, REST APIs, and containers in different combinations by region and access tier. Sources: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/overview and https://learn.microsoft.com/en-us/azure/ai-services/speech-service/regions
• Speech-to-text modes include real-time transcription, fast transcription, batch transcription, and custom speech. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-to-text
• Text-to-speech modes include real-time synthesis, asynchronous batch synthesis for long-form audio, standard/prebuilt neural voices, HD voices, custom voice, personal voice, and avatar outputs. Sources: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/text-to-speech and https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-synthesis

Production routing:

1. Live captions, live dictation, live customer-support assist, or interactive voice UI → real-time speech to text through Speech SDK, Speech CLI, or real-time REST where supported.
2. One prerecorded file where the editor needs quick transcript/subtitle timing → fast transcription API, if the file and region fit.
3. Many prerecorded files, long back catalog, storage-based processing, diarization at scale, or custom speech model use → batch transcription.
4. Narration, voiceover, accessibility read-aloud, or generated dialogue → text to speech with a selected standard/HD/custom/personal voice and SSML.
5. Audiobook, long lesson, long article, or thousands of narration segments → batch synthesis; do not force real-time synthesis for long-form work.
6. Source speech in one language with output text or spoken translation → speech translation or LLM Speech depending on latency, prompt-tuning, and output needs.
7. Existing video localization with translated video output → video translation.
8. Talking avatar driven by Azure Speech voice output → text-to-speech avatar. Use only when the avatar itself is part of the deliverable; otherwise generate audio and composite separately.
9. Domain jargon or acoustic mismatch causing transcription errors → phrase list/custom speech evaluation, then custom speech training if the measured gain justifies it.

Do not represent diarization as speaker identity. Azure Speech diarization labels speakers such as Guest1/Guest2 and is not designed to identify individuals. Source: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/speech-service/speech-to-text/transparency-note

Speech to text for captions, transcripts, and editorial logging

Real-time transcription

Use real-time transcription when the user needs immediate partial/final results for live captions, live notes, dictation, or a responsive voice application.

Documented facts:

• Real-time speech to text can produce intermediate results for live audio inputs and is available through Speech SDK, Speech CLI, and short-audio REST paths. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-to-text
• Real-time diarization is available through the real-time diarization workflow, but the short-audio REST API does not support real-time diarization. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-stt-diarization
• Real-time diarization had a documented Standard (S0) maximum audio length of 240 minutes per session on 2026-07-10. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-quotas-and-limits

Production heuristics:

• For live captions, optimize for latency and readability first; preserve a post-event correction pass for names, jargon, and sentence breaks.
• Ask whether captions are for accessibility, live monitoring, edit prep, or legal/archive transcript. These require different tolerances.
• For action-triggering voice interfaces, show or play back recognized text and ask for confirmation before irreversible actions.

Fast transcription

Use fast transcription when the user needs a synchronous transcript for one audio/video file with predictable latency, for example quick subtitles, edit prep, meeting notes, or voicemail.

Documented facts:

• Fast transcription returns results synchronously and is designed to be faster than real time. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/fast-transcription-create
• It returns display-form transcription, not lexical form. Display form includes human-readable punctuation and capitalization. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/fast-transcription-create
• As of 2026-07-10, fast transcription documented support included transcription, diarization, one/two-channel handling, profanity filtering, locale specification, phrase lists for the default mode, segment-level timestamps, and word-level timestamps; LLM Speech variants changed the feature matrix. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/fast-transcription-create
• Fast transcription had documented Standard (S0) limits of audio input size under 500 MB, audio length under 5 hours per file, and 600 requests per minute on 2026-07-10. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-quotas-and-limits

Production heuristics:

• Prefer fast transcription for a single urgent editorial job; prefer batch for a library, queue, or repeatable pipeline.
• Specify the known locale when possible to improve accuracy and latency. Use language identification only when genuinely unknown or multilingual.
• For subtitles, request word/segment timing, then run a subtitle segmentation pass; raw transcript timing is not automatically broadcast-ready line breaking.

Batch transcription

Use batch transcription when the user has many prerecorded files, long material, storage-based ingest, asynchronous processing, diarization, word-level timestamps, or a custom speech model.

Documented facts:

• Batch transcription submits audio, processes asynchronously, and stores results in a storage container for retrieval. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-transcription
• Batch is designed for large amounts of audio in storage; provide multiple files per request or point to an Azure Blob Storage container for concurrent processing. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-transcription
• Batch jobs are best-effort scheduled; Microsoft documents that at peak hours jobs might take up to 30 minutes to start and up to 24 hours to complete. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/batch-transcription-get
• On 2026-07-10, batch transcription documented limits included REST API 600 requests/minute, up to 1 GB audio input file size, up to 1,000 files per transcription request when using multiple content URLs, and 240 minutes per file when diarization is enabled. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/speech-services-quotas-and-limits
• Custom speech models can be used with batch transcription without deploying a hosted custom endpoint if only batch transcription is needed. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/custom-speech-overview

Production heuristics:

• Build the batch job around storage provenance: source URI/SAS, locale, model, diarization, word timestamps, profanity policy, TTL, output storage, and a retry plan.
• For archive media, separate ingest metadata from transcript text: file ID, show/episode, rights owner, input language, expected speakers, and noise notes.
• Poll asynchronously with exponential backoff and make the UI truthful: “queued/running/succeeded/failed,” not a fake progress bar.

Diarization and speaker separation

Documented facts:

• Diarization differentiates speakers and adds speaker information to transcribed segments. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/get-started-stt-diarization
• The Speech transparency note says diarization is not speaker recognition and cannot be used to identify individuals; guest labels are random per conversation. Source: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/speech-service/speech-to-text/transparency-note
• In data-processing documentation, speaker separation/diarization uses voice-characteristic signals temporarily for annotating output; signal data is discarded after processing and does not support tracking speakers across multiple files. Source: https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/speech-service/speech-to-text/data-privacy-security

Production heuristics:

• If the user needs “Alice vs Bob,” diarization alone is insufficient. Use it as a draft segmentation aid, then map labels to names manually or through a supervised editorial process.
• Diarization is most useful when speakers are acoustically separable. It degrades with overlap, heavy noise, distant mics, and many speakers.
• Keep separate QA tracks for word accuracy and speaker-turn accuracy; a transcript can be textually accurate while assigning words to the wrong speaker.

Captions and subtitles

Documented facts:

• Microsoft positions fast transcription for quick audio/video transcription, subtitles, and edit workflows. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/fast-transcription-create
• Azure Speech and Whisper-via-Azure-Speech are recommended by Microsoft for large-file batch transcription, diarization, and word-level timestamps, while Azure OpenAI Whisper is positioned for individual fast files and prompting/translation-to-English use cases. Source: https://learn.microsoft.com/en-us/azure/ai-services/speech-service/whisper-overview

Production heuristics:

• Generate captions in two passes: transcription/timing first, caption formatting second.
• Caption QA should check line length, reading speed, shot-change collisions, punctuation, speaker labels, music/SFX cues where required, profanity policy, and burned-in vs sidecar deliverables.
• For professional subtitles, do not trust raw punctuation blindly; edit sentence boundaries around meaning and cadence.
• If legal compliance matters, ask the user for the target standard, platform, or region rather than assuming an accessibility spec.

Text to speech for narration and synthetic voices

Voice selection

Documented facts: