Analyze Content Across Documents, Audio, Video, and Images

This skill covers the Azure AI Content Understanding SDK for Python, which analyzes documents, images, audio, and video to extract structured content. It ships five prebuilt analyzers: prebuilt-documentSearch (extracts markdown from documents for RAG applications), prebuilt-imageSearch (extracts content from images), prebuilt-audioSearch (transcribes audio with timing), prebuilt-videoSearch (extracts frames, transcripts, and summaries from video), and prebuilt-invoice (extracts invoice fields specifically).

It covers analyzing documents (from a file or URL, accessing markdown content and document content details), images, video (accessing AudioVisualContent, transcript phrases with timing, and key frames), and audio (accessing the transcript). Beyond prebuilt analyzers, it supports creating custom analyzers with a defined field schema built on top of a base analyzer - for example, a custom invoice analyzer extending prebuilt-documentSearch with fields like vendor_name, invoice_total, and line items - plus analyzer management (list, get, delete custom analyzers) and both sync and async clients.

Use this when extracting structured content from documents, images, audio, or video in a Python application - whether via a prebuilt analyzer for common cases (RAG document ingestion, invoice extraction, video transcription) or a custom analyzer with a specific field schema for specialized extraction needs.

Not the right skill for pure content moderation/safety scoring (see Azure AI Content Safety instead) or for languages/SDKs other than Python.

Inputs: a document, image, audio, or video file (or URL), and either a prebuilt analyzer ID or a custom analyzer with a defined field schema.

Outputs: extracted markdown content (for RAG), extracted image content, audio/video transcripts with timing and key frames, or custom-schema-extracted fields (e.g. invoice vendor name, total, line items) depending on the analyzer used.

Install with:

pip install azure-ai-contentunderstanding

Authenticate and run a document analysis:

import os
from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.identity import DefaultAzureCredential

endpoint = os.environ["CONTENTUNDERSTANDING_ENDPOINT"]
credential = DefaultAzureCredential()
client = ContentUnderstandingClient(endpoint=endpoint, credential=credential)

Content Understanding operations are asynchronous long-running operations: begin the analysis with begin_analyze() (returns a poller), poll for results with .result(), then process the structured results from AnalyzeResult.contents.

Azure AI Content Understanding service, Python (sync and async clients). Extracted content is typed by class: DocumentContent (for PDF, images, and Office docs) provides pages, tables, figures, and paragraphs, while AudioVisualContent (for audio and video) provides transcript phrases, timing, and key frames - both derive from a shared MediaContent base that provides basic info and a markdown representation.

Documented best practices: use begin_analyze with AnalyzeInput (the correct method signature), access results via result.contents[0] since results are returned as a list, prefer prebuilt analyzers for common scenarios and reserve custom analyzers for domain-specific field extraction, use the async client for high-throughput scenarios, plan for long-running operations since video/audio analysis can take minutes, and use URL sources where possible to avoid upload overhead.

Python developers building RAG ingestion pipelines, invoice/document extraction, or audio/video transcription and summarization who need prebuilt or custom multimodal content extraction.