Build Multimodal AI Pipelines

This skill designs and implements multimodal AI pipelines that integrate text, image, audio, and video processing. Architecture principles call for modular, swappable per-modality components with consistent interfaces and dependency injection, graceful degradation when a modality is unavailable, asynchronous processing for independent modality streams, memory-mapped files for large media, lazy model loading, and per-modality batching. It provides a preprocessing class that tokenizes text, runs a vision processor on images, converts audio to a mel-spectrogram via librosa, and samples and processes evenly-spaced video frames via OpenCV.

It covers two fusion architecture patterns: early fusion with cross-modal attention (projecting each modality to a shared dimension, running cross-attention between modality pairs, and combining the results through a fusion MLP) and late fusion with adaptive weighting (per-modality prediction heads combined through learned, input-dependent attention weights rather than fixed averaging). A production pipeline class processes a batch of multimodal inputs asynchronously, running each available modality's preprocessing in parallel via a thread pool and gracefully filtering out any modality that failed to process before running inference.

Model optimization guidance covers memory management (gradient checkpointing for large multimodal transformers, model sharding for distributed inference, INT8/FP16 quantization per modality, LRU-based model caching) and inference acceleration (cuDNN benchmarking, TorchScript compilation, automatic mixed precision). Error handling covers graceful degradation with confidence thresholding and per-modality performance logging, plus a quality-monitor class that validates output confidence and processed-modality count against configurable thresholds. Deployment guidance includes a CUDA-based Docker configuration and seven production best practices: circuit breakers for external model APIs, database connection pooling, per-client rate limiting, correlation-ID logging, health checks that validate every modality component, GPU memory monitoring with automatic scaling, and an A/B testing framework for model updates.

def process_text(self, text: str, max_length: int = 512) -> Dict[str, torch.Tensor]:
    return self.text_tokenizer(
        text,
        max_length=max_length,
        padding='max_length',
        truncation=True,
        return_tensors='pt'
    )

Use this skill when building a pipeline that needs to process and fuse multiple input modalities - choosing between early and late fusion, handling missing or failed modalities gracefully, optimizing memory and inference speed, or deploying the pipeline to production with proper monitoring.

It is not a fit for single-modality model pipelines, where the fusion architecture, per-modality graceful degradation, and multimodal quality monitoring this skill provides add unnecessary complexity.

Inputs are any combination of text, image, audio, and video data per request. Outputs are fused model predictions along with metadata on how many modalities were successfully processed, plus the supporting pipeline code: preprocessing, a chosen fusion architecture, optimization configuration, and production deployment and monitoring setup.

Built on Hugging Face Transformers (tokenizers and vision processors), librosa for audio, OpenCV for video frame extraction, PyTorch (including TorchScript and automatic mixed precision) for modeling, and a CUDA-based Docker image for GPU deployment.

ML engineers building production multimodal AI systems who need concrete patterns for modality-agnostic preprocessing, fusion architecture selection, graceful degradation when a modality fails, and the memory/inference optimizations and monitoring a multimodal pipeline needs at scale.