Optimize AI Models for Efficiency

AI Model Compression Expert Agent recommends and implements neural network compression strategies. It considers the trade-offs among model size, inference speed, and accuracy, and covers pruning, quantization, knowledge distillation, architecture optimization, and weight sharing. It tracks measures such as megabytes saved, latency per sample, throughput, accuracy tolerance, and memory bandwidth.

The quantization material distinguishes post-training dynamic quantization, static quantization with calibration, and quantization-aware training. The pruning material includes magnitude-based unstructured and structured pruning, plus a gradual sparsity schedule. The skill is intended to help users choose techniques and deployment optimizations for different neural network architectures.

Use this agent when a neural network must fit deployment constraints or improve inference performance while maintaining acceptable model quality. It can help compare compression approaches, plan measurements, and implement PyTorch techniques. It is useful when model size, latency, throughput, accuracy, or memory bandwidth are relevant constraints.

Compression decisions involve trade-offs. A smaller model is not automatically a better model if accuracy degradation exceeds the accepted tolerance or inference performance does not improve for the target deployment. Evaluating these competing metrics rather than optimizing a single number. Validate technique compatibility with the actual architecture and deployment environment before relying on a compressed model.

The examples use PyTorch and its quantization and pruning utilities. They reference torch.nn.Linear, torch.nn.Conv2d, torch.qint8, FBGEMM configuration, calibration loaders, and model validation. The skill also discusses knowledge distillation and efficient architecture families such as MobileNets and EfficientNets.

This agent is for machine learning engineers optimizing trained neural networks for deployment. It suits users who can provide architecture and workload context and measure model quality and runtime behavior. It helps frame compression as an empirical trade-off among efficiency and accuracy.

import torch
import torch.quantization as quant

Post-training dynamic quantization converts weights with limited calibration requirements, while static quantization uses representative calibration data before conversion. Quantization-aware training prepares the model for fake quantization during training, then validates and converts the prepared model. Examples use FBGEMM configuration and separate training and validation loaders. Pruning can remove individual weights or structured channels and filters; a gradual schedule increases sparsity over training rather than applying the entire target at once. Knowledge distillation trains a smaller student model to reproduce a teacher's outputs. After compression, compare accuracy against the accepted tolerance alongside model size, latency, throughput, and memory use. The deployment section covers ONNX export for cross-platform use and TensorRT optimization for building an optimized engine. The skill emphasizes measuring the resulting model because sparsity alone does not guarantee a faster runtime.