Optimize AI Models for Efficiency

The AI Model Compression Expert Agent covers the core techniques for shrinking neural networks while managing the tradeoff between model size, inference speed, and accuracy: pruning (structured and unstructured removal of redundant weights/neurons), quantization (reducing numerical precision to INT8, INT4, or mixed precision), knowledge distillation (training a smaller student model to mimic a larger teacher), architecture optimization (efficient designs like MobileNets and EfficientNets), and weight sharing. For quantization it provides PyTorch code for post-training dynamic quantization, static quantization with a calibration loop via torch.quantization.prepare/convert, and quantization-aware training (QAT) with fake-quantization during the training loop. For pruning it covers magnitude-based unstructured pruning with torch.nn.utils.prune, structured channel/filter pruning by L2 norm, and a GradualPruner class that ramps sparsity from an initial to a final value over a configurable step range using a cubic progress curve. For knowledge distillation it supplies a DistillationLoss module combining a temperature-scaled KL-divergence soft-target loss with standard cross-entropy, plus the teacher-student training loop. It also documents architecture-specific strategies (attention-head pruning and layer removal for Transformers, channel pruning and depthwise separable convolutions for CNNs, weight sharing and recurrent-connection pruning for RNNs), a five-step compression pipeline (baseline, gradual compression, fine-tuning, validation, hardware profiling), and deployment code for exporting a compressed model to ONNX and building a TensorRT engine with FP16 precision.

Use it when a trained neural network needs to be made smaller or faster for deployment: choosing between pruning, quantization, and distillation, implementing PyTorch quantization or pruning code, designing a gradual sparsity schedule, or exporting a compressed model to ONNX/TensorRT for production inference. It also covers evaluating compression quality via accuracy metrics (Top-1/Top-5, F1, BLEU), efficiency metrics (FLOPs reduction, memory usage, inference time), and robustness/A-B testing. It is not for training a model from scratch, and it is not a substitute for architecture-specific judgment - the skill explicitly warns against applying generic compression techniques without considering the target architecture, and against over-aggressive compression that causes accuracy collapse.

Inputs are a trained PyTorch model plus calibration/validation data loaders. Outputs are a compressed model (pruned, quantized, or distilled) with measured size reduction, inference latency, throughput, and accuracy-degradation figures against baseline, and deployment artifacts such as an exported ONNX file or a TensorRT engine configured for FP16 inference.

PyTorch and its quantization/pruning APIs (torch.quantization, torch.nn.utils.prune), ONNX for cross-platform model export, and TensorRT (via the Builder/BuilderConfig API with FP16 precision) for optimized GPU deployment.

ML engineers preparing a trained model for production deployment on latency- or memory-constrained hardware, who need concrete PyTorch implementations of pruning, quantization, and distillation rather than a general theory overview.