Design and Deploy Production-Ready ML API Endpoints

This skill builds production-ready machine learning API endpoints, primarily with FastAPI. Design fundamentals cover stateless request design, consistent success/error response formats, model versioning for backward compatibility, rigorous input validation before inference, and async patterns for I/O-bound operations. Model serving architecture covers loading the model once at startup rather than per-request, batching requests where possible, caching predictions for repeated inputs, and monitoring CPU/memory/GPU usage.

A basic endpoint loads the model in a FastAPI startup event handler, validates input feature count via a Pydantic validator, and returns a typed PredictionResponse with a request ID for tracing. A batch endpoint validates an upper bound on instance count before running inference over the whole batch at once.

class PredictionInput(BaseModel):
    features: List[float]
    @validator('features')
    def validate_features(cls, v):
        if len(v) != 10:
            raise ValueError('Expected 10 features')
        return v

Error handling defines a typed ErrorCode enum (invalid input, model error, rate limit, internal error) and a custom exception handler translating validation errors into a structured ErrorResponse with a 400 status. Performance optimization implements a TTL-based prediction cache keyed by an MD5 hash of the input features, returning cached results within the TTL window and computing/caching fresh predictions otherwise. Monitoring covers a /health endpoint reporting model-loaded status and a /metrics endpoint exposing request count, average prediction latency, and error rate. Security is covered via HTTPBearer-based API key authentication injected as a FastAPI dependency, rejecting invalid tokens with a 401. Deployment configuration includes a multi-worker uvicorn Dockerfile and a pydantic Settings class loading model path, batch size limit, cache TTL, and worker timeout from environment variables. Best practices reinforce async I/O, thorough input validation, response caching for deterministic models, graceful degradation on model failure, structured logging of predictions/latency/errors, multi-version model support, resource limits to prevent exhaustion, and both unit and integration test coverage.

Use this skill when building or hardening a production ML model-serving API - implementing input-validated prediction endpoints (single or batch), TTL-based prediction caching, API key authentication, health/metrics endpoints, or containerized deployment configuration.

It is not the right tool for ad-hoc, notebook-based model inference with no API surface needed, or for very high-throughput/low-latency serving requirements better served by a dedicated model-serving platform (TensorFlow Serving, Triton, KServe) rather than a hand-rolled FastAPI wrapper.

Input: a trained model (loaded via joblib or similar) and its expected input feature schema, plus the required auth, caching, and monitoring needs. Output: a FastAPI application with validated single/batch prediction endpoints, structured error responses, TTL-based prediction caching, API key authentication, health/metrics endpoints, and Docker/environment-based deployment configuration.

Built on FastAPI and Pydantic for request/response validation, joblib/numpy for model loading and inference, uvicorn as the ASGI server, and Docker for containerized deployment with environment-driven settings.

ML engineers and backend developers deploying trained models as production APIs - particularly those needing input validation, prediction caching, authentication, and observability around a served model.