Compare LLM outputs across 300+ models via unified API

This promptfoo example uses AI/ML API, which provides access to 300+ models through a single API key, to compare three different language models - DeepSeek R1 (advanced reasoning), GPT-4.1 Mini (fast and cost-effective), and Claude 4 Sonnet (balanced performance) - on a joke-telling task. Each model is asked to tell jokes about different topics, and results are evaluated for whether the joke contains relevant keywords and whether it's actually funny, judged by an LLM.

Use this example as a template for comparing models from different providers (OpenAI, Anthropic, DeepSeek) side by side without managing separate API keys for each, since AI/ML API exposes them all through one key. It requires an AI/ML API key from aimlapi.com; the specific joke-telling task is illustrative, meant to be swapped out for whatever comparison task you actually need.

npx promptfoo@latest init --example provider-aiml-api
cd provider-aiml-api

export AIML_API_KEY=your_api_key_here

npx promptfoo@latest eval

Results are scored on two dimensions: keyword relevance to the requested joke topic, and an LLM-judged funniness assessment, giving both a mechanical check and a qualitative one for the same output.

Set AIML_API_KEY, then reference any of AI/ML API's 300+ supported models by name in the provider config - no separate provider setup per model vendor is needed.

Teams that want to benchmark models from multiple providers side by side through one API key, using this joke-telling comparison as a starting template for their own evaluation criteria.