Compare AI Models Across Providers

This promptfoo example benchmarks four models hosted on Azure AI Foundry side-by-side: OpenAI gpt-5.1, Anthropic claude-sonnet-4-6, Meta Llama-4-Maverick-17B-128E-Instruct-FP8, and Mistral Mistral-Large-2411. Running npx promptfoo@latest eval sends the same test questions to all four deployments and scores their responses, letting you compare quality, cost, and behavior across providers without writing custom evaluation code.

Use it when you have already deployed multiple provider models in the same Azure AI Foundry project and want a direct, apples-to-apples comparison on your own tasks - benchmarking accuracy, evaluating cost-effectiveness, choosing a model for a project, or A/B testing after a model update. Do not use it if you have not yet deployed the models in Azure AI Foundry, since each provider's deployment and endpoint must exist and be configured with its own apiHost in promptfooconfig.yaml before the comparison can run.

Setup requires the AZURE_API_KEY environment variable plus a per-provider apiHost entry in promptfooconfig.yaml pointing at each model's deployment. Inputs are the test questions defined in the config; outputs are graded evaluation results viewable via npx promptfoo@latest view, covering response quality alongside cost and performance metrics across the four models.

Runs through the promptfoo CLI (npx promptfoo@latest init --example azure/comparison) against Azure AI Foundry deployments of OpenAI, Anthropic, Meta, and Mistral models. Configuration is entirely promptfooconfig.yaml-driven - add or remove models, change test questions, or adjust evaluation criteria by editing that file rather than the eval harness itself.

Teams evaluating which model to standardize on for a project, or engineers who need a repeatable way to re-benchmark model choices whenever Azure AI Foundry adds a new deployment or a provider ships a model update.