Benchmark OpenAI vs Azure OpenAI Performance and Cost
Compare OpenAI and Azure OpenAI side-by-side to benchmark speed, cost, and output quality differences using identical models.
Why it matters
Compare OpenAI and Azure OpenAI services side-by-side using identical models to evaluate differences in response speed, API costs, and output quality, helping teams make informed decisions about which cloud provider to use for their AI applications.
Outcomes
What it gets done
Run parallel evaluations against OpenAI and Azure OpenAI endpoints with the same prompts
Measure and compare response latency between the two services
Calculate and compare API costs for identical workloads
Analyze output quality differences between OpenAI and Azure implementations
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-openai-azure-comparison | bash Steps
Steps in the chain
Overview
Openai Azure Comparison
This prompt chain benchmarks OpenAI against Azure OpenAI using the same model (GPT-5-mini) to compare differences in speed, cost, and output between the two services. Use this workflow when you need to compare OpenAI and Azure OpenAI service performance using the same model. Do NOT use this if you are comparing different model families or architectures - this workflow is designed specifically for same-model cross-platform comparison only.
What it does
This prompt chain benchmarks OpenAI against Azure OpenAI using the same model (GPT-5-mini) to compare differences in speed, cost, and output between the two services.
When to use - and when NOT to
Use this workflow when you need to compare OpenAI and Azure OpenAI service performance using the same model. Do NOT use this if you are comparing different model families or architectures - this workflow is designed specifically for same-model cross-platform comparison only.
Inputs and outputs
You provide API keys for both OpenAI and Azure OpenAI, your Azure deployment name, and your Azure host endpoint in the configuration file. The workflow executes evaluations against both services and displays the results through a view command.
How to use
Initialize the example and navigate to the project directory:
npx promptfoo@latest init --example openai-azure-comparison
cd openai-azure-comparison
Configure your API credentials:
export OPENAI_API_KEY=your-openai-key
export AZURE_API_KEY=your-azure-key
Update promptfooconfig.yaml with your Azure deployment name and host. Run the evaluation:
npx promptfoo@latest eval
View the comparison results:
npx promptfoo@latest view
Integrations
OpenAI API - Integration with OpenAI's GPT models.
Azure OpenAI Service - Connects to Microsoft Azure's hosted OpenAI endpoints.
promptfoo - Built on the promptfoo evaluation framework for LLM testing.
Who it's for
ML engineers and platform architects evaluating cloud provider options for LLM deployments can use this to compare OpenAI and Azure OpenAI services. DevOps teams managing multi-cloud AI infrastructure can use this for service comparison. Technical teams considering deployment options can use this workflow to compare the two platforms.
Source README
openai-azure-comparison (OpenAI vs Azure OpenAI Comparison)
You can run this example with:
npx promptfoo@latest init --example openai-azure-comparison
cd openai-azure-comparison
This example compares OpenAI and Azure OpenAI using the same model (GPT-5-mini) to benchmark differences in speed, cost, and output between the two services.
See the guide for more details.
Setup
Set the required environment variables:
export OPENAI_API_KEY=your-openai-key
export AZURE_API_KEY=your-azure-key
Update promptfooconfig.yaml with your Azure deployment name and host.
Run
npx promptfoo@latest eval
View
npx promptfoo@latest view
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.