What Azure MAI models does this example support?

This example focuses on MAI-Image-2.5 for image generation via the azure:image provider. It also includes configuration for reasoning/chat models (MAI-Thinking-1, MAI-DS-R1, MAI-Code-1-Flash) via azure:chat, though these are largely not yet deployable—MAI-DS-R1 is deprecated and the others are in private preview.

What credentials and setup are required to run this example?

You need AZURE_API_HOST (your Foundry resource endpoint) and AZURE_API_KEY (or Entra ID via az login). You must deploy MAI-Image-2.5 to your Azure resource using the provided az cognitiveservices deployment command, then set the environment variables and run promptfoo eval.

How are generated images handled and what do assertions need to accept?

Images are returned as base64 PNG. Promptfoo stores large base64 media as blob references (promptfoo://blob/...), so assertions should accept either the blob reference or inline data:image/... URLs.

How can I grade the quality of generated images?

Use the vision-judge config (promptfooconfig.vision-judge.yaml), which uses a vision-capable LLM judge with a custom rubricPrompt that passes the image as an image_url block. Run it with PROMPTFOO_INLINE_MEDIA=true to keep output as inline base64 so the hosted grader can read it.

What are the image dimension requirements and cost considerations?

Images require a minimum dimension of 768 pixels, with width times height capped at 1,048,576. Each image costs about $0.03, so caching is disabled by default for fresh runs.

Prompt Chain

Generate and Review Code with Azure Mai

Name: Azure Mai
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example evaluating Microsoft MAI image generation on Azure AI Foundry, graded by a vision LLM judge.

Copy chain

Promptfoo

Own this? Claim it

Maintainer of this project? Claim this page to edit the listing.

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

gpt 4o

Add to Favorites

Why it matters

Automate code generation and review processes using the Azure Mai prompt chain. This asset helps streamline development by producing code and ensuring its quality through review.

Outcomes

What it gets done

Generate code snippets based on prompts.

Debug generated code for errors.

Review code for quality and adherence to standards.

Integrate with development workflows for automated code tasks.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-azure-mai | bash

Steps

Steps in the chain

Deploy MAI image model to Foundry

Configure Azure API credentials

Run the evaluation

View generated images

Overview

Azure Mai

A promptfoo example evaluating Microsoft's MAI-Image-2.5 model on Azure AI Foundry, with per-image cost tracking and vision-LLM-judge grading of generated images. Use when evaluating Microsoft MAI image generation quality on Azure, including genuine vision-based grading rather than text-only judging.

What it does

This example evaluates Microsoft's first-party MAI models with promptfoo. It focuses on image generation with MAI-Image-2.5 (a Preview Foundry Model sold directly by Azure) via the dedicated azure:image provider, and shows how to wire reasoning/chat MAI models via azure:chat, though those currently have limited availability. The image is returned as a base64 PNG; promptfoo stores large base64 media as a blob reference (promptfoo://blob/...), so assertions should accept either the blob reference or an inline data:image/... URL.

When to use - and when NOT to

Use this example when you want to evaluate Microsoft MAI image generation quality on Azure, including vision-LLM-as-judge grading of the generated images. The reasoning/chat MAI models (MAI-Thinking-1, MAI-DS-R1, MAI-Code-1-Flash) are largely not deployable today - MAI-DS-R1 is deprecated and the others are private-preview and absent from the public CLI catalog - so the chat provider config ships commented out until a subscription can actually deploy one.

Inputs and outputs

Requires AZURE_API_HOST (the Foundry resource endpoint, e.g. your-resource.services.ai.azure.com) and AZURE_API_KEY (or Entra ID via az login). Deploy the model, then scaffold and run:

az cognitiveservices account deployment create \
  --name <RESOURCE> --resource-group <RG> \
  --deployment-name mai-image-2-5 \
  --model-name MAI-Image-2.5 --model-format Microsoft \
  --model-version 2026-06-02 --sku-name GlobalStandard --sku-capacity 1

Then set AZURE_API_HOST/AZURE_API_KEY and run promptfoo eval --no-cache (images cost about $0.03 each, so caching is disabled for fresh runs) followed by promptfoo view. The example ships promptfooconfig.yaml (generates images with MAI-Image-2.5, reporting per-image token usage and cost) and promptfooconfig.vision-judge.yaml (grades generated images with a vision-capable LLM judge). The image provider config sets model: MAI-Image-2.5, width/height (minimum 768, with width times height capped at 1,048,576).

Integrations

Integrates Azure AI Foundry's azure:image provider for MAI-Image-2.5 generation and azure:chat for MAI reasoning/chat models (auto-detected by name, sending max_completion_tokens and dropping temperature). The vision-judge config uses a custom rubricPrompt that passes the image to the grader as an image_url block so it evaluates the actual picture rather than a text description; running it requires PROMPTFOO_INLINE_MEDIA=true promptfoo eval -c promptfooconfig.vision-judge.yaml --no-cache so the output stays an inline base64 data URL the hosted grader can read, since the default blob-reference storage isn't fetchable by a hosted grader's API. Available deployable models for a given subscription can be checked with az cognitiveservices model list --location <region>.

Who it's for

Teams evaluating Microsoft's MAI image models on Azure who need both cost/token-usage tracking and genuine vision-based quality grading of generated images, and who want to be ready to test MAI's reasoning/chat models as they roll out of preview.

Source README

azure-mai (Microsoft MAI models on Azure AI Foundry)

You can run this example with:

npx promptfoo@latest init --example azure-mai
cd azure-mai

Evaluate Microsoft's first-party MAI models with promptfoo. This example focuses on image generation with MAI-Image-2.5 (a Preview Foundry Model sold by Azure) via the dedicated azure:image provider, and shows how to wire reasoning/chat MAI models via azure:chat - though those have limited availability today (see Notes).

Environment Variables

This example requires:

AZURE_API_HOST - your Foundry resource endpoint, e.g. your-resource.services.ai.azure.com
AZURE_API_KEY - a resource key (or authenticate with az login for Microsoft Entra ID)

Quick Start

### 1. Deploy an MAI image model to a Microsoft Foundry (AIServices) resource
az cognitiveservices account deployment create \
  --name <RESOURCE> --resource-group <RG> \
  --deployment-name mai-image-2-5 \
  --model-name MAI-Image-2.5 --model-format Microsoft \
  --model-version 2026-06-02 --sku-name GlobalStandard --sku-capacity 1

### 2. Point promptfoo at the resource
export AZURE_API_HOST=<RESOURCE>.services.ai.azure.com
export AZURE_API_KEY=<key>

### 3. Run the eval (images cost ~$0.03 each, so disable caching for fresh runs)
promptfoo eval --no-cache

### 4. View the generated images
promptfoo view

What's in this Example

promptfooconfig.yaml - generates images with MAI-Image-2.5 (Preview) through the Microsoft azure:image provider, reporting per-image token usage and cost from the API's token counts
promptfooconfig.vision-judge.yaml - uses a vision LLM as a judge on the generated images (see below)
Shows how to wire reasoning/chat MAI models via azure:chat (these are deprecated/private-preview today - see Notes)

Providers

Image generation (`azure:image`)

providers:
  - id: azure:image:mai-image-2-5
    config:
      model: MAI-Image-2.5 # cost-reporting id (deployment names can't contain dots)
      width: 1024 # min 768; width * height <= 1,048,576
      height: 1024

The image is returned as a base64 PNG. promptfoo stores large base64 media as a blob reference (promptfoo://blob/...), so assertions should accept either the blob ref or an inline data:image/... URL.

Reasoning chat (`azure:chat`)

MAI-Thinking-1 and MAI-DS-R1 are reasoning models (auto-detected by name - promptfoo sends max_completion_tokens and drops temperature). They aren't deployable on most subscriptions today (MAI-DS-R1 is deprecated; MAI-Thinking-1 / MAI-Code-1-Flash are private preview and not in the public CLI catalog), so the chat provider in promptfooconfig.yaml is commented out. Uncomment it once your subscription can deploy one:

providers:
  - id: azure:chat:mai-thinking-1
    config:
      max_completion_tokens: 2048
      # omitDefaults: true   # if the deployment rejects top_p / penalties

LLM-as-judge on images (vision grading)

promptfooconfig.vision-judge.yaml grades each generated image with a vision-capable LLM. It uses a custom rubricPrompt that passes the image to the grader as an image_url block, so the judge evaluates the actual picture rather than a text description.

Run it with inline media so {{output}} is a base64 data URL the grader can read:

PROMPTFOO_INLINE_MEDIA=true promptfoo eval -c promptfooconfig.vision-judge.yaml --no-cache

Why inline media? With promptfoo's default media handling, an image output is stored as a promptfoo://blob/... reference, which a hosted grader's API can't fetch. PROMPTFOO_INLINE_MEDIA=true keeps the output as an inline data URL the vision model can read directly.

Notes

The MAI image models are currently Preview. The MAI text models have limited availability: MAI-DS-R1 is deprecated in the Azure catalog, and MAI-Thinking-1 / MAI-Code-1-Flash are in private preview and aren't yet in the public CLI catalog. Run az cognitiveservices model list --location <region> to see what your subscription can actually deploy. See the Azure provider docs for details.

FAQ

Common questions

Discussion