What Mistral models does this example include?

The example includes Magistral Medium (native reasoning model), Mistral Medium, Mistral Large, and Mistral Small chat models, plus mistral-large-latest for grading and mistral-embed for semantic similarity checks.

How do I run this example?

Set your MISTRAL_API_KEY environment variable, then run `promptfoo eval` for the default config or `npx promptfoo@latest eval -c ` for a specific use case like math reasoning or model comparison.

What use cases does this example cover?

It covers mathematical reasoning (AIME2024 problems), multi-model comparison, cost optimization across pricing tiers, tool use, code generation, multimodal tasks, JSON mode, and self-evaluation using Mistral models for grading.

What are the pricing and context window differences between Mistral models?

Magistral Medium costs $2/$5 per 1M tokens with 128k context; Mistral Medium costs $1.50/$7.50 with 256k context; Mistral Large costs $0.50/$1.50 with 256k context; and Mistral Small costs $0.15/$0.60 with 256k context.

Prompt Chain

Evaluate and Generate Code with Mistral Models

Name: Mistral
Availability: OnlineOnly
Author: Promptfoo

A Promptfoo example demonstrating Mistral's reasoning, chat, and multimodal models, plus Mistral-powered grading and embeddings for evaluation.

Copy chain

Works with github

Promptfoo

Own this? Claim it

Maintainer of this project? Claim this page to edit the listing.

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

mistral large

Add to Favorites

Why it matters

Leverage Mistral AI's advanced chat and reasoning models to automate code evaluation, grading, and generation tasks. This asset integrates with Mistral's capabilities for enhanced code quality and development efficiency.

Outcomes

What it gets done

Utilize Mistral models for code generation.

Employ Mistral for evaluation grading of code.

Integrate Mistral's reasoning models into development pipelines.

Explore Mistral's multimodal capabilities for code-related tasks.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-mistral | bash

Steps

Steps in the chain

Initialize Mistral example project

Set Mistral API key

Run evaluation

View results in web UI

Overview

Mistral

A Promptfoo example demonstrating Mistral's reasoning, chat, and multimodal models across nine configuration files, with Mistral-based grading and embedding similarity checks. Use it to evaluate or compare Mistral models across reasoning, chat, coding, and tool-use tasks, or to build a Mistral-native evaluation pipeline.

What it does

Demonstrates Mistral AI's model lineup inside a Promptfoo evaluation suite - reasoning models, current multimodal chat models, Mistral-based evaluation grading, and Mistral embeddings for similarity checks. The reasoning showcase runs AIME2024 competition math problems against Magistral Medium, a native-reasoning model at $2/$5 per million tokens with 128k context, while Magistral Small has been folded into Mistral Small 4 - the magistral-small-latest alias now resolves to mistral-small-2603, a hybrid model reachable via mistral-small-latest, with reasoning mode enabled through reasoning_effort: high; the standalone magistral-small-2509 snapshot is deprecated and retires 2026-07-31. Chat capabilities cover three current multimodal models: Mistral Medium 3.5 (frontier agentic/coding, $1.50/$7.50 per million, 256k context), Mistral Large 3 (general-purpose, $0.50/$1.50 per million, 256k context), and Mistral Small 4 (hybrid instruct/reasoning/coding, $0.15/$0.60 per million, 256k context). Evaluation itself runs on Mistral: mistral-large-latest grades outputs as an LLM judge instead of OpenAI, and mistral-embed powers semantic-similarity checks. The example ships nine separate configuration files covering mathematical reasoning (AIME2024), step-by-step logical reasoning, cross-model comparison, multi-language code generation with Codestral, vision-plus-text multimodal tasks, function calling and tool routing (including tool-only, mixed content-plus-tool-calls, and file-based tool scenarios), structured JSON-mode output, and a main combined example. The tool-routing configuration specifically runs end-to-end QA across tool-only responses, mixed content-plus-tool-calls responses, file-based tools, and plain chat output, exercising the model's ability to pick the right response shape for each case.

When to use - and when NOT to

Use it to evaluate or compare Mistral models across reasoning, chat, coding, multimodal, and tool-use tasks, or as a template for building an eval that uses Mistral end-to-end for both generation and grading, rather than an evaluation that only tests a single Mistral model in isolation.

Inputs and outputs

Input is a MISTRAL_API_KEY plus a chosen configuration file - math reasoning, model comparison, code generation, multimodal, tool use, tool routing, or JSON mode. Output is a Promptfoo evaluation run with per-model results, cost tracking across tiers, and grading via llm-rubric and similarity assertions, viewable through promptfoo view.

Integrations

Built on Promptfoo (promptfoo eval/promptfoo view) against the Mistral API, using mistral-large-latest for LLM-as-judge grading and mistral-embed for embedding-based similarity assertions, with provider overrides so grading and embeddings can be swapped independently of the models under test.

Who it's for

For developers evaluating or comparing Mistral's model lineup across reasoning, chat, multimodal, and coding tiers, or building a Mistral-native evaluation pipeline that doesn't depend on OpenAI for grading or embeddings.

### Set your API key
export MISTRAL_API_KEY=your_api_key_here

### Run the evaluation
promptfoo eval

### View results in the web UI
promptfoo view

Source README

mistral (Mistral AI Chat Models)

This example demonstrates Mistral AI's chat models, including Magistral reasoning models, current multimodal models, and shows how to use Mistral models for evaluation grading and embeddings.

You can run this example with:

npx promptfoo@latest init --example mistral
cd mistral

Environment Variables

This example requires:

MISTRAL_API_KEY - Your Mistral API key (get it from console.mistral.ai)

What This Example Shows

Mathematical Reasoning: AIME2024 competition problems with Magistral Medium
Model Comparison: Compare Mistral's different model capabilities
Reasoning Models: Showcase Magistral Medium (native reasoning) vs. Mistral Small 4
Chat Capabilities: General conversation and task completion
Mistral-powered Evaluation: Use Mistral models for grading instead of OpenAI
Mistral Embeddings: Use Mistral's embedding model for similarity checks

Models Demonstrated

Reasoning Models

Magistral Medium (magistral-medium-latest → magistral-medium-2509): Native reasoning model ($2/$5 per 1M tokens, 128k context) - the reasoning showcase in these examples.

Mistral folded Magistral Small into Mistral Small 4: the magistral-small-latest alias now resolves to mistral-small-2603 (a hybrid model, $0.15/$0.60 per 1M), so these examples use the canonical mistral-small-latest id. Enable Small 4's reasoning mode with reasoning_effort: high. The standalone magistral-small-2509 snapshot is deprecated (retires 2026-07-31).

Chat Models

Mistral Medium 3.5 (mistral-medium-latest → mistral-medium-2604): Frontier agentic/coding multimodal model ($1.50/$7.50 per 1M, 256k context)
Mistral Large 3 (mistral-large-latest → mistral-large-2512): General-purpose multimodal model ($0.50/$1.50 per 1M, 256k context)
Mistral Small 4 (mistral-small-latest → mistral-small-2603): Hybrid instruct/reasoning/coding model ($0.15/$0.60 per 1M, 256k context)

Evaluation Models

Grading: Uses mistral-large-latest for LLM-as-a-judge evaluation
Embeddings: Uses mistral-embed for semantic similarity checks

Key Features Demonstrated

Multi-model comparison: Compare performance across different Mistral models
Reasoning capabilities: Step-by-step problem solving with Magistral models
Cost optimization: Balance performance vs. cost across model tiers
Self-evaluation: Use Mistral models to grade their own outputs
Semantic similarity: Mistral embeddings for content comparison

Running the Example

### Set your API key
export MISTRAL_API_KEY=your_api_key_here

### Run the evaluation
promptfoo eval

### View results in the web UI
promptfoo view

Configuration Highlights

This example showcases several advanced promptfoo features:

Provider overrides for grading and embeddings
Multiple assertion types including llm-rubric and similarity
Cost tracking across different model tiers
Mixed scenarios from simple chat to complex reasoning

The evaluation uses Mistral models end-to-end, providing a comprehensive view of their ecosystem capabilities.

Available Configurations

This example includes multiple configuration files for different use cases:

Mathematical Reasoning

promptfooconfig.aime2024.yaml - Advanced mathematical competition problems (AIME2024 dataset)
promptfooconfig.reasoning.yaml - Step-by-step logical problem solving

Model Capabilities

promptfooconfig.comparison.yaml - Compare reasoning across all Mistral models
promptfooconfig.code-generation.yaml - Multi-language programming with Codestral
promptfooconfig.multimodal.yaml - Vision and text processing with current Mistral multimodal models

Advanced Features

promptfooconfig.tool-use.yaml - Function calling and tool integration
promptfooconfig.tool-routing.yaml - End-to-end QA for tool-only, mixed content+tool_calls, file-based tools, and plain chat output
promptfooconfig.json-mode.yaml - Structured JSON output generation
promptfooconfig.yaml - Main example with evaluation using Mistral models

Run any specific configuration:

npx promptfoo@latest eval -c promptfooconfig.aime2024.yaml  # Mathematical reasoning
npx promptfoo@latest eval -c promptfooconfig.comparison.yaml  # Model comparison

Additional Resources

Mistral Provider Documentation - Complete API reference and configuration options
Mistral Magistral Announcement - Official announcement and technical details

FAQ

Common questions

Discussion