Integrate MLflow AI Gateway with Promptfoo
A promptfoo example using MLflow AI Gateway as an LLM provider, including as its own grading model.
Why it matters
Leverage MLflow AI Gateway as a provider within promptfoo for enhanced LLM testing and evaluation. Streamline your MLOps pipeline by integrating powerful AI governance with your prompt development process.
Outcomes
What it gets done
Configure promptfoo to use MLflow AI Gateway.
Test and evaluate LLM responses through the MLflow provider.
Integrate AI governance into your prompt testing framework.
Automate LLM provider selection and management.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-provider-mlflow-gateway | bash Steps
Steps in the chain
Overview
Provider Mlflow Gateway
A promptfoo example using MLflow AI Gateway as both the model provider and the llm-rubric grading provider, requiring no separate API key. Use it when evaluating models served through MLflow AI Gateway - update the endpoint name in the config to match what you actually created before running.
What it does
This promptfoo example demonstrates using MLflow AI Gateway as an LLM provider in promptfoo evaluations. Setup installs and starts MLflow (pip install mlflow[genai], then mlflow server --host 127.0.0.1 --port 5000), creates a gateway endpoint through the MLflow UI's AI Gateway section, and points promptfoo at it via a MLFLOW_GATEWAY_URL environment variable. The example configuration also uses that same gateway endpoint as the llm-rubric grader, so the whole evaluation runs without needing a separate OpenAI API key.
When to use - and when NOT to
Use it when you want to evaluate prompts or models served through MLflow AI Gateway, or want a self-contained promptfoo setup that doesn't depend on an external grading API key. Update the my-chat-endpoint reference in promptfooconfig.yaml to match the actual endpoint name created in the MLflow UI before running - the config won't work against a differently-named endpoint.
Inputs and outputs
Input is a running MLflow server with a configured gateway endpoint and the MLFLOW_GATEWAY_URL environment variable. Output is a promptfoo evaluation run scored by the llm-rubric grader, which itself runs through the same MLflow Gateway endpoint.
Integrations
It integrates promptfoo with MLflow AI Gateway as both the model provider under test and the grading provider, requiring no separate LLM API key.
Who it's for
Teams already running MLflow AI Gateway who want to evaluate models served through it with promptfoo, without provisioning a separate grading API key.
Source README
provider-mlflow-gateway (MLflow AI Gateway)
This example demonstrates how to use MLflow AI Gateway as an LLM provider in promptfoo.
To get started:
npx promptfoo@latest init --example provider-mlflow-gateway
Setup
- Install and start MLflow:
pip install mlflow[genai]
mlflow server --host 127.0.0.1 --port 5000
Create a gateway endpoint in the MLflow UI at http://localhost:5000 (AI Gateway → Create Endpoint).
Set environment variables:
export MLFLOW_GATEWAY_URL=http://localhost:5000
- Run the evaluation:
promptfoo eval -c promptfooconfig.yaml
Configuration
Update my-chat-endpoint in promptfooconfig.yaml with the name of the gateway endpoint you created.
The example also uses that endpoint as the llm-rubric grader, so it runs without a separate OpenAI API key.
See the MLflow Gateway provider docs for all configuration options.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.