Run LLM Evaluations and Experiments

Patronus AI MCP Server exposes the Patronus SDK through a standardized MCP interface, letting an AI assistant run evaluations, batch scoring, and experiments against LLM system outputs - checking for hallucinations, conciseness, and other custom-defined criteria.

Use it when you want to evaluate LLM outputs systematically - running a single evaluation with a configurable evaluator, batch-evaluating multiple outputs with several evaluators at once, running experiments against a dataset, or defining custom evaluator functions with the @evaluator decorator. It's useful for LLM QA workflows like checking whether a model's output hallucinates, or scoring response brevity against custom criteria. Do not use it without a PATRONUS_API_KEY - it's required to authenticate with the Patronus service and can be supplied via command line or environment variable.

• initialize: initialize Patronus with an API key and project settings.
• evaluate: run a single evaluation with a configurable evaluator.
• batch_evaluate: run batch evaluations with multiple evaluators.
• run_experiment: run experiments against datasets.
• list_evaluator_info: get a full overview of all available evaluators and their associated criteria.
• create_criteria: create new evaluator criteria in the Patronus API.
• custom_evaluate: evaluate task outputs using a custom evaluator function decorated with @evaluator.

git clone https://github.com/yourusername/patronus-mcp-server.git
cd patronus-mcp-server
python -m venv .venv
source .venv/bin/activate
uv pip install -e .
uv pip install -e ".[dev]"

Run with an API key via command line:

python src/patronus_mcp/server.py --api-key your_api_key_here

Or via environment variable:

export PATRONUS_API_KEY=your_api_key_here
python src/patronus_mcp/server.py

LLM developers and evaluation/QA teams who want to run systematic evaluations, batch scoring, and dataset-driven experiments against model outputs - including custom evaluator logic - through a standardized MCP interface.