Process and Route AI Evaluation Results Programmatically

This example shows how to use extension hooks to process evaluation results programmatically. It enables you to send metrics to monitoring systems, trigger alerts based on success rates, export data to custom formats, and integrate with CI/CD pipelines through four lifecycle hooks (beforeAll, beforeEach, afterEach, afterAll).

Use this when you need to automatically route evaluation results to external systems - for example, posting metrics to monitoring systems after each test run, sending alerts when success rates drop below thresholds, or exporting results to custom formats for downstream analysis.

Do NOT use this if you only need to view results in the promptfoo UI or export to standard formats already supported natively. Avoid it if you're running one-off manual evaluations where manual review is sufficient.

You configure extensions via YAML or CLI flags, pointing to handler files. The afterAll hook receives a context object containing:

{
  evalId: string;              // Unique evaluation ID
  config: UnifiedConfig;       // Full evaluation configuration
  suite: TestSuite;            // Test suite that was evaluated
  results: EvaluateResult[];   // All individual test results
  prompts: CompletedPrompt[];  // Prompts with metrics
}

You receive programmatic control over evaluation data at specific lifecycle points, allowing you to process, transform, or transmit results to any external system.

Extensions can be configured via YAML or CLI flags:

# promptfooconfig.yaml
extensions:
  - file://result-handler.js:afterAll

Or via CLI:

# Run with a one-off extension
promptfoo eval -x file://result-handler.js:afterAll

# Combine with config file extensions
promptfoo eval -x file://alert-on-failure.js:afterAll

The example includes result-handler.js (JavaScript handler with monitoring examples), result-handler.py (Python handler with webhook integration), and promptfooconfig.yaml configuration.

This is for teams building evaluation workflows who need to integrate test results into monitoring systems, alerting infrastructure, or CI/CD pipelines.

Quick start:

npx promptfoo@latest init --example config-result-hooks
npx promptfoo@latest eval