Extend Test Lifecycle with Custom Setup and Teardown Hooks
Custom setup/teardown hooks for promptfoo tests via the extensions API - beforeAll, afterAll, beforeEach, afterEach.
Why it matters
Configure and execute custom pre- and post-test actions in your AI evaluation pipeline using Python or JavaScript extensions that run at specific points in the test lifecycle.
Outcomes
What it gets done
Run beforeAll and afterAll hooks to set up and tear down entire test suites
Execute beforeEach and afterEach hooks for individual test preparation and cleanup
Manage environment resources like servers, databases, and credentials programmatically
Integrate external systems and services into your testing workflow
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-config-extension-api | bash Steps
Steps in the chain
Overview
Config Extension Api
Demonstrates promptfoo's extensions API for custom test setup/teardown via beforeAll, afterAll, beforeEach, and afterEach hooks defined in Python or JavaScript. Use when tests need custom environment setup/teardown, external-system integration, or logging around promptfoo evaluations.
What it does
This promptfoo example demonstrates the extensions API for implementing custom setup and teardown hooks around individual tests, defined in either Python or JavaScript. Four hook types trigger at specific points in the evaluation lifecycle: beforeAll (once before the entire test suite), afterAll (once after the suite completes), beforeEach (before each individual test), and afterEach (after each individual test). Each hook receives a hookName parameter and a context object; beforeAll and beforeEach must return the context object, while afterAll and afterEach return nothing. Extensions are declared in promptfooconfig.yaml with the function name appended after the file path via a colon:
extensions:
- file://path/to/your/extension.py:extension_hook
- file://path/to/your/extension.js:extensionHook
When to use - and when NOT to
Use this example when custom pre- and post-test actions are needed: dynamically preparing or cleaning up test environments, integrating with external systems during testing, or implementing custom logging or reporting - functioning like the before/after hooks familiar from unit testing frameworks.
Inputs and outputs
Run the example via npx promptfoo@latest init --example config-extension-api, then LOG_LEVEL=debug promptfoo eval followed by promptfoo view. Six practical use cases are named: environment setup/teardown (dynamically creating and removing files or resources per test), AI agent testing (e.g. deploying Kubernetes manifests before evaluation), model loading (loading ML models into memory before tests and unloading afterward), local server management (starting and stopping local LLM servers like llama.cpp or ollama), connection management (opening and closing connection tunnels or database connections), and credential handling (securely loading and managing credentials or auth tokens for the test environment).
Integrations
Built on promptfoo's extension-hooks configuration mechanism, referencing Python or JavaScript hook functions by file path plus function name, separated by a colon. Extensions behave like the before/after hooks familiar from popular unit testing frameworks, letting a test suite set up and tear down its environment with precision. Points to the official Extension Hooks documentation for comprehensive implementation details.
Who it's for
Engineers writing promptfoo evaluations that need environment setup/teardown, external-system integration, or custom logging around tests - deploying test infrastructure, loading models, or managing credentials - rather than manually scripting setup and cleanup outside the eval run.
Source README
config-extension-api (Custom Extensions Example for promptfoo)
You can run this example with:
npx promptfoo@latest init --example config-extension-api
cd config-extension-api
This example demonstrates how to leverage promptfoo's powerful extensions API to implement custom setup and teardown hooks for individual tests. These extensions can be defined in either Python or JavaScript, providing flexibility for your preferred programming environment.
Extension Hooks Overview
promptfoo supports four types of extension hooks, each triggered at a specific point in the evaluation lifecycle:
beforeAll: Executed once before the entire test suite beginsafterAll: Executed once after the entire test suite completesbeforeEach: Executed before each individual testafterEach: Executed after each individual test
Each hook receives a hookName parameter and a context object containing relevant data for that specific hook type.
The beforeAll and beforeEach hooks should return the context object, while the afterAll and afterEach hooks should not return anything.
For comprehensive information on implementing and using these hooks, refer to the Extension Hooks section in the official documentation.
Configuring Extensions
Specify your extensions in the promptfooconfig.yaml file:
extensions:
- file://path/to/your/extension.py:extension_hook
- file://path/to/your/extension.js:extensionHook
Note: You must include the function name after the file path, separated by a colon (:).
Why Use Extensions?
Extensions in promptfoo empower you to:
- Enhance test functionality with custom pre- and post-test actions
- Dynamically prepare and clean up test environments
- Seamlessly integrate with external systems or services for comprehensive testing
- Implement custom logging or reporting mechanisms
These extensions function similarly to "before" and "after" hooks in popular unit testing frameworks, allowing you to set up and tear down test environments with precision and control.
Practical Use Cases for Extensions
- Environment Setup/Teardown: Dynamically create and remove files or resources for each test.
- AI Agent Testing: Configure complex scenarios, such as deploying Kubernetes manifests before evaluation.
- Model Loading: Efficiently load ML models into memory before tests and unload them afterward.
- Local Server Management: Programmatically start and stop local language model servers (e.g., llama.cpp, ollama) for testing.
- Connection Management: Automate the opening and closing of connection tunnels or database connections.
- Credential Handling: Securely load and manage credentials or authentication tokens for testing environments.
Running the Example
To execute this example, use the following command in your terminal:
LOG_LEVEL=debug promptfoo eval
Then
promptfoo view
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.