Generate Dynamic Prompts Using Executable Scripts
Demonstrates using executable scripts or binaries as dynamic prompt generators in promptfoo.
Why it matters
Enable developers to dynamically generate AI prompts using any executable script or binary (shell, Ruby, Python, etc.) that can process test variables and context to produce customized prompt text for evaluation and testing workflows.
Outcomes
What it gets done
Execute shell scripts, Ruby scripts, or binaries to generate prompts from external data sources
Pass JSON context with test variables, provider info, and configuration to executable generators
Capture generated prompt output from stdout and handle errors from stderr
Configure executable prompts with custom parameters like timeout, temperature, and style settings
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-config-executable-prompts | bash Steps
Steps in the chain
Overview
Config Executable Prompts
Demonstrates using executable scripts or binaries (shell, Ruby, etc.) as dynamic, context-aware prompt generators in promptfoo via the exec: mechanism. Use when generating prompts from external data sources or tooling outside JavaScript or Python.
What it does
This promptfoo example demonstrates using executable scripts and binaries - shell scripts, Python scripts, or any other binary - as dynamic prompt generators. Run it with:
npx promptfoo@latest init --example config-executable-prompts
cd config-executable-prompts
promptfoo eval
Your executable receives a JSON context as its first command-line argument containing vars (test variables from configuration), provider (info about the current provider), and config (any additional configuration passed to the prompt); it outputs the generated prompt to stdout, writes errors to stderr, and a non-zero exit code fails prompt generation. Three usage methods are shown: the exec: prefix (exec:./my-script.sh), direct file reference auto-detected by a .sh extension (Python .py files are instead processed as Python prompt templates by default unless given the exec: prefix), and a configured form with a label, raw: exec: reference, and a config block (temperature, style, timeout - default 60 seconds).
When to use - and when NOT to
Use this example when generating prompts based on external data sources, creating context-aware prompts dynamically, integrating with existing tools and scripts, or generating prompts in a language other than JavaScript or Python.
Inputs and outputs
Requires bash/zsh or PowerShell, jq for the shell example, and Ruby 3.x for the Ruby example, with scripts made executable via chmod +x. Given the JSON context on stdin/argv, the script's stdout becomes the prompt sent to the provider.
Integrations
Built on promptfoo's exec: prompt mechanism, demonstrated with a shell script (prompt-generator.sh) and a Ruby script (template-prompt.rb).
Who it's for
Engineers who need prompts generated by tooling outside JavaScript or Python - shell scripts, Ruby, compiled binaries - or who want to integrate existing data-fetching or templating tools directly into promptfoo's prompt generation step.
Source README
config-executable-prompts (Executable Prompts)
This example demonstrates how to use executable scripts and binaries as prompt generators in promptfoo.
To initialize and run it:
npx promptfoo@latest init --example config-executable-prompts
cd config-executable-prompts
promptfoo eval
Prerequisites
- bash/zsh (macOS/Linux) or PowerShell (Windows)
- jq (for the shell example): https://stedolan.github.io/jq/
- Ruby 3.x (for the Ruby example)
- Make scripts executable:
chmod +x prompt-generator.sh template-prompt.rb
Overview
Promptfoo supports using any executable (shell scripts, Python scripts, binaries, etc.) to dynamically generate prompts. This is useful when you need to:
- Generate prompts based on external data sources
- Create context-aware prompts dynamically
- Integrate with existing tools and scripts
- Generate prompts in languages other than JavaScript or Python
How It Works
Script Input: Your executable receives a JSON context as its first command-line argument containing:
vars: Test variables from your configurationprovider: Information about the current providerconfig: Any additional configuration passed to the prompt
Script Output: Your script should output the generated prompt to stdout
Error Handling:
- Errors should be written to stderr
- Non-zero exit codes will cause the prompt generation to fail
Usage Methods
Method 1: Using exec: prefix
prompts:
- exec:./my-script.sh
- exec:/usr/bin/custom-prompt-generator
Method 2: Direct file reference (auto-detected)
prompts:
- ./my-script.sh # Detected by .sh extension
# Note: Python files (.py) are processed as Python prompt templates by default.
# To run a Python script as an executable prompt, use the exec: prefix:
# - exec:./generator.py
Method 3: With configuration
prompts:
- label: 'Custom Prompt'
raw: exec:./generator.sh
config:
temperature: 0.7
style: technical
timeout: 30000 # 30 second timeout (default: 60s)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.