Integrate LLM Gateway with Promptfoo
TrueFoundry LLM Gateway example configuration for testing OpenAI GPT-5 and Anthropic Claude Sonnet 4.5 models with logging, metadata, and assertion-based
Why it matters
Connect your LLM Gateway to promptfoo for robust testing and evaluation of your AI applications. Streamline your development process by integrating powerful LLM capabilities with a leading prompt testing framework.
Outcomes
What it gets done
Configure promptfoo to interact with Truefoundry's LLM Gateway.
Test and evaluate LLM prompts and responses using promptfoo.
Integrate LLM Gateway into your existing CI/CD pipelines.
Automate prompt testing for improved AI application quality.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-truefoundry | bash Capabilities
What this chain does
Configure promptfoo to interact with Truefoundry's LLM Gateway.
Test and evaluate LLM prompts and responses using promptfoo.
Integrate LLM Gateway into your existing CI/CD pipelines.
Automate prompt testing for improved AI application quality.
Overview
Truefoundry
What it does
This prompt workflow demonstrates how to configure and test multiple large language models through the TrueFoundry LLM Gateway. It sets up a helpful assistant prompt that answers questions concisely, routes requests through TrueFoundry to both GPT-5 and Claude Sonnet 4.5, and validates responses using assertion-based tests including basic string matching and LLM-rubric evaluation.
How it connects
Use this configuration when you need to evaluate and compare multiple LLM providers through TrueFoundry's gateway with consistent logging and metadata tracking across development environments. It's ideal when you want to test basic knowledge retrieval, technical explanations, and creative writing tasks with automated validation. Do not use this if you need production-grade configurations with custom rate limiting or advanced routing logic, as this is explicitly labeled as an example configuration for demonstration purposes. Avoid this setup if you require models beyond GPT-5 and Claude Sonnet
Source README
yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: TrueFoundry LLM Gateway Example
| prompts: |
|---|
You are a helpful assistant. Answer the following question concisely:
{{question}}
providers:
Basic GPT-5 configuration through TrueFoundry
- id: truefoundry:openai-main/gpt-5
label: 'GPT-5 via TrueFoundry'
config:
temperature: 0.7
max_completion_tokens: 500
metadata:
user_id: 'example-user'
environment: 'development'
loggingConfig:
enabled: true
Claude Sonnet 4.5 for comparison
- id: truefoundry:anthropic-main/claude-sonnet-4.5
label: 'Claude Sonnet 4.5 via TrueFoundry'
config:
temperature: 0.7
max_tokens: 500
metadata:
user_id: 'example-user'
environment: 'development'
loggingConfig:
enabled: true
tests:
description: 'Basic knowledge test'
vars:
question: 'What is the capital of France?'
assert:- type: contains
value: 'Paris'
- type: contains
description: 'Technical explanation'
vars:
question: 'Explain machine learning in simple terms'
assert:- type: llm-rubric
value: 'Provides a clear, simple explanation of machine learning'
- type: llm-rubric
description: 'Creative writing'
vars:
question: 'Write a haiku about coding'
assert:- type: llm-rubric
value: 'Responds with a proper haiku (5-7-5 syllable structure)'
- type: llm-rubric
defaultTest:
options:
provider:
config:
temperature: 0.7
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.