Trace LLM Provider Operations with OpenTelemetry
A promptfoo example asserting on OpenTelemetry trace spans - counts, durations, and error rates for a RAG agent.
Why it matters
Enhance your LLM development process by integrating OpenTelemetry tracing into your Promptfoo evaluations. Gain deep visibility into the internal workings of your LLM providers to identify bottlenecks and improve performance.
Outcomes
What it gets done
Instrument Promptfoo evaluations for detailed tracing.
Monitor and analyze LLM provider performance during testing.
Debug and optimize LLM interactions using trace data.
Integrate tracing into your CI/CD pipeline for automated quality checks.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-opentelemetry-tracing | bash Overview
Opentelemetry Tracing
This promptfoo example asserts directly on OpenTelemetry trace spans emitted by a RAG agent provider, checking span counts, durations, and error rates via promptfoo's trace-based assertions. Use it when the provider under test emits OTLP traces and you need to verify internal execution behavior, not just the final output.
What it does
This promptfoo example evaluates a custom provider (file://provider-simple-traced.js) that emits OpenTelemetry traces for a simulated RAG agent workflow, then asserts directly on those traces rather than just the text output. It checks that expected spans exist (rag_agent_workflow, retrieve_document_*, reasoning_*), that spans complete within duration limits, and that no error spans occur.
When to use - and when NOT to
Use this example when your LLM application is instrumented with OpenTelemetry and you want to verify its internal execution behavior - not just the final answer - such as confirming a RAG pipeline retrieves exactly 3 documents, completes reasoning steps, finishes within a time budget, and does not error. It requires the provider under test to actually emit OTLP traces that promptfoo can receive on its configured OTLP HTTP endpoint; it is not useful for providers that don't produce trace spans.
Inputs and outputs
Each test enables tracing via metadata: { tracingEnabled: true } and asserts on span counts and durations with pattern matching, for example:
assert:
# Ensure all expected spans are present
- type: trace-span-count
value:
pattern: 'rag_agent_workflow'
min: 1
max: 1
# Ensure we retrieve exactly 3 documents
- type: trace-span-count
value:
pattern: 'retrieve_document_*'
min: 3
max: 3
# Ensure the overall workflow completes quickly
- type: trace-span-duration
value:
pattern: 'rag_agent_workflow'
max: 5000 # 5 seconds max
defaultTest adds metric-only assertions (weight: 0) for p95 span duration and an allowed error rate of up to 5% (trace-error-spans with max_percentage: 5), which track metrics without failing the test outright. Tracing itself is configured via:
tracing:
enabled: true
otlp:
http:
enabled: true
port: 4318
acceptFormats: ['json']
Integrations
Runs against a custom traced JavaScript provider and promptfoo's built-in OTLP HTTP receiver (port 4318, JSON format), using the trace-span-count, trace-span-duration, and trace-error-spans assertion types alongside a standard javascript assertion loaded from trace-assertions.js.
Who it's for
Teams building instrumented, multi-step LLM agents (RAG pipelines, tool-calling workflows) who want to assert on internal execution behavior - span counts, latencies, and error rates - not just the final generated text.
Source README
yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: OpenTelemetry tracing with trace-based assertions
providers:
- file://provider-simple-traced.js
prompts:
- 'Explain how {{topic}} works in simple terms'
tests:
vars:
topic: 'quantum computing'
metadata:
tracingEnabled: true
testCaseId: 'test-case-1'
assert:Original javascript assertion
- type: javascript
value: file://trace-assertions.js
Ensure all expected spans are present
- type: trace-span-count
value:
pattern: 'rag_agent_workflow'
min: 1
max: 1
Ensure we retrieve exactly 3 documents
- type: trace-span-count
value:
pattern: 'retrieve_document_*'
min: 3
max: 3
Ensure all reasoning steps occur
- type: trace-span-count
value:
pattern: 'reasoning_*'
min: 3
Ensure the overall workflow completes quickly
- type: trace-span-duration
value:
pattern: 'rag_agent_workflow'
max: 5000 # 5 seconds max
Ensure individual operations don't take too long
- type: trace-span-duration
value:
pattern: '*'
max: 1000 # No single span should exceed 1 second
Ensure no errors occur
- type: trace-error-spans
value:
max_count: 0
- type: javascript
vars:
topic: 'machine learning'
metadata:
tracingEnabled: true
testCaseId: 'test-case-2'
assert:Original javascript assertion
- type: javascript
value: file://trace-assertions.js
Same trace assertions as above
type: trace-span-count
value:
pattern: 'rag_agent_workflow'
min: 1
max: 1type: trace-span-count
value:
pattern: 'retrieve_document_*'
min: 3
max: 3type: trace-span-count
value:
pattern: 'reasoning_*'
min: 3type: trace-span-duration
value:
pattern: 'rag_agent_workflow'
max: 5000type: trace-span-duration
value:
pattern: '*'
max: 1000type: trace-error-spans
value:
max_count: 0
- type: javascript
Default assertions that apply to all test cases
defaultTest:
assert:
# Monitor 95th percentile latency (metric only, won't fail tests)
- type: trace-span-duration
value:
pattern: '*'
max: 2000
percentile: 95
weight: 0 # This makes it a metric-only assertion
metric: p95_latency
# Ensure retrieval operations are fast
- type: trace-span-duration
value:
pattern: 'retrieve_document_*'
max: 300 # Each document retrieval should be under 300ms
# Allow up to 5% error rate (more forgiving for production)
- type: trace-error-spans
value:
max_percentage: 5
pattern: '*'
metric: error_rate
Tracing configuration
tracing:
enabled: true
otlp:
http:
enabled: true
port: 4318
acceptFormats: ['json']
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.