Integrate AI Observability with OpenTelemetry

Built In provides automatic OpenTelemetry instrumentation for promptfoo LLM provider calls. It captures detailed span data following GenAI semantic conventions, including token usage, model parameters, and provider-specific attributes across every evaluation run. Tracing is enabled by default and exports to promptfoo's UI or external OTLP backends like Jaeger and Honeycomb.

Use Built In when you need to debug LLM provider behavior, analyze token consumption patterns across test cases, or integrate promptfoo evaluations into existing observability stacks. It is useful for evaluation pipelines where you need visibility into model calls, or when comparing behavior across multiple providers. Do NOT use this if you need to disable tracing for specific requirements, which can be done by setting PROMPTFOO_DISABLE_TRACING to true.

You provide API keys for the LLM providers you want to test and optionally configure environment variables to control tracing behavior. The system automatically instruments every LLM call during evaluation. You receive detailed trace spans in the promptfoo UI Traces tab or your configured OTLP backend, containing GenAI semantic convention attributes including gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.request.max_tokens, gen_ai.request.temperature, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.total_tokens, gen_ai.usage.cached_tokens, gen_ai.usage.reasoning_tokens, gen_ai.response.model, gen_ai.response.id, and gen_ai.response.finish_reasons, plus promptfoo-specific attributes including promptfoo.provider.id, promptfoo.eval.id, promptfoo.test.index, and promptfoo.prompt.label.

Built In supports LLM providers with full tracing instrumentation: OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Vertex AI, Ollama, Mistral, Cohere, Huggingface, IBM Watsonx, HTTP, OpenRouter, Replicate, OpenAI-compatible providers (inherited), and Cloudflare AI (inherited). For observability backends, it exports to any OTLP-compatible system including Jaeger (open-source distributed tracing) and Honeycomb (observability platform). Configuration uses standard OpenTelemetry environment variables (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME) and promptfoo-specific controls (PROMPTFOO_DISABLE_TRACING).

Initialize and run the example:

npx promptfoo@latest init --example integration-opentelemetry/built-in
cd integration-opentelemetry/built-in

Set provider credentials:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

Run evaluation and view traces:

npx promptfoo eval -c promptfooconfig.yaml
npx promptfoo view

Export to Jaeger:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 16686:16686 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npx promptfoo eval

Built In is for ML engineers and DevOps teams running systematic LLM evaluations who need observability into provider behavior, token economics, and model performance. It serves platform teams integrating promptfoo into existing OpenTelemetry-instrumented infrastructure, and data scientists debugging prompt performance across multiple providers. Unlike manual logging approaches, Built In provides standardized GenAI semantic conventions and zero-configuration instrumentation for supported providers.