Integrate LLM Providers with LiteLLM

This example demonstrates how to use the LiteLLM provider with promptfoo to evaluate multiple models through a unified interface. LiteLLM provides a unified interface to 400+ LLMs, so instead of managing different APIs and authentication for each provider you use one interface to access models from OpenAI, Anthropic, Google, and many more. The example evaluates translation and creative writing tasks across three chat providers (GPT-4.1, Claude Sonnet 4.6, Gemini 2.5 Pro) plus an embedding model for similarity assertions.

Use this example when you want a single provider syntax to run evals across many different model providers without configuring each one's API separately, and when you want centralized cost tracking or load balancing across providers via the LiteLLM proxy. It is not a guide to LiteLLM's full feature set - it documents the promptfoo-specific proxy setup and provider syntax.

Set OPENAI_API_KEY and optionally ANTHROPIC_API_KEY/GOOGLE_AI_API_KEY, then start the LiteLLM proxy with the provided ./start-proxy.sh script or manually via pip install litellm[proxy] and litellm --model gpt-4.1 --model claude-sonnet-4-6 --model gemini-2.5-pro --model text-embedding-3-large. Run the eval with npx promptfoo@latest eval. The LiteLLM provider connects to the proxy server (default port 4000), which handles authentication/routing to each provider, standardizes request/response formats, handles errors and retries, and optionally provides caching and rate limiting. Three files make up the example: promptfooconfig.yaml (main eval config), litellm_config.yaml (proxy server config), and start-proxy.sh (helper script). Config uses providers like litellm:gpt-4.1, litellm:claude-sonnet-4-6, and litellm:gemini-2.5-pro, with defaultTest.options.provider.embedding set to litellm:embedding:text-embedding-3-large for similarity assertions.

Integrates LiteLLM as a unified proxy in front of OpenAI, Anthropic, Google, and other providers. A custom proxy host/port is set via apiBaseUrl in the provider config (for a LiteLLM proxy running on a different host or port than the local default), for example configuring the litellm:gpt-4.1 provider's config.apiBaseUrl to point at https://your-litellm-server.com; more complex setups run the proxy with litellm --config litellm_config.yaml rather than the inline --model flags. Documented troubleshooting: "Connection refused 0.0.0.0:4000" means the proxy isn't running (start it first); "API key not found" means the provider's env var wasn't set before starting the proxy; "Model not found" means the model wasn't included when starting the proxy. Setup can be verified two ways: confirming the proxy is up with curl http://localhost:4000/health, and confirming a key is actually set by echoing the relevant environment variable (e.g. echo $OPENAI_API_KEY).

Teams evaluating models across multiple providers (OpenAI, Anthropic, Google, or others in LiteLLM's 400+ catalog) who want one provider syntax, centralized cost tracking, and load balancing instead of configuring each provider's API separately in promptfoo.