Integrate Truefoundry with Promptfoo for Code Testing

A Promptfoo example collection for using TrueFoundry's AI Gateway (LLM, MCP, and Agent Gateway) as a unified provider. It covers two configurations: a basic one comparing GPT-4 via OpenAI and Claude 3.5 Sonnet via Anthropic with custom metadata and logging, and an advanced one exercising TrueFoundry's MCP (Model Context Protocol) server integration - web search, code execution tools, multi-step tool-use reasoning, and configurable iteration limits. Every model is addressed through a provider-account/model-name identifier format, where the provider-account is the name of that provider's integration inside TrueFoundry, such as openai-main or anthropic-main. It documents the specific models available through each backing provider: OpenAI (gpt-5, gpt-4o, gpt-4o-mini, o1, o1-mini), Anthropic (claude-sonnet-4.5, claude-3-5-sonnet-20241022, claude-3-opus-20240229), Google Gemini via Vertex AI (gemini-2.5-pro, gemini-2.5-flash, gemini-1.5-pro), and others like Groq's llama-3.3-70b-versatile and Mistral's mistral-large-latest. TrueFoundry also proxies embedding models, including OpenAI's text-embedding-3-large and Cohere's embed-english-v3.0, the latter requiring an explicit input_type of search_query, search_document, classification, or clustering. MCP server configuration lets you name an integration_fqn such as common-tools, selectively enable specific tools like web_search and code_executor rather than all available tools, and cap the agent's iteration_limit. When loggingConfig.enabled is set, every request is logged to the TrueFoundry dashboard for monitoring request and response logs, tracking performance metrics, analyzing cost, debugging, and filtering by the custom metadata attached to each config.

Use when routing LLM, embedding, or MCP-agent calls through TrueFoundry's AI Gateway from Promptfoo - comparing models across providers behind one gateway, testing MCP tool-use workflows, or setting up centralized observability and cost tracking for LLM calls. Not the right example for calling a provider directly without TrueFoundry's gateway layer, since the model-ID format and config options here are specific to how TrueFoundry names and routes its provider integrations.

Input is a TRUEFOUNDRY_API_KEY environment variable plus a chosen config, either the basic multi-model or the MCP-server example. Output, after running the eval, is per-model comparison results viewable via the Promptfoo UI, plus, when logging is enabled, full request and response logs, performance metrics, and cost data in the TrueFoundry dashboard.

providers:
  - id: truefoundry:openai-main/gpt-5
    config:
      temperature: 0.7
      max_completion_tokens: 500

Built on TrueFoundry's AI Gateway via Promptfoo's TrueFoundry provider, fronting OpenAI, Anthropic, Google Vertex AI, Groq, Mistral, and Cohere models and embeddings, with MCP server tool integration and a TrueFoundry dashboard for observability.

Teams standardizing LLM access behind a gateway who want to evaluate multiple providers' models, MCP tool-use behavior, and embedding models through one consistent TrueFoundry configuration with built-in logging and cost tracking.