Evaluate prompts across Atlas Cloud model families
Configures Atlas Cloud's OpenAI-compatible LLM API in promptfoo for multi-model-family evaluation.
Why it matters
Enable developers to systematically test and compare prompt performance across multiple LLM families hosted on Atlas Cloud using a unified OpenAI-compatible API, ensuring consistent quality before production deployment.
Outcomes
What it gets done
Configure multiple Atlas Cloud-hosted chat models for side-by-side evaluation
Run deterministic and rubric-based assertions against prompt outputs
Compare generation parameters across DeepSeek, Qwen, and other model families
Set up custom gateway endpoints with flexible API key management
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-provider-atlascloud | bash Steps
Steps in the chain
Overview
Provider Atlascloud
Configures Atlas Cloud's OpenAI-compatible LLM API as a promptfoo provider for evaluating multiple model families, with an optional custom gateway/proxy setup. Use when evaluating multiple model families hosted behind a single Atlas Cloud account.
What it does
This promptfoo example configures Atlas Cloud, which exposes an OpenAI-compatible LLM API, making it a good fit for prompt and model evaluation across multiple model families behind a single provider account. Models are referenced by provider ID:
providers:
- id: atlascloud:deepseek-ai/DeepSeek-V3-0324
- id: atlascloud:qwen/qwen3-32b
A custom gateway variant lets a request route through a proxy by setting apiBaseUrl and a custom apiKeyEnvar instead of the default Atlas Cloud endpoint and API key variable.
When to use - and when NOT to
Use this example when evaluating multiple model families (e.g. DeepSeek, Qwen) hosted behind a single Atlas Cloud account, or when Atlas Cloud requests need to route through a custom gateway/proxy.
Inputs and outputs
Requires an Atlas Cloud API key set via ATLASCLOUD_API_KEY. Run via npx promptfoo@latest init --example provider-atlascloud then npx promptfoo eval -c promptfooconfig.yaml. The included config demonstrates multiple Atlas Cloud-hosted chat models, standard OpenAI-style generation parameters, and a mix of deterministic and rubric-based assertions.
Integrations
Built on Atlas Cloud's OpenAI-compatible API, with a custom-gateway option for routing through a proxy via apiBaseUrl and apiKeyEnvar. Points to the promptfoo Atlas Cloud provider docs and Atlas Cloud's own documentation for further configuration detail.
Who it's for
Engineers evaluating multiple LLM model families through a single Atlas Cloud account who want standard OpenAI-compatible configuration and, optionally, routing through a custom gateway or proxy.
Source README
provider-atlascloud (Atlas Cloud Example)
This directory contains an example configuration for using Atlas Cloud with promptfoo.
Atlas Cloud exposes an OpenAI-compatible LLM API, so it is a good fit for prompt and model evaluation workflows across multiple model families behind a single provider account.
Prerequisites
Create an Atlas Cloud API key from the Atlas Cloud docs.
Set the environment variable:
export ATLASCLOUD_API_KEY=your_api_key_here
Quick Start
npx promptfoo@latest init --example provider-atlascloud
cd provider-atlascloud
npx promptfoo eval -c promptfooconfig.yaml
Example Config
The included promptfooconfig.yaml demonstrates:
- Multiple Atlas Cloud-hosted chat models
- Standard OpenAI-style generation parameters
- A mix of deterministic and rubric-based assertions
Provider Syntax
providers:
- id: atlascloud:deepseek-ai/DeepSeek-V3-0324
- id: atlascloud:qwen/qwen3-32b
Custom Gateway Example
providers:
- id: atlascloud:deepseek-ai/DeepSeek-V3-0324
config:
apiBaseUrl: https://proxy.example.com/atlas/v1
apiKeyEnvar: MY_ATLASCLOUD_TOKEN
Resources
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.