Test Databricks Foundation Model APIs
Promptfoo example testing Databricks Foundation Model APIs, including Llama 3.3 text generation and vision models.
Why it matters
Automate the testing of Databricks Foundation Model APIs. Ensure your models are performing as expected by integrating with promptfoo for robust evaluation.
Outcomes
What it gets done
Test Databricks Foundation Model APIs
Integrate promptfoo for automated testing
Evaluate model performance and accuracy
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-provider-databricks | bash Steps
Steps in the chain
Overview
Provider Databricks
A promptfoo example testing Databricks Foundation Model APIs, covering Llama 3.3 text generation with cost tracking and a separate vision-model configuration. Use when your models run on Databricks Foundation Model APIs and you need to validate text or vision output quality and cost through promptfoo.
What it does
This example tests Databricks Foundation Model APIs with promptfoo. It demonstrates using Databricks pay-per-token endpoints (Foundation Models), basic text generation with Llama 3.3, cost tracking with usage context, and simple assertions and quality checks. A dedicated configuration also covers vision-model capabilities.
When to use - and when NOT to
Use this example when your models run on Databricks Foundation Model APIs and you want to evaluate text generation quality and cost via promptfoo, including a separate vision-capable configuration. It is not a Databricks workspace setup guide - it assumes a Databricks workspace with Foundation Model APIs already enabled and a valid access token.
Inputs and outputs
Requires a Databricks workspace with Foundation Model APIs enabled and a Databricks access token, set as DATABRICKS_WORKSPACE_URL (e.g. https://your-workspace.cloud.databricks.com) and DATABRICKS_TOKEN, either in a .env file or the environment:
export DATABRICKS_WORKSPACE_URL=https://your-workspace.cloud.databricks.com
export DATABRICKS_TOKEN=your-databricks-token
Scaffold with npx promptfoo@latest init --example provider-databricks, run with npx promptfoo@latest eval, and view results with npx promptfoo@latest view. For vision capabilities, run the dedicated configuration with npx promptfoo@latest eval -c promptfooconfig.vision.yaml.
Integrations
Integrates Databricks Foundation Model APIs (pay-per-token endpoints) as a promptfoo provider, tested here with Llama 3.3 for text generation, with cost tracked via usage context and validated through simple assertions and quality checks.
Who it's for
Teams running models through Databricks Foundation Model APIs who want to evaluate text-generation quality, track per-request cost, and test vision-model configurations using promptfoo's standard eval workflow.
Source README
provider-databricks (Databricks Provider)
Test Databricks Foundation Model APIs with promptfoo.
Getting Started
You can run this example with:
npx promptfoo@latest init --example provider-databricks
cd provider-databricks
Prerequisites
- A Databricks workspace with Foundation Model APIs enabled
- A Databricks access token
Environment Variables
This example requires the following environment variables:
DATABRICKS_WORKSPACE_URL- Your Databricks workspace URL (e.g., https://your-workspace.cloud.databricks.com)DATABRICKS_TOKEN- Your Databricks access token
You can set these in a .env file or directly in your environment:
export DATABRICKS_WORKSPACE_URL=https://your-workspace.cloud.databricks.com
export DATABRICKS_TOKEN=your-databricks-token
Running the Example
### Run the evaluation
npx promptfoo@latest eval
### View results in the web UI
npx promptfoo@latest view
What This Example Demonstrates
- Using Databricks pay-per-token endpoints (Foundation Models)
- Basic text generation with Llama 3.3
- Cost tracking with usage context
- Simple assertions and quality checks
Vision Models
For vision capabilities, use the dedicated configuration:
npx promptfoo@latest eval -c promptfooconfig.vision.yaml
Learn More
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.