What model does this Databricks prompt chain use?

This configuration uses the databricks-meta-llama-4-scout-instruct model with a pay-per-token endpoint, temperature set to 0.7, and max_tokens set to 500.

What types of test cases are included in this prompt chain?

The configuration includes three test cases: programming humor (dark mode joke), creative writing (haiku generation about debugging), and knowledge retrieval (Databricks Lakehouse concepts).

What assertion types does this configuration use to validate responses?

The configuration uses four assertion types: contains-any (checks for specific keywords), llm-rubric (evaluates subjective quality), javascript (validates haiku structure by counting lines), and max-length (ensures responses stay under 200 characters).

What framework does this prompt chain require?

This configuration requires the promptfoo evaluation framework and uses promptfoo's config schema for test structure and variable interpolation.

What Databricks configuration is needed to use this prompt chain?

You need to configure a workspaceUrl pointing to your Databricks workspace (format: https://your-workspace.cloud.databricks.com) to connect to the Databricks Foundation Model API endpoint.

Prompt Chain

Test Databricks Foundation Model APIs

Name: Databricks
Availability: OnlineOnly
Author: Promptfoo

A promptfoo config that tests Databricks Foundation Model APIs like Llama 4 Scout with rubric and content checks.

Copy chain

Works with databricks

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated yesterday

Version 0.121.19

Models

llama 3

Add to Favorites

Why it matters

Automate the testing of Databricks Foundation Model APIs. Ensure your AI models are performing as expected before deploying them to production.

Outcomes

What it gets done

Integrate with Databricks Foundation Model APIs

Generate test cases for API endpoints

Review and debug API responses

Automate the testing pipeline

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-databricks | bash

Overview

Databricks

This promptfoo config tests a Databricks Foundation Model API endpoint (Llama 4 Scout Instruct) using keyword checks, an LLM-graded rubric, and a custom JavaScript assertion for haiku format. Use it as a starting template when evaluating a model served through Databricks Foundation Model APIs with promptfoo.

What it does

This is a promptfoo example configuration for testing Databricks Foundation Model APIs. It targets the Databricks pay-per-token endpoint for Llama 4 Scout Instruct, sending a single templated prompt ("You are a helpful AI assistant. Answer the following: {{question}}") and running it through three distinct test cases with different assertion styles.

When to use - and when NOT to

Use this as a starting config when you want to evaluate a model served through Databricks' Foundation Model API endpoints - checking both response quality and specific content requirements - rather than building a promptfoo config for Databricks from scratch. It is a single-provider example (Databricks Llama 4 Scout only), not a multi-provider comparison config.

providers:
  - id: databricks:databricks-meta-llama-4-scout-instruct
    config:
      isPayPerToken: true
      temperature: 0.7
      max_tokens: 500
      workspaceUrl: https://your-workspace.cloud.databricks.com
      usageContext:
        project: 'promptfoo-example'
        team: 'engineering'

Inputs and outputs

The config points at a workspaceUrl (to be replaced with your own Databricks workspace) and sets isPayPerToken: true, temperature: 0.7, and max_tokens: 500, plus a usageContext block for cost attribution by project and team. Three test cases exercise different assertion types: a programming-joke prompt checked with contains-any keywords (light, eyes, bright, see, bugs) and an llm-rubric judging whether the response is funny; a haiku-writing prompt checked with a javascript assertion that counts output lines for rough haiku format, plus an llm-rubric on haiku validity; and a Databricks Lakehouse knowledge question checked with contains-any keywords (unified, data, analytics, warehouse, lake) and a max-length assertion capping the response at 200 characters.

Who it's for

Teams already using Databricks Foundation Model APIs who want a ready-made promptfoo config combining keyword checks, LLM-graded rubrics, and custom JavaScript assertions as a template for their own Databricks-served model evals.

Source README

yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

description: Test Databricks Foundation Model APIs

prompts:

'You are a helpful AI assistant. Answer the following: {{question}}'

providers:

Databricks pay-per-token endpoint for Llama 4

id: databricks:databricks-meta-llama-4-scout-instruct
config:
isPayPerToken: true
temperature: 0.7
max_tokens: 500
Replace with your workspace URL
workspaceUrl: https://your-workspace.cloud.databricks.com
Track usage for cost attribution
usageContext:
project: 'promptfoo-example'
team: 'engineering'

tests:

description: 'Test with a programming joke'
vars:
question: 'Why do programmers prefer dark mode?'
assert:
- type: contains-any
  value: ['light', 'eyes', 'bright', 'see', 'bugs']
- type: llm-rubric
  value: 'Is this a funny response to a programming joke?'
description: 'Test creative writing'
vars:
question: 'Write a haiku about debugging code'
assert:
- type: javascript
  value: |
  // Check if it's roughly haiku format (3 lines)
  const lines = output.split('\n').filter(l => l.trim());
  return lines.length >= 3 ? 'pass' : 'fail';
- type: llm-rubric
  value: 'Is this a valid haiku about debugging?'
description: 'Test knowledge about Databricks'
vars:
question: 'What is a Databricks Lakehouse in one sentence?'
assert:
- type: contains-any
  value: ['unified', 'data', 'analytics', 'warehouse', 'lake']
- type: max-length
  value: 200

FAQ

Common questions

Discussion