What models does this prompt chain compare?

This example compares three models available through IBM watsonx.ai: IBM Granite 3.3 8B, Meta Llama 3.3 70B, and Mistral Large.

How do I authenticate with IBM watsonx.ai for this example?

You can authenticate using either IAM credentials (WATSONX_AI_APIKEY plus WATSONX_AI_PROJECT_ID) or a bearer token (WATSONX_AI_BEARER_TOKEN plus WATSONX_AI_PROJECT_ID).

How do I set up and run this example?

Run `npx promptfoo@latest init --example provider-watsonx`, then navigate to the directory and execute `promptfoo eval` to run the comparison and `promptfoo view` to see results.

What prerequisites are needed to use this example?

You need access to IBM watsonx.ai, an IBM Cloud account, and a project ID already set up — this example does not cover initial account setup.

Prompt Chain

Compare IBM WatsonX Models for Code Generation Tasks

Name: Provider Watsonx
Availability: OnlineOnly
Author: Promptfoo

Compares IBM Granite, Meta Llama, and Mistral models available through IBM watsonx.ai in promptfoo.

Copy chain

Works with watsonxibm

Promptfoo

Own this? Claim it

Maintainer of this project? Claim this page to edit the listing.

Spark score

out of 100

Updated 15 days ago

Version 0.121.18

Models

llama 3 mistral large

Add to Favorites

Why it matters

Evaluate and benchmark multiple large language models (IBM Granite, Meta Llama, Mistral) available through IBM watsonx.ai to determine which performs best for your specific use case, enabling data-driven model selection decisions.

Outcomes

What it gets done

Configure authentication with IBM Cloud API keys or bearer tokens

Run comparative evaluations across IBM Granite 3.3, Llama 3.3, and Mistral Large models

Execute standardized test prompts against all three model providers simultaneously

View side-by-side performance results to identify the optimal model for your workload

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-provider-watsonx | bash

Steps

Steps in the chain

Setup IAM Authentication

Run Evaluation

View Results

Overview

Provider Watsonx

Compares IBM Granite 3.3, Meta Llama 3.3, and Mistral Large models available through the IBM watsonx.ai provider in promptfoo. Use when comparing IBM Granite, Llama, and Mistral model responses through IBM watsonx.ai.

What it does

This promptfoo example compares IBM Granite, Meta Llama, and Mistral models available through IBM watsonx.ai. Three specific models are tested side by side: IBM Granite 3.3 8B (the latest recommended Granite model), Meta Llama 3.3 70B (the latest Llama model), and Mistral Large (Mistral's flagship model).

When to use - and when NOT to

Use this example when comparing IBM Granite, Llama, and Mistral model responses through the IBM watsonx.ai provider.

Inputs and outputs

Requires IBM watsonx.ai authentication via either IAM API key (recommended):

export WATSONX_AI_APIKEY=your-ibm-cloud-api-key
export WATSONX_AI_PROJECT_ID=your-project-id

or a bearer token (WATSONX_AI_BEARER_TOKEN plus WATSONX_AI_PROJECT_ID). Run via npx promptfoo@latest init --example provider-watsonx then promptfoo eval (or npm run local -- eval --config examples/provider-watsonx/promptfooconfig.yaml for a local build), then view results with promptfoo view.

Integrations

Built on the IBM watsonx.ai provider, comparing IBM Granite 3.3 8B, Meta Llama 3.3 70B, and Mistral Large.

Who it's for

Engineers evaluating which model available through IBM watsonx.ai - Granite, Llama, or Mistral - best fits their use case, using promptfoo to run the comparison side by side.

Source README

provider-watsonx (IBM WatsonX Model Comparison)

This example compares IBM Granite, Meta Llama, and Mistral models available through IBM watsonx.ai.

You can run this example with:

npx promptfoo@latest init --example provider-watsonx
cd provider-watsonx

Setup

Set up authentication and project ID:

IAM Authentication (Recommended)

export WATSONX_AI_APIKEY=your-ibm-cloud-api-key
export WATSONX_AI_PROJECT_ID=your-project-id

Bearer Token Authentication

export WATSONX_AI_BEARER_TOKEN=your-bearer-token
export WATSONX_AI_PROJECT_ID=your-project-id

Follow the instructions in watsonx.md to retrieve your credentials and project ID.

Running the Example

promptfoo eval

Or with the local build:

npm run local -- eval --config examples/provider-watsonx/promptfooconfig.yaml

Afterwards, view the results:

promptfoo view

Models Tested

IBM Granite 3.3 8B - Latest recommended Granite model
Meta Llama 3.3 70B - Latest Llama model
Mistral Large - Flagship Mistral model

FAQ

Common questions

Discussion