Which NVIDIA NIM models does this example compare?

This example compares three models: Llama 3.3 70B, Nemotron 70B, and Qwen 2.5 Coder 32B.

What type of assertions does this example use for evaluation?

This example uses deterministic `icontains` and `icontains-any` assertions rather than model-graded rubrics, allowing the entire evaluation to run with only a single API key.

What task does this example demonstrate?

This example demonstrates a summarization-style task comparing the three NVIDIA NIM models.

Prompt Chain

Generate and Review Nvidia Provider Code

Name: Provider Nvidia
Availability: OnlineOnly
Author: Promptfoo

Compares Llama 3.3, Nemotron 70B, and Qwen 2.5 Coder models hosted on NVIDIA NIM in a promptfoo eval.

Copy chain

Works with nvidia

Promptfoo

Maintainer?

Spark score

out of 100

Updated 15 days ago

Version 0.121.18

Models

llama 3 qwen 2 5

Add to Favorites

Why it matters

Streamline your code generation and review process for Nvidia provider integrations. This asset helps ensure code quality and efficiency when working with Nvidia services.

Outcomes

What it gets done

Generate code snippets for Nvidia provider interactions.

Review generated code for potential issues and improvements.

Assist in debugging code related to Nvidia services.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-provider-nvidia | bash

Steps

Steps in the chain

Initialize NVIDIA NIM example project

Set NVIDIA API key environment variable

Run evaluation

View results

Overview

Provider Nvidia

Compares Llama 3.3 70B, Nemotron 70B, and Qwen 2.5 Coder 32B models hosted on NVIDIA NIM on a summarisation task via promptfoo. Use when comparing Llama, Nemotron, or Qwen models hosted on NVIDIA NIM.

What it does

This promptfoo example compares three models hosted on NVIDIA NIM - Llama 3.3 70B, Nemotron 70B, and Qwen 2.5 Coder 32B - on a short summarisation task, using deterministic icontains and icontains-any assertions, so the example runs end-to-end with only NVIDIA_API_KEY set.

When to use - and when NOT to

Use this example when comparing Llama, Nemotron, or Qwen models hosted on NVIDIA NIM on a summarisation task.

Inputs and outputs

Requires an NVIDIA_API_KEY, obtained by signing into build.nvidia.com, opening any model card, and clicking Get API Key. Run via npx promptfoo@latest init --example provider-nvidia then promptfoo eval, and view results with promptfoo view.

Integrations

Built on NVIDIA NIM, hosting Llama 3.3 70B, Nemotron 70B, and Qwen 2.5 Coder 32B in this example. Points to the full NVIDIA provider reference documentation for further configuration.

Who it's for

Engineers comparing Llama, Nemotron, or Qwen models hosted on NVIDIA NIM who want a minimal, single-API-key working example rather than configuring each model provider from scratch.

Source README

provider-nvidia (NVIDIA NIM)

You can run this example with:

npx promptfoo@latest init --example provider-nvidia
cd provider-nvidia

Usage

Set your NVIDIA_API_KEY environment variable. You can get a key from build.nvidia.com - sign in, open any model card, and click Get API Key.

Then run:

promptfoo eval

View the results with promptfoo view.

What this example does

Compares three models hosted on NVIDIA NIM (Llama 3.3 70B, Nemotron 70B, Qwen 2.5 Coder 32B) on a short summarisation task with deterministic icontains and icontains-any assertions, so the example runs end-to-end with only NVIDIA_API_KEY set.

See docs/providers/nvidia.md for the full provider reference.

FAQ

Common questions

Discussion