Spark
Sign In
Prompt Chain

Generate and Test Code with Ollama

Ollama evaluation examples for promptfoo that compare censored vs uncensored models and demonstrate function calling with tiny 1B parameter models.

Copy chain
Works with ollama
Promptfoo
Maintainer?
92
Spark score
out of 100
Updated 14 days ago
Version 0.121.18
Models
llama 3
Add to Favorites

Why it matters

Leverage Ollama's capabilities to automate code generation, review, and debugging processes, streamlining your software development lifecycle.

Outcomes

What it gets done

01

Generate code snippets using Ollama.

02

Review generated code for quality and correctness.

03

Debug code with AI assistance.

04

Integrate Ollama into your existing development pipeline.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-ollama | bash

Steps

Steps in the chain

01
Install Ollama
02
Pull required models for comparison
03
Pull required models for function calling
04
Run model comparison evaluation
05
Run function calling evaluation
06
View evaluation results

Overview

Ollama

Two promptfoo evaluation configurations: one comparing censored vs uncensored Ollama models, another demonstrating function calling with 1B parameter models. Use when evaluating Ollama model behavior on edge cases or testing function calling with resource-constrained models. Skip if you need production implementations or are unfamiliar with promptfoo.

What it does

This prompt workflow provides two complete evaluation examples for testing Ollama models with promptfoo. The first example compares different Ollama models (llama4:scout, llama2-uncensored) against OpenAI models using controversial and edge-case questions to validate response behavior. The second example demonstrates function calling capabilities using tiny models like llama3.2:1b.

When to use - and when NOT to

Use this workflow when you need to evaluate whether censored or uncensored Ollama models better suit your application, especially for edge cases and controversial topics. It's ideal when testing function calling with resource-constrained environments since it supports 1B parameter models. Use it when you want side-by-side comparisons with assertions that validate models don't refuse legitimate queries.

Do NOT use this if you need production-ready function implementations - the weather API example is for demonstration only. Avoid this workflow if you're not already familiar with promptfoo's evaluation framework, as it assumes you understand assertion syntax and config structure.

Inputs and outputs

You provide test questions in the tests section of the config files, model selections, and assertion rules. For function calling, you define tool schemas in YAML format.

To get started:

npx promptfoo@latest init --example ollama
cd ollama

Then pull required models:

# For comparison example (default)
ollama pull llama4:scout
ollama pull llama2-uncensored

# For function calling example
ollama pull llama3.2:1b

Run evaluations:

npx promptfoo@latest eval -c promptfooconfig.yaml

For function calling:

npx promptfoo@latest eval -c promptfooconfig.function-calling.yaml

Expected output shows:

 Evaluation complete
Pass Rate: 100.00%
Successes: 3

Integrations

Requires Ollama installation. The comparison example uses llama4:scout and llama2-uncensored models. The function calling example supports models with function calling capabilities including llama3.2:1b, llama3.2:3b, llama3.1, llama3.3, and qwen2.5. The function calling example uses an OpenAI-compatible tool format and includes the is-valid-openai-tools-call assertion. View results with npx promptfoo@latest view.

Who it's for

AI engineers evaluating whether to use censored or uncensored models for their applications, especially those handling sensitive or edge-case content. Developers implementing function calling with resource constraints who need to test tiny models (1B parameters) before scaling up. Teams comparing Ollama models against OpenAI baselines to make informed deployment decisions. The model comparison example includes a tutorial at https://promptfoo.dev/docs/guides/censored-vs-uncensored-ollama/ for those new to this evaluation pattern.

Source README

ollama (Ollama Examples)

This directory contains examples demonstrating different capabilities of Ollama with promptfoo.

You can run this example with:

npx promptfoo@latest init --example ollama
cd ollama

Prerequisites

  1. Install Ollama
  2. Pull the required models:
### For comparison example (default)
ollama pull llama4:scout
ollama pull llama2-uncensored

### For function calling example
ollama pull llama3.2:1b

Available Examples

This directory contains two different Ollama examples:

1. Model Comparison (Default)

Config: promptfooconfig.yaml

Compares different Ollama models (llama4:scout, llama2-uncensored) with OpenAI models using various prompts and assertions.

Running:

npx promptfoo@latest eval

Or with a specific config:

npx promptfoo@latest eval -c promptfooconfig.yaml

What this tests:

  • Compares uncensored vs standard models
  • Tests with controversial/edge-case questions
  • Validates models don't refuse legitimate queries
  • Demonstrates prompt format differences (Llama vs OpenAI)

Tutorial: See the accompanying guide at https://promptfoo.dev/docs/guides/censored-vs-uncensored-ollama/

2. Function Calling

Config: promptfooconfig.function-calling.yaml

Demonstrates Ollama's function calling capabilities using a tiny 1B parameter model.

Running:

npx promptfoo@latest eval -c promptfooconfig.function-calling.yaml

What this tests:

  • Function calling with llama3.2:1b (tiny, fast model)
  • OpenAI-compatible tool format
  • Weather API function example with location extraction
  • Validates tool calls with is-valid-openai-tools-call assertion

Expected output:

✔ Evaluation complete
Pass Rate: 100.00%
Successes: 3

Each test generates a tool call:

[
  {
    "function": {
      "name": "get_current_weather",
      "arguments": "{\"location\":\"Boston\",\"unit\":\"celsius\"}"
    }
  }
]

Supported models: Models with function calling support include llama3.2:1b, llama3.2:3b, llama3.1, llama3.3, and qwen2.5.

Customization

For Comparison Example

Edit the prompts and test cases in promptfooconfig.yaml. You can modify:

  • Models being compared
  • Test questions in the tests section
  • Assertions to validate different behaviors
  • Prompt formats in the prompts/ directory

For Function Calling Example

Edit promptfooconfig.function-calling.yaml to:

  • Change the test cities
  • Modify the tool definition in get_current_weather.yaml
  • Add additional functions
  • Test with different models

File Structure

examples/ollama/
├── README.md                              # This file
├── promptfooconfig.yaml                   # Model comparison (default)
├── promptfooconfig.function-calling.yaml  # Function calling example
├── prompts/
│   ├── llama_prompt.txt                   # Llama-style prompt format
│   └── openai_prompt.json                 # OpenAI chat format
├── prompts.txt                            # Additional prompt examples
└── get_current_weather.yaml               # Tool definition for function calling

Viewing Results

After running an evaluation, view the results in the web UI:

npx promptfoo@latest view

This opens an interactive comparison showing:

  • Side-by-side model outputs
  • Pass/fail status for each assertion
  • Token usage and latency metrics

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.