Generate and Test Code with Ollama

This directory contains two promptfoo examples demonstrating different Ollama capabilities. The first, Model Comparison, compares different Ollama models (llama4:scout, llama2-uncensored) against OpenAI models using various prompts and assertions - testing uncensored versus standard models, controversial or edge-case questions, whether models refuse legitimate queries, and prompt format differences between Llama and OpenAI styles. The second, Function Calling, demonstrates Ollama's function calling using a tiny 1B-parameter model (llama3.2:1b), testing an OpenAI-compatible tool format with a weather API function example that extracts a location, validated with an is-valid-openai-tools-call assertion.

Use this to bootstrap a promptfoo eval against locally-run Ollama models, either to compare model behavior (including uncensored vs standard models) or to test tool/function-calling support on a small local model. It assumes Ollama is already installed and running locally, and requires pulling the specific models used by each example (llama4:scout and llama2-uncensored for the comparison example; llama3.2:1b for the function-calling example) before running.

npx promptfoo@latest init --example ollama
cd ollama

Run the comparison example with npx promptfoo@latest eval (or -c promptfooconfig.yaml explicitly); run the function-calling example with npx promptfoo@latest eval -c promptfooconfig.function-calling.yaml. A successful function-calling run reports a 100% pass rate across 3 test successes, with each test generating a tool call such as a get_current_weather function invoked with location and unit arguments. Supported function-calling models listed in the source include llama3.2:1b, llama3.2:3b, llama3.1, llama3.3, and qwen2.5. Results for either example can be viewed in an interactive web UI via npx promptfoo@latest view, showing side-by-side model outputs, pass/fail status per assertion, and token usage and latency metrics.

The directory is structured with a promptfooconfig.yaml for the comparison example, a promptfooconfig.function-calling.yaml for the tool-calling example, a prompts/ folder holding Llama-style and OpenAI-style prompt formats, and a get_current_weather.yaml tool definition. Both examples can be customized: editing prompts, test questions, and assertions for the comparison config, or editing test cities, the tool definition, and target models for the function-calling config.

Developers evaluating locally-run Ollama models - comparing model behavior head-to-head, or validating function/tool-calling support on small models - who want ready-made promptfoo configs instead of writing eval scaffolding from scratch. The repository's own file listing shows the shape of the example: a README.md, the two config files, a prompts/ folder holding llama_prompt.txt and openai_prompt.json, a standalone prompts.txt with additional prompt examples, and get_current_weather.yaml defining the function-calling tool - a small, self-contained layout that is easy to copy into a new project and adapt.