What does the E2B Code Eval pipeline do?

It provides a complete pipeline for evaluating code by integrating LLM output with sandboxed execution and metric calculation to assess the quality, correctness, or performance of code snippets.

When should I use this pipeline?

Use it when you need to automatically assess code generated by or for an LLM and verify execution in a safe environment before deploying. Do not use it if you only need simple text-based evaluation without code execution or if your evaluation requires complex, multi-stage human review.

What does the pipeline output?

The outputs include the LLM's response, the results of sandboxed code execution, and calculated metrics based on the execution outcomes.

What is E2B and how does it integrate with this pipeline?

E2B is a platform for running code in secure, ephemeral sandbox environments, which allows this pipeline to safely execute potentially untrusted code.

Prompt Chain

Automate Code Evaluation Pipeline

Name: E2B Code Eval
Availability: OnlineOnly
Author: promptfoo/promptfoo

E2B Code Eval is a promptfoo workflow that generates Python functions with LLMs, executes them in sandboxed E2B environments, and produces JSON metrics

Copy chain

Works with github

promptfoo/promptfoo

Own this? Claim it

Spark score

out of 100

Updated 10 days ago

Source checked Sep 10, 2026

Version 0.123.0

Add to Favorites

Why it matters

Automate the entire code evaluation process, from prompt to LLM execution and sandboxed testing, ensuring code quality and correctness.

Outcomes

What it gets done

Generate code based on prompts.

Execute code in a sandboxed environment.

Evaluate code execution results.

Automate code testing and debugging.

Install

Add it to your toolbox

Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-e2b-code-eval | bash

After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.

Reports

Agent outcome reports

No reports yet

Overview

E2B Code Eval

E2B Code Eval is a multi-step workflow that uses promptfoo to manage LLM prompts, generates Python functions from problem descriptions, executes them safely in E2B sandboxes via e2b-code-interpreter, creates verification unit tests with OpenAI, and produces per-run JSON metrics plus markdown reports. It provides a complete pipeline from prompt to sandboxed execution to evaluation metrics. Use this when you need to evaluate LLM-generated Python code in isolated environments with automated testing and detailed metrics collection. It's ideal for benchmarking code generation models, optimizing code-generation prompts, or building reproducible code evaluation pipelines with safe execution guarantees.

What it does

E2B Code Eval is a complete prompt-to-execution-to-metric pipeline that generates Python code using LLMs, runs it safely in E2B sandboxes, and evaluates the results. It combines promptfoo for prompt management, an LLM provider (like OpenAI) to generate Python functions from problem descriptions, e2b-code-interpreter for sandboxed execution, and automated verification unit tests that re-run in the sandbox to validate correctness.

When to use - and when NOT to

Use this workflow when you need to evaluate LLM-generated code in a safe, isolated environment with automated testing and metrics collection. It's ideal for benchmarking code generation models, building code evaluation pipelines, or testing prompt variations that produce executable Python. Do NOT use this if you need to evaluate non-Python languages or if you require execution environments with specific system dependencies beyond what E2B sandboxes provide.

Inputs and outputs

You provide short problem prompts that describe the Python function you want generated, along with your E2B and OpenAI API keys. The workflow outputs generated Python functions, execution results from the E2B sandbox, per-run JSON metrics written to .promptfoo_results/, and a human-friendly markdown report produced by report.py.

Integrations

This workflow integrates promptfoo for LLM prompt execution and evaluation case management, e2b-code-interpreter for safe sandboxed code execution, and OpenAI (or your chosen LLM provider) for generating both the initial Python functions and small verification unit tests.

Who it's for

This workflow is for ML engineers and researchers evaluating code generation models, prompt engineers optimizing code-generation prompts, and developers building automated code evaluation pipelines. It's particularly useful for teams that need reproducible, safe code execution with detailed metrics.

Getting started

Initialize the example:

npx promptfoo@latest init --example integration-e2b
cd integration-e2b

Set up your environment:

# create & activate venv
python -m venv .venv
source .venv/bin/activate

# install Python packages
pip install --upgrade pip
pip install e2b-code-interpreter
npm i -g promptfoo

Configure your API keys:

# Required
export E2B_API_KEY="e2b_xxx_your_key_here"
export OPENAI_API_KEY="sk_xxx_your_key_here"

# Recommended
export PROMPTFOO_PYTHON="$(pwd)/.venv/bin/python"

Run the evaluation:

promptfoo eval

View results interactively:

promptfoo view

Source README

integration-e2b (E2B Code Evaluation)

What This Example Demonstrates

This example shows a complete prompt→LLM→sandboxed-execution→metric pipeline using:

promptfoo to run LLM prompts and manage evaluation cases.
An LLM provider to generate Python functions from a short problem prompt.
e2b sandboxes (via e2b-code-interpreter) to run generated code safely.
OpenAI step to generate small verification unit tests and re-run them in the sandbox.
Per-run JSON metrics written to .promptfoo_results/ and a human-friendly markdown report produced by report.py.

You can run this example with:

npx promptfoo@latest init --example integration-e2b
cd integration-e2b

Environment Variables

Set these in your shell before running the example.

# Required
export E2B_API_KEY="e2b_xxx_your_key_here"        # e2b sandbox API key
export OPENAI_API_KEY="sk_xxx_your_key_here"     # OpenAI key (or your chosen LLM provider)

# Recommended
export PROMPTFOO_PYTHON="$(pwd)/.venv/bin/python"  # tell promptfoo which Python/venv to use

If you use a different provider name in promptfooconfig.yaml, add that provider's key instead.

Prerequisites

Install and prepare a Python virtual environment, and install the required packages.

# create & activate venv
python -m venv .venv
source .venv/bin/activate

# install Python packages
pip install --upgrade pip
pip install e2b-code-interpreter
npm i -g promptfoo

Running the Example

Activate venv and ensure env vars are set:

source .venv/bin/activate
export E2B_API_KEY="e2b_xxx"
export OPENAI_API_KEY="sk_xxx"
export PROMPTFOO_PYTHON="$(pwd)/.venv/bin/python"