Spark
Sign In
Prompt Chain

Integrate Python with OpenAI API for Promptfoo

Integrate custom Python logic with promptfoo for advanced prompt testing and evaluation using the OpenAI API.

Copy chain
Works with pythonopenai
Promptfoo
Maintainer?
54
Spark score
out of 100
Updated 2 days ago
Version code-scan-action-0.1
Models
gpt 4o
Add to Favorites

Why it matters

Extend promptfoo's capabilities by creating a custom Python provider. This allows seamless integration with the OpenAI API for advanced prompt testing and evaluation.

Outcomes

What it gets done

01

Develop a custom provider using Python.

02

Connect promptfoo to the OpenAI API.

03

Enable programmatic interaction with LLMs for testing.

04

Facilitate code generation and review within promptfoo.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-python-provider | bash

Capabilities

What this chain does

Generate code

Writes source code or scripts from a description.

Debug

Traces errors to their root cause and suggests fixes.

Review code

Analyzes code for bugs, style issues, and improvements.

Overview

Python Provider

What it does

This example demonstrates creating a custom Python provider for promptfoo. It enables the integration of your own Python scripts to interact with external services, such as the OpenAI API, directly within prompt testing workflows.

How it connects

Use this when you require custom Python logic for promptfoo evaluations, like calling external APIs or performing complex data transformations. Avoid it if your needs are met by built-in providers or if you lack custom Python code for integration.

Source README

yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

description: Custom Python provider with functions and multi-format configs

prompts:

  • 'Write a very concise funny tweet about {{topic}}'

providers:

  • id: file://provider.py # defaults to call_api function
    config:
    someOption: foobar

  • id: file://provider.py:some_other_function

  • id: file://provider.py:async_provider
    label: async provider

Demonstrates how to load configuration from various file formats

  • id: 'file://provider.py:call_api'
    config:

    YAML

    settings: 'file://configs/fileConfig.yaml'

    JavaScript file

    formatting: 'file://configs/fileConfig.js:getFormatConfig'
    nested: # Python file
    parameters: 'file://configs/fileConfig.py:get_params'

tests:

  • vars:
    topic: bananas
    assert:
    • type: contains
      value: Bananamax
  • vars:
    topic: fruits
    assert:
    • type: llm-rubric
      value: includes at least one emoji
  • vars:
    topic: turtles
    assert:
    • type: llm-rubric
      value: is funny

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.