Evaluate JSON Output from LLMs
Promptfoo example workflow for evaluating JSON output from language models.
Why it matters
Ensure your Large Language Model outputs are consistently valid JSON. This asset helps you programmatically check and validate the structure and content of JSON responses.
Outcomes
What it gets done
Validate LLM-generated JSON against a schema.
Extract specific data points from JSON output.
Classify JSON output based on predefined criteria.
Automate the evaluation of LLM responses.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-eval-json-output | bash Steps
Steps in the chain
Overview
Eval Json Output
Eval Json Output is a promptfoo example workflow for evaluating JSON-formatted responses from language models. It demonstrates how to use promptfoo's evaluation framework to test LLM outputs and view results. Use this workflow when you want to learn how promptfoo works with JSON outputs from language models. It's an example for understanding evaluation setup and execution during development.
What it does
Eval Json Output is a promptfoo example workflow that works with JSON-formatted responses from language models. It provides a starting point for testing LLM outputs in your development environment.
When to use - and when NOT to
Use this workflow when you want to explore promptfoo's evaluation capabilities with JSON outputs from language models. It's a learning example to help you understand how to set up and run evaluations.
Do NOT use this if you're looking for a production-ready validation system or need real-time processing. This is an example workflow for development and learning purposes.
Inputs and outputs
You provide a promptfooconfig.yaml file and your OPENAI_API_KEY environment variable. The workflow runs evaluations and produces results that you can view through the promptfoo view command.
Integrations
This workflow requires OpenAI's API (OPENAI_API_KEY) and runs within the promptfoo evaluation framework.
Who it's for
This workflow is designed for AI engineers and developers who want to learn how to use promptfoo for evaluating language model outputs, particularly those working with JSON responses.
Getting started
To run this example:
npx promptfoo@latest init --example eval-json-output
cd eval-json-output
Set your OPENAI_API_KEY environment variable, edit the promptfooconfig.yaml file, then execute:
promptfoo eval
After evaluation completes, view the results by running promptfoo view.
Source README
eval-json-output (Json Output)
You can run this example with:
npx promptfoo@latest init --example eval-json-output
cd eval-json-output
Usage
To get started, set your OPENAI_API_KEY environment variable.
Next, edit promptfooconfig.yaml.
Then run:
promptfoo eval
Afterwards, you can view the results by running promptfoo view
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.