What does this eval do?

This promptfoo eval validates SQL generated by an LLM by checking syntactic correctness using the `node-sql-parser` npm package, rather than relying on loose text-similarity checks.

What dependencies do I need to set up?

You need to install `node-sql-parser` via `npm i node-sql-parser` and set the `OPENAI_API_KEY` environment variable.

When should I use this eval?

Use this eval when you're scoring an LLM's ability to generate valid SQL and need actual syntax validation. It's not needed for evals unrelated to SQL generation.

How do I get started with this example?

Run `npx promptfoo@latest init --example eval-sql-validation`, then navigate to the `eval-sql-validation` directory and follow the setup instructions.

Prompt Chain

Validate SQL Queries with Promptfoo

Name: Eval Sql Validation
Availability: OnlineOnly
Author: Promptfoo

Validate that model-generated SQL is syntactically correct using node-sql-parser in a promptfoo eval.

Copy chain

Works with githubpromptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Add to Favorites

Why it matters

Ensure the accuracy and correctness of your SQL queries by automating their validation. This asset helps you catch errors and inconsistencies before they impact your database operations.

Outcomes

What it gets done

Define SQL validation test cases.

Execute SQL queries against a database.

Compare query results against expected outputs.

Generate reports on query validation status.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-sql-validation | bash

Steps

Steps in the chain

Initialize example project

Navigate to project directory

Set up environment and dependencies

Configure promptfoo

Run evaluation

View results

Overview

Eval Sql Validation

Validates model-generated SQL for syntactic correctness in a promptfoo eval using the node-sql-parser library as a peer dependency. Use when an eval is scoring an LLM's SQL generation and needs real syntax validation rather than a loose text check.

What it does

This example demonstrates a promptfoo eval that validates model-generated SQL, using the node-sql-parser npm package as a peer dependency (npm i node-sql-parser) to check syntactic correctness of generated queries.

When to use - and when NOT to

Use this when your eval is scoring an LLM's ability to generate valid SQL and you need actual syntax validation rather than a loose text-similarity check.

Not needed for evals unrelated to SQL generation.

Inputs and outputs

Requires OPENAI_API_KEY and npm i node-sql-parser. Input is promptfooconfig.yaml. Output is a normal promptfoo eval / promptfoo view run.

Integrations

Uses the node-sql-parser library for SQL syntax validation, with OpenAI (OPENAI_API_KEY) generating the SQL under test.

Who it's for

Teams evaluating text-to-SQL generation who need actual SQL syntax validation as part of the eval.

npx promptfoo@latest init --example eval-sql-validation
cd eval-sql-validation

Source README

eval-sql-validation (Sql Validation)

You can run this example with:

npx promptfoo@latest init --example eval-sql-validation
cd eval-sql-validation

Usage

To get started, set your OPENAI_API_KEY environment variable and npm i node-sql-parser to install the peer dependency.

Next, edit promptfooconfig.yaml.

Then run:

promptfoo eval

Afterwards, you can view the results by running promptfoo view

FAQ

Common questions

Discussion