Run LLM Providers with Promptfoo

This example demonstrates how to evaluate and compare image generation models using promptfoo via fal.ai. It compares two different fal.ai models side by side across 7 creative test-case prompts spanning artistic styles, uses fixed seeds for reproducible outputs, and applies different configuration settings per model. The evaluation produces a comparison table showing model performance across prompts, generated images rendered in markdown format, and execution time/caching behavior.

Use this example when you need to compare image generation quality, settings, and performance across fal.ai models before choosing one for a product feature. It is not a fal.ai account or model-catalog guide beyond what's needed to run the comparison - it assumes an existing fal.ai API key and only documents the promptfoo-side eval wiring.

Get an API key from the fal.ai dashboard, set it as FAL_KEY, and install the client:

npx promptfoo@latest init --example provider-fal
cd provider-fal
export FAL_KEY=your-fal-key-here
npm install @fal-ai/client

Run with promptfoo eval and view results with promptfoo view, which opens the web interface to browse generated images and compare model outputs. Edit promptfooconfig.yaml to try different models from fal.ai's model catalog, add custom prompts, adjust model parameters, add output-quality assertions, or route SDK requests through a proxy via config.client.proxyUrl where required.

Integrates fal.ai's image generation API (via the @fal-ai/client npm package) as a promptfoo provider, comparing multiple fal.ai models within one eval config, with an optional proxy setting for restricted network environments.

Teams evaluating fal.ai image generation models who need a reproducible, side-by-side comparison of output quality, settings, and performance before selecting a model for production use.