Compare AI Image Generation Models

This promptfoo example compares QuiverAI's Arrow models - Arrow 1.1 and Arrow 1.1 Max - across three workflows: text-to-SVG generation, image-to-SVG vectorization, and a chained GPT Image-2 -> QuiverAI vectorize pipeline. Every workflow is scored with an LLM-as-judge rubric so quality can be compared side-by-side.

export QUIVERAI_API_KEY=your-api-key
export OPENAI_API_KEY=your-openai-key  # Required for the pipeline + llm-rubric grader
npx promptfoo@latest init --example provider-quiverai

The generation suite (npx promptfoo@latest eval) compares Arrow 1.1, Arrow 1.1 Max, and an Arrow 1.1 variant using instructions style guidance side-by-side. The vectorize suite (-c promptfooconfig.vectorize.yaml) converts raster reference images into SVGs with both Arrow models to compare fidelity, using repo-hosted fixture images to keep the walkthrough stable against third-party image-host changes. The pipeline suite (-c promptfooconfig.pipeline.yaml) chains OpenAI's gpt-image-2 (high-quality raster generation) with the QuiverAI vectorize endpoint to produce a coherent icon set, implemented as a custom JS provider in pipeline-provider.js that hits both APIs serially per call, so it runs slower wall-clock than a single-provider eval. A May 2026 live cost reference: the raster step (gpt-image-2) bills at OpenAI's image pricing, the arrow-1.1 vectorize step costs 15 credits, and arrow-1.1-max costs 20 credits - credits flow through to result.metadata.credits for eval budgeting, and current pricing can be checked via GET /v1/models' pricing_credits field since QuiverAI prices are model- and operation-specific.

It demonstrates is-xml assertions to validate SVG structure, a custom llm-rubric with a custom rubricPrompt tailored for SVG-specific evaluation, and streaming enabled by default for faster generation. Common configuration options: for generation, instructions (style guidance separate from the prompt), references (reference images as a URL string, { url }, or { base64 }), and n (1-16 outputs per request); for vectorization, image (override the image input), auto_crop (crop to the dominant subject before vectorizing), and target_size (square resize target, 128-4096px); and shared across both, temperature (0-2, default 1), max_output_tokens (1-131,072), and stream (set false to enable response caching).

Use it as a reference for comparing QuiverAI's Arrow models on text-to-SVG generation or image vectorization quality, or for building a raster-then-vectorize pipeline chaining GPT Image-2 with QuiverAI, with LLM-judged SVG-specific scoring via is-xml and a custom rubric.

It requires both QUIVERAI_API_KEY and OPENAI_API_KEY (the latter needed for both the pipeline step and the llm-rubric grader). The pipeline suite runs noticeably slower than the single-provider suites since each call hits two APIs serially - budget wall-clock time accordingly, and track result.metadata.credits to budget QuiverAI usage cost.

Input is text prompts (for generation) or raster reference images (for vectorization or the pipeline), plus the QUIVERAI_API_KEY and OPENAI_API_KEY environment variables. Output is generated or vectorized SVGs, validated for structure via is-xml and scored via a custom SVG-specific llm-rubric, with credit-cost metadata attached per result for budgeting.

Integrates QuiverAI's Arrow 1.1/Arrow 1.1 Max generate and vectorize endpoints with OpenAI's gpt-image-2 model via a custom JS provider, evaluated through promptfoo's is-xml and llm-rubric assertions.

Developers comparing SVG generation/vectorization quality across QuiverAI's Arrow models, or building a raster-to-vector pipeline combining GPT Image-2 with QuiverAI, who need LLM-judged, SVG-aware evaluation and cost tracking.