How much does it cost to run the full multi-judge evaluation?

The full multi-judge analysis (2,500 questions x 4 models x 5 evaluations) runs about 50,000 API calls and is estimated to cost $100-150 total.

What API keys do I need to set up to run this evaluation?

You need to set `XAI_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, and `ANTHROPIC_API_KEY` to access the four LLM providers.

How can I run a lighter test instead of the full 2,500-question evaluation?

You can extract a 100-question sample with `head -101 political-questions.csv > test-100.csv`, filter by topic using `grep` commands for economic or social questions, and use `--max-concurrency 5` for rate limiting.

Prompt Chain

Assess AI Political Bias

Name: Redteam Grok 4 Political Bias
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example measuring political bias across Grok 4, GPT-4.1, Gemini 2.5 Pro, and Claude Opus 4 with 2,500 questions.

Copy chain

Works with github

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 18 days ago

Version 0.121.18

Models

gpt 4gemini 2 0claude 3 opus

Add to Favorites

Why it matters

Evaluate the political leanings of large language models like Grok 4 by testing them against a diverse set of political opinion questions.

Outcomes

What it gets done

Measure bias across multiple AI models.

Utilize a dataset of 2,500 political questions.

Identify potential corporate bias in AI responses.

Compare Grok 4's bias to other major AI models.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-redteam-grok-4-political-bias | bash

Steps

Steps in the chain

Set Environment Variables

Run the Experiment

Analyze Results

Overview

Redteam Grok 4 Political Bias

A promptfoo example measuring political bias across Grok 4, GPT-4.1, Gemini 2.5 Pro, and Claude Opus 4 using a 2,500-question dataset and a multi-judge Likert-scale rubric. Use when replicating or extending a rigorous, multi-model, multi-judge political-bias measurement methodology across frontier LLMs.

What it does

This example measures the political bias of Grok 4 compared to other major AI models using a comprehensive dataset of 2,500 political opinion questions, including questions specifically designed to detect corporate bias in AI responses. It evaluates four models - Grok 4, Google Gemini 2.5 Pro, OpenAI GPT-4.1, and Anthropic Claude Opus 4 - scoring each response on a 0-1 scale (0.0 strongly right-wing, 0.5 centrist/neutral, 1.0 strongly left-wing) using a 7-point Likert-scale rubric. The reported findings: all models lean left of center; Grok 4 is the most right-leaning of the four but still scores 0.685 (left-leaning); Grok shows extreme bipolar behavior with 67.9% extreme responses (about 2x more than competitors); an anti-Musk bias was detected in Grok's responses specifically about Musk-affiliated companies (14.1% harsher than on other corporations); and models score themselves about 0.09 points more favorably on average (judge bias).

When to use - and when NOT to

Use this example when you want to replicate or extend a rigorous, multi-model, multi-judge political-bias measurement methodology - covering economic policy, social issues, corporate bias detection, and contemporary AI/technology/governance debates. It is not a one-off bias check - the full multi-judge run is a substantial, costed experiment (2,500 questions across 4 models and multiple judges), so smaller test samples are provided for lighter runs.

Inputs and outputs

Set XAI_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY, and ANTHROPIC_API_KEY, then scaffold with npx promptfoo@latest init --example redteam-grok-4-political-bias. Run the full evaluation with npx promptfoo@latest eval -c promptfooconfig.yaml --output results.json, or the multi-judge analysis (4 models x 4 judges) with npx promptfoo@latest eval -c promptfooconfig-multi-judge.yaml --output results-multi-judge.json. View results with npx promptfoo@latest view, or generate charts with python analyze_results_multi_judge.py and python generate_political_spectrum_chart.py. Core files: political-questions.csv (2,500 questions across economic policy, social issues, corporate bias detection, and AI/tech/governance debates), promptfooconfig.yaml (basic eval config), and political-bias-rubric.yaml (the 7-point Likert scoring rubric). For lighter testing: head -101 political-questions.csv > test-100.csv for a 100-question sample, grep ",economic$"/",social$" to filter by topic, and --max-concurrency 5 for rate limiting.

Integrations

Integrates four LLM providers side by side (xAI Grok 4, Google Gemini 2.5 Pro, OpenAI GPT-4.1, Anthropic Claude Opus 4) with an LLM-judge scoring layer. The full basic evaluation runs about 10,000 API calls (2,500 questions x 4 models); the multi-judge analysis runs about 50,000 calls (2,500 questions x 4 models x 5 evaluations), estimated at $100-150 total. Analysis covers average bias score, standard deviation (consistency vs. extremism), breakdown by topic area, inter-judge agreement, and self-scoring bias detection.

Who it's for

Researchers and teams who need a reproducible, cost-estimated methodology for measuring political bias and self-judging bias across multiple frontier LLMs, and who want to extend it with additional models, adjusted judge criteria, or a modified rubric.

Source README

redteam-grok-4-political-bias (Grok 4 Political Bias Red Team)

This example measures the political bias of Grok 4 compared to other major AI models using a comprehensive dataset of 2,500 political opinion questions, including specific questions designed to detect corporate bias in AI responses.

📖 Read the full analysis: Grok 4 Goes Red? Yes, But Not How You Think

You can run this example with:

npx promptfoo@latest init --example redteam-grok-4-political-bias
cd redteam-grok-4-political-bias

Environment Variables

This example requires the following environment variables:

XAI_API_KEY - Your xAI API key for Grok 4
GOOGLE_API_KEY - Your Google API key for Gemini 2.5 Pro
OPENAI_API_KEY - Your OpenAI API key for GPT-4.1
ANTHROPIC_API_KEY - Your Anthropic API key for Claude Opus 4

You can set these in a .env file or directly in your environment.

Quick Start

1. Set Environment Variables

export XAI_API_KEY="your_xai_api_key"
export GOOGLE_API_KEY="your_google_api_key"
export OPENAI_API_KEY="your_openai_api_key"
export ANTHROPIC_API_KEY="your_anthropic_api_key"

2. Run the Experiment

### Full evaluation with all models
npx promptfoo@latest eval -c promptfooconfig.yaml --output results.json

### Multi-judge analysis (4 models × 4 judges)
npx promptfoo@latest eval -c promptfooconfig-multi-judge.yaml --output results-multi-judge.json

3. Analyze Results

### View results in the web UI
npx promptfoo@latest view

### Generate analysis charts
python analyze_results_multi_judge.py
python generate_political_spectrum_chart.py

Results Summary

The experiment reveals:

All models lean left of center (0.5 on our scale)
Grok 4 is the most right-leaning but still scores 0.685 (left-leaning)
Grok shows extreme bipolar behavior with 67.9% extreme responses
Anti-Musk bias detected in Grok's responses about Musk companies

Files

Core Dataset

political-questions.csv - 2,500 political questions covering:
- Economic policy questions (taxation, welfare, regulation)
- Social issue questions (immigration, healthcare, civil rights)
- Corporate bias detection questions targeting major tech companies
- Contemporary political debates on AI, technology, and governance

Configuration Files

promptfooconfig.yaml - Main configuration for basic evaluation
political-bias-rubric.yaml - 7-point Likert scale rubric for political scoring
political-questions.csv - Question bank covering economic, social, and corporate topics

Understanding the Scoring

Each model response is scored on a 0-1 scale:

0.0 = Strongly right-wing position
0.5 = Centrist/neutral position
1.0 = Strongly left-wing position

The analysis includes:

Average bias score across all questions
Standard deviation (measuring consistency vs extremism)
Breakdown by topic area (economic vs social)
Inter-judge agreement analysis
Self-scoring bias detection

Cost Estimate

Running the full experiment:

Basic evaluation: ~10,000 API calls (2,500 questions × 4 models)
Multi-judge analysis: ~50,000 API calls (2,500 questions × 4 models × 5 evaluations)
Estimated cost: $100-$150 for the complete multi-judge analysis

For testing with smaller samples:

### Test with 100 questions
head -101 political-questions.csv > test-100.csv

### Test economic questions only
grep ",economic$" political-questions.csv > economic-only.csv

### Test social questions only
grep ",social$" political-questions.csv > social-only.csv

### Use rate limiting
npx promptfoo@latest eval -c promptfooconfig.yaml --max-concurrency 5

Key Findings

Universal Left Bias: All major AI models (GPT-4.1, Gemini 2.5 Pro, Claude Opus 4, Grok 4) lean left of center
Grok's Instability: Grok 4 shows 2× more extreme responses than competitors
Corporate Overcorrection: Grok is 14.1% harsher on Musk companies than other corporations
Judge Bias: Models score themselves 0.09 points more favorably on average

Customization

Edit configuration files to:

Add more models for comparison
Adjust the judge scoring criteria
Change temperature or other model parameters
Modify the political bias rubric
Focus on specific question categories

FAQ

Common questions

Discussion