What models does this political bias evaluation compare?

It evaluates four models: Grok 4, Google Gemini 2.5 Pro, OpenAI GPT-4.1, and Anthropic Claude Opus 4, scoring each response on a 0-1 scale (0.0 strongly right-wing, 0.5 centrist/neutral, 1.0 strongly left-wing).

What were the key findings about Grok 4's political bias?

Grok 4 is the most right-leaning of the four models but still scores 0.685 (left-leaning); it shows extreme bipolar behavior with 67.9% extreme responses (about 2x more than competitors); and an anti-Musk bias was detected with responses about Musk-affiliated companies being 14.1% harsher than on other corporations.

How much does it cost to run the full multi-judge evaluation?

The full multi-judge analysis (2,500 questions x 4 models x 5 evaluations) runs about 50,000 API calls and is estimated to cost $100-150 total.

What API keys do I need to set up to run this evaluation?

You need to set `XAI_API_KEY`, `GOOGLE_API_KEY`, `OPENAI_API_KEY`, and `ANTHROPIC_API_KEY` to access the four LLM providers.

How can I run a lighter test instead of the full 2,500-question evaluation?

You can extract a 100-question sample with `head -101 political-questions.csv > test-100.csv`, filter by topic using `grep` commands for economic or social questions, and use `--max-concurrency 5` for rate limiting.

Prompt Chain

Assess AI Political Bias

Name: Redteam Grok 4 Political Bias
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example measuring political and corporate bias across Grok 4, Gemini, GPT-4.1, and Claude with 2,500 questions.

Copy chain

Works with github

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 10 days ago

Source checked Sep 10, 2026

Version 0.123.0

Models

gpt 4gemini 2 0claude 3 opus

Add to Favorites

Why it matters

Evaluate the political leanings of large language models like Grok 4 by testing them against a diverse set of political opinion questions.

Outcomes

What it gets done

Measure bias across multiple AI models.

Utilize a dataset of 2,500 political questions.

Identify potential corporate bias in AI responses.

Compare Grok 4's bias to other major AI models.

Install

Add it to your toolbox

Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-redteam-grok-4-political-bias | bash

After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.

Reports

Agent outcome reports

No reports yet

Steps

Steps in the chain

Set Environment Variables

Run the Experiment

Analyze Results

Overview

Redteam Grok 4 Political Bias

This promptfoo example measures political and corporate bias across Grok 4, Gemini 2.5 Pro, GPT-4.1, and Claude Opus 4 using a 2,500-question dataset scored on a left-right scale with multi-judge analysis. Use it to reproduce promptfoo's published bias analysis or as a template for a similar large-scale, multi-model bias measurement; the full run costs roughly $100-150 in API calls.

What it does

This promptfoo example measures political bias across four major AI models - Grok 4, Gemini 2.5 Pro, GPT-4.1, and Claude Opus 4 - using a 2,500-question political-opinion dataset scored on a 0, strongly right-wing, to 1, strongly left-wing, scale, including questions specifically designed to detect corporate bias in each model's answers about its own or competitors' companies.

When to use - and when NOT to

Use it to reproduce or extend promptfoo's own published political-bias analysis of Grok 4, or as a template for building a similar large-scale, multi-model, multi-judge bias measurement of your own. It is resource-intensive: the full basic evaluation runs roughly 10,000 API calls, and the multi-judge analysis (4 models times 4 judges) runs roughly 50,000, estimated at $100-150 - for smaller test runs, trim the question CSV (for example head -101 political-questions.csv) or filter by topic with grep.

Inputs and outputs

Requires XAI_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY, and ANTHROPIC_API_KEY. political-questions.csv supplies 2,500 questions across economic policy, social issues, corporate bias, and AI/technology governance topics; political-bias-rubric.yaml defines a 7-point Likert scoring rubric. Running npx promptfoo eval -c promptfooconfig.yaml --output results.json, or promptfooconfig-multi-judge.yaml for the multi-judge variant, produces scored results, viewable with npx promptfoo view or summarized by the bundled analyze_results_multi_judge.py and generate_political_spectrum_chart.py scripts, which report average bias score, standard deviation, per-topic breakdown, inter-judge agreement, and self-scoring bias.

Integrations

xAI (Grok 4), Google (Gemini 2.5 Pro), OpenAI (GPT-4.1), and Anthropic (Claude Opus 4) as the models under test, plus promptfoo's eval and multi-judge scoring framework and Python analysis scripts for charting the results.

Who it's for

AI researchers and teams who want to measure or monitor political-lean and corporate-bias signals across major LLMs, either to reproduce promptfoo's published findings or to run the same methodology against their own model set. The published results found all four models leaning left of center on the 0-1 scale; Grok 4 scored 0.685, the most right-leaning of the four yet still left-leaning overall, while also showing 67.9% extreme responses, roughly twice the rate of its competitors, and a detectable anti-Musk bias in its answers about Musk-affiliated companies, 14.1% harsher than its treatment of other corporations. Judges also scored their own model's outputs about 0.09 points more favorably on average than they scored other models, a self-scoring bias the multi-judge design is meant to surface. The bundled configs, question bank, and rubric can all be edited to add more models, change judge criteria or temperature, or focus on a specific question category.

Source README

redteam-grok-4-political-bias (Grok 4 Political Bias Red Team)

This example measures the political bias of Grok 4 compared to other major AI models using a comprehensive dataset of 2,500 political opinion questions, including specific questions designed to detect corporate bias in AI responses.

📖 Read the full analysis: Grok 4 Goes Red? Yes, But Not How You Think

You can run this example with:

npx promptfoo@latest init --example redteam-grok-4-political-bias
cd redteam-grok-4-political-bias

Environment Variables

This example requires the following environment variables:

XAI_API_KEY - Your xAI API key for Grok 4
GOOGLE_API_KEY - Your Google API key for Gemini 2.5 Pro
OPENAI_API_KEY - Your OpenAI API key for GPT-4.1
ANTHROPIC_API_KEY - Your Anthropic API key for Claude Opus 4

You can set these in a .env file or directly in your environment.

Quick Start

1. Set Environment Variables

export XAI_API_KEY="your_xai_api_key"
export GOOGLE_API_KEY="your_google_api_key"
export OPENAI_API_KEY="your_openai_api_key"
export ANTHROPIC_API_KEY="your_anthropic_api_key"

2. Run the Experiment

### Full evaluation with all models
npx promptfoo@latest eval -c promptfooconfig.yaml --output results.json

### Multi-judge analysis (4 models × 4 judges)
npx promptfoo@latest eval -c promptfooconfig-multi-judge.yaml --output results-multi-judge.json

3. Analyze Results

### View results in the web UI
npx promptfoo@latest view

### Generate analysis charts
python analyze_results_multi_judge.py
python generate_political_spectrum_chart.py

Results Summary

The experiment reveals:

All models lean left of center (0.5 on our scale)
Grok 4 is the most right-leaning but still scores 0.685 (left-leaning)
Grok shows extreme bipolar behavior with 67.9% extreme responses
Anti-Musk bias detected in Grok's responses about Musk companies

Files

Core Dataset

political-questions.csv - 2,500 political questions covering:
- Economic policy questions (taxation, welfare, regulation)
- Social issue questions (immigration, healthcare, civil rights)
- Corporate bias detection questions targeting major tech companies
- Contemporary political debates on AI, technology, and governance

Configuration Files

promptfooconfig.yaml - Main configuration for basic evaluation
political-bias-rubric.yaml - 7-point Likert scale rubric for political scoring
political-questions.csv - Question bank covering economic, social, and corporate topics

Understanding the Scoring

Each model response is scored on a 0-1 scale:

0.0 = Strongly right-wing position
0.5 = Centrist/neutral position
1.0 = Strongly left-wing position

The analysis includes:

Average bias score across all questions
Standard deviation (measuring consistency vs extremism)
Breakdown by topic area (economic vs social)
Inter-judge agreement analysis
Self-scoring bias detection

Cost Estimate

Running the full experiment:

Basic evaluation: ~10,000 API calls (2,500 questions × 4 models)
Multi-judge analysis: ~50,000 API calls (2,500 questions × 4 models × 5 evaluations)
Estimated cost: $100-$150 for the complete multi-judge analysis

For testing with smaller samples:

### Test with 100 questions
head -101 political-questions.csv > test-100.csv

### Test economic questions only
grep ",economic$" political-questions.csv > economic-only.csv

### Test social questions only
grep ",social$" political-questions.csv > social-only.csv

### Use rate limiting
npx promptfoo@latest eval -c promptfooconfig.yaml --max-concurrency 5

Key Findings

Universal Left Bias: All major AI models (GPT-4.1, Gemini 2.5 Pro, Claude Opus 4, Grok 4) lean left of center
Grok's Instability: Grok 4 shows 2× more extreme responses than competitors
Corporate Overcorrection: Grok is 14.1% harsher on Musk companies than other corporations
Judge Bias: Models score themselves 0.09 points more favorably on average

Customization

Edit configuration files to:

Add more models for comparison
Adjust the judge scoring criteria
Change temperature or other model parameters
Modify the political bias rubric
Focus on specific question categories

FAQ

Common questions

Discussion