What DeepSeek model does this red team evaluation use?

This example uses DeepSeek R1 0528, accessed through OpenRouter's API.

What API key do I need to run this?

You need an OPENROUTER_API_KEY to run this evaluation.

How do I set up and run this red team evaluation?

Set it up with `npx promptfoo@latest init --example redteam-deepseek`, run the standard evaluation with `promptfoo eval`, and run the red team with `promptfoo redteam run -c redteamconfig.yaml`.

What does this evaluation test for?

It runs both a standard CSV-based evaluation against test cases and an automated red team to identify content-moderation weaknesses in DeepSeek models.

Prompt Chain

Red Team DeepSeek Models for Security Vulnerabilities

Name: Redteam Deepseek
Availability: OnlineOnly
Author: Promptfoo

Run a CSV-based eval and an automated red team against DeepSeek R1 0528 via OpenRouter using promptfoo.

Copy chain

Works with openrouterdeepseek

Promptfoo

Own this? Claim it

Maintainer of this project? Claim this page to edit the listing.

Spark score

out of 100

Updated 16 days ago

Version 0.121.18

Models

deepseek v3

Add to Favorites

Why it matters

Systematically test DeepSeek AI models for security vulnerabilities, content moderation weaknesses, and censorship behaviors through automated adversarial evaluation and red team testing.

Outcomes

What it gets done

Run automated red team attacks against DeepSeek R1 models via OpenRouter

Evaluate model responses from CSV test cases for security issues

Assess content moderation and censorship patterns in model outputs

Generate comprehensive vulnerability reports with visual results

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-redteam-deepseek | bash

Steps

Steps in the chain

Set up OPENROUTER_API_KEY environment variable

Run the standard evaluation

View the evaluation results

Set up OPENROUTER_API_KEY for red team

Edit redteamconfig.yaml

Run the red team evaluation

Overview

Redteam Deepseek

A promptfoo example running both a CSV-driven standard eval and a separate automated red-team pass against DeepSeek R1 0528 accessed through OpenRouter. Use it to evaluate DeepSeek R1 0528's quality and separately probe its content-moderation behavior. Requires an OpenRouter API key.

What it does

This promptfoo example demonstrates two evaluation modes against DeepSeek models accessed through OpenRouter: a standard eval driven by a tests.csv file of test cases, and a separate automated red-team run configured in redteamconfig.yaml. Both target DeepSeek R1 0528 via an OPENROUTER_API_KEY.

When to use - and when NOT to

Use this example when you want to both evaluate DeepSeek's ordinary response quality against a fixed set of test cases and separately probe it adversarially for safety and content-moderation behavior. It requires an OpenRouter API key rather than a direct DeepSeek key, so it's specifically the OpenRouter-hosted DeepSeek R1 0528 endpoint being tested, not a self-hosted or direct-API DeepSeek deployment.

Inputs and outputs

npx promptfoo@latest init --example redteam-deepseek
cd redteam-deepseek

promptfoo eval
promptfoo view

The standard eval reads its cases from tests.csv and is configured via promptfooconfig.yaml; the red team run uses a separate redteamconfig.yaml and is launched with promptfoo redteam run -c redteamconfig.yaml, which you can edit first to adjust test parameters.

Integrations

Set OPENROUTER_API_KEY before running either mode. For context on the motivation behind this example, promptfoo's own DeepSeek Censorship blog post analyzes DeepSeek models' content-moderation behavior in more depth.

Who it's for

Teams evaluating DeepSeek R1 0528 through OpenRouter who want both a standard quality eval and a dedicated adversarial red-team pass, particularly around content-moderation behavior.

Source README

redteam-deepseek (DeepSeek Red Team Evaluation)

This example demonstrates how to run both an eval from a csv file and an automated red team against DeepSeek models.

You can run this example with:

npx promptfoo@latest init --example redteam-deepseek
cd redteam-deepseek

Environment Variables

This example requires the following environment variables:

OPENROUTER_API_KEY - Your OpenRouter API key for accessing DeepSeek R1 0528

You can set this in a .env file or directly in your environment.

Getting Started

Running an eval

Set up your OPENROUTER_API_KEY environment variable
Run the standard evaluation:
```
promptfoo eval
```
View the results:
```
promptfoo view
```

Running the redteam

Set up your OPENROUTER_API_KEY environment variable
Edit redteamconfig.yaml if needed for your test parameters

Run the red team evaluation:

promptfoo redteam run -c redteamconfig.yaml

Configuration

This example includes:

promptfooconfig.yaml - Configuration for standard evaluation tests
redteamconfig.yaml - Configuration for red team evaluation tests
tests.csv - Test cases for evaluating model responses

Additional Resources

For more information about this evaluation and our analysis of DeepSeek models' content moderation, read our detailed DeepSeek Censorship Blog Post.

FAQ

Common questions

Discussion