Moderate AI Content with Llama Guard
Promptfoo example moderating content with Meta's LlamaGuard 3/4 models across 14 safety categories on Replicate.
Why it matters
Leverage Llama Guard to automatically moderate AI-generated content, ensuring it adheres to safety guidelines and preventing harmful outputs. This prompt chain acts as a crucial layer of security for AI applications.
Outcomes
What it gets done
Classify user inputs and AI responses for safety violations.
Apply moderation policies to filter out inappropriate content.
Integrate with existing AI pipelines for seamless security enforcement.
Automate the process of content review and flagging.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-llama-guard-moderation | bash Steps
Steps in the chain
Overview
Llama Guard Moderation
A promptfoo example moderating content with Meta's LlamaGuard 3 and 4 models on Replicate across 14 standardized safety categories. Use when you need standardized, category-labeled content moderation for LLM inputs or outputs rather than a custom moderation prompt.
What it does
This example demonstrates how to use Meta's LlamaGuard models for content moderation in promptfoo, using LlamaGuard 3 (and a placeholder for LlamaGuard 4 when available on Replicate) to check content against specific safety categories, with examples of content that might trigger each. LlamaGuard monitors 14 safety categories: S1 Violent Crimes, S2 Non-Violent Crimes, S3 Sex Crimes, S4 Child Exploitation, S5 Defamation, S6 Specialized Advice (dangerous financial/medical/legal advice), S7 Privacy, S8 Intellectual Property, S9 Indiscriminate Weapons (WMD creation), S10 Hate, S11 Self-Harm, S12 Sexual Content, S13 Elections (misinformation), and S14 Code Interpreter Abuse (LlamaGuard 4 only).
When to use - and when NOT to
Use this example when you need to moderate LLM inputs or outputs against a standardized, category-labeled safety taxonomy rather than a custom moderation prompt. It is not a general content-safety policy guide - it documents specifically how to wire LlamaGuard 3/4 into promptfoo's moderation provider.
Inputs and outputs
Get a Replicate API token from replicate.com/account/api-tokens, set it as REPLICATE_API_TOKEN, then scaffold and run:
npx promptfoo@latest init --example provider-replicate/llama-guard-moderation
cd provider-replicate/llama-guard-moderation
followed by promptfoo eval.
Integrations
LlamaGuard 4 is a 12B-parameter model adding the S14 code-interpreter-abuse category, and is the default moderation provider for promptfoo on Replicate; it is used automatically or can be specified explicitly as replicate:moderation:meta/llama-guard-4-12b, and the example config demonstrates S14 detection. For compatibility, LlamaGuard 3 remains available as replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8, covering categories S1-S13 but without S14.
Who it's for
Teams that need standardized, category-labeled content moderation for LLM inputs or outputs - covering violent crime, hate, self-harm, privacy, election misinformation, and (with LlamaGuard 4) code-interpreter abuse - integrated directly into their promptfoo evals via Replicate. Because the category taxonomy is the same one Meta publishes for LlamaGuard itself, teams get a moderation signal that is directly comparable across projects and over time, rather than a bespoke pass/fail judgment that changes meaning every time the underlying prompt or model is swapped. Keeping LlamaGuard 3 available alongside the newer LlamaGuard 4 default also lets teams that specifically need S1-S13 coverage without the S14 code-interpreter-abuse category pin to the older model explicitly instead of being forced onto the newest one.
Source README
provider-replicate/llama-guard-moderation (LlamaGuard Content Moderation)
You can run this example with:
npx promptfoo@latest init --example provider-replicate/llama-guard-moderation
cd provider-replicate/llama-guard-moderation
This example demonstrates how to use Meta's LlamaGuard models for content moderation in promptfoo.
Features
This example shows:
- Using LlamaGuard 3 for content moderation
- Checking for specific safety categories
- Examples of content that might trigger different categories
- Placeholder for LlamaGuard 4 (when available on Replicate)
LlamaGuard Categories
LlamaGuard monitors these safety categories:
| Code | Category | Description |
|---|---|---|
| S1 | Violent Crimes | Content enabling violent crimes |
| S2 | Non-Violent Crimes | Content enabling non-violent crimes |
| S3 | Sex Crimes | Content enabling sex-related crimes |
| S4 | Child Exploitation | Content depicting child abuse |
| S5 | Defamation | Defamatory statements |
| S6 | Specialized Advice | Dangerous financial, medical, or legal advice |
| S7 | Privacy | Sensitive personal information |
| S8 | Intellectual Property | IP violations |
| S9 | Indiscriminate Weapons | WMD creation |
| S10 | Hate | Hateful content |
| S11 | Self-Harm | Content enabling self-harm |
| S12 | Sexual Content | Adult content |
| S13 | Elections | Election misinformation |
| S14 | Code Interpreter Abuse | Code exploitation (LlamaGuard 4 only) |
Setup
Get a Replicate API token from https://replicate.com/account/api-tokens
Set the environment variable:
export REPLICATE_API_TOKEN=r8_your_token_hereRun the evaluation:
promptfoo eval
LlamaGuard 4
LlamaGuard 4 is a 12B parameter model that adds the S14 category for code interpreter abuse detection. It's the default moderation provider for promptfoo on Replicate.
Using LlamaGuard 4:
- It's automatically used as the default moderation provider
- You can explicitly specify it with:
replicate:moderation:meta/llama-guard-4-12b - The example in the configuration file demonstrates S14 category detection
For compatibility, you can still use LlamaGuard 3:
- Specify:
replicate:moderation:meta/llama-guard-3-8b:146d1220d447cdcc639bc17c5f6137416042abee6ae153a2615e6ef5749205c8 - Provides coverage for categories S1-S13
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.