Moderate LLM Inputs and Outputs

Llama Guard Moderator Pack

This pack is to utilize Llama Guard to safeguard the LLM inputs and outputs of a RAG pipeline. Llama Guard is an input-output safeguard model. It can be used for classifying content in both LLM inputs (prompt classification) and LLM responses (response classification). This pack can moderate inputs/outputs based on the default out-of-the-box safety taxonomy for the unsafe categories which are offered by Llama Guard, see details below. It also allows the flexibility to customize the taxonomy for the unsafe categories to tailor to your particular requirements, see sample usage scenarios 3 and 4 below.

Llama Guard safety taxonomy:

• Violence & Hate: Content promoting violence or hate against specific groups.
• Sexual Content: Encouraging sexual acts, particularly with minors, or explicit content.
• Guns & Illegal Weapons: Endorsing illegal weapon use or providing related instructions.
• Regulated Substances: Promoting illegal production or use of controlled substances.
• Suicide & Self Harm: Content encouraging self-harm or lacking appropriate health resources.
• Criminal Planning: Encouraging or aiding in various criminal activities.

CLI Usage

You can download llamapacks directly using llamaindex-cli, which comes installed with the llama-index python package:

llamaindex-cli download-llamapack LlamaGuardModeratorPack --download-dir ./llamaguard_pack

You can then inspect the files at ./llamaguard_pack and use them as a template for your own project.

Code Usage

Prerequisites

Llama Guard's source code is located in a gated GitHub repository. What it means is that you need to request access from both Meta and Hugging Face in order to use LlamaGuard-7b, and obtain a Hugging Face access token, with write privileges for interactions with LlamaGuard-7b. The detailed instructions and form to fill out are listed on the LlamaGuard-7b model card. It took me less than 24 hours to get access from both Meta and Hugging Face.

Please note that running LlamaGuard-7b requires hardware, both GPU and high RAM. I tested in Google Colab and ran into OutOfMemory error with T4 high RAM, even V100 high RAM was on the borderline, may or may not run into memory issue depending on demands. A100 worked well.

Download the pack

You can download the pack to a the ./llamaguard_pack directory:

from llama_index.core.llama_pack import download_llama_pack

### download and install dependencies
LlamaGuardModeratorPack = download_llama_pack(
    "LlamaGuardModeratorPack", "./llamaguard_pack"
)

Construct the pack

Before constructing the pack, be sure to set your Hugging Face access token (see Prerequisites section above) as your environment variable.

os.environ["HUGGINGFACE_ACCESS_TOKEN"] = "hf_###############"

You then construct the pack with either a blank constructor, see below, which uses the out-of-the-box safety taxonomy:

llamaguard_pack = LlamaGuardModeratorPack()

Or you can construct the pack by passing in your custom taxonomy for unsafe categories (see sample custom taxonomy at the bottom of this page):

llamaguard_pack = LlamaGuardModeratorPack(custom_taxonomy)

Run the pack

From here, you can use the pack, or inspect and modify the pack in ./llamaguard_pack.

The run() function takes the input/output message string, moderate it through Llama Guard to get a response of safe or unsafe. When it's unsafe, it also outputs the unsafe category from the taxonomy.

moderator_response = llamaguard_pack.run("I love Christmas season!")

Usage Pattern in RAG Pipeline

We recommend you first define a function such as the sample function moderate_and_query below, which takes the query string as the input, moderates it against Llama Guard's default or customized taxonomy, depending on how your pack i