Prompt Chain

Collect Human Feedback via Slack

Name: Collect Human Feedback via Slack
Availability: OnlineOnly
Author: Promptfoo

A prompt workflow that collects human feedback via Slack to evaluate AI responses, enabling team-based review of model outputs directly in Slack channels.

Copy chain

Works with slack

Promptfoo

Maintainer?

Spark score

out of 100

Updated 2 days ago

Version code-scan-action-0.1

Models

gpt 4o

Add to Favorites

Why it matters

Streamline AI response evaluation by collecting human feedback directly through Slack. Automate the process of gathering insights to improve AI model performance.

Outcomes

What it gets done

Send AI responses to Slack for review.

Collect human feedback on AI outputs.

Classify feedback for analysis.

Integrate with Slack for seamless communication.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-integration-slack | bash

Capabilities

What this chain does

Notify

Sends alerts or messages via email, Slack, or other channels.

Classify

Labels or categorizes text, files, or data points.

Overview

Integration Slack

What it does

This prompt workflow connects your AI evaluation pipeline to Slack, allowing team members to review and provide feedback on AI-generated responses directly within Slack channels. It sends model outputs to Slack, collects human ratings and comments, and integrates that feedback into your evaluation process.

How it connects

Use this when you need human judgment to validate AI responses before production deployment, when your team already collaborates in Slack, or when you want to gather qualitative feedback from subject-matter experts without requiring them to use separate evaluation tools.

Source README

integration-slack (Slack Integration)

This example shows how to collect human feedback via Slack for evaluating AI responses.

Prerequisites

Install the Slack Web API (optional dependency):
```
npm install @slack/web-api
```
Create a Slack App:
- Go to https://api.slack.com/apps
- Click "Create New App" → "From scratch"
- Give it a name (e.g., "Promptfoo Evaluator")
- Select your workspace
Add Bot Token Scopes:
- Go to "OAuth & Permissions"
- Under "Bot Token Scopes", add:
  - chat:write
  - channels:history
  - channels:read
  - groups:history
  - groups:read
  - im:history
  - im:read
Install to Workspace:
- Click "Install to Workspace"
- Copy the Bot User OAuth Token (starts with xoxb-)

Set Environment Variable:

export SLACK_BOT_TOKEN=xoxb-your-token-here

Invite Bot to Channel:
- In Slack, go to your test channel
- Type /invite @your-bot-name

Running the Example

You can run this example with:

npx promptfoo@latest init --example integration-slack
cd integration-slack

Then update the channel ID in promptfooconfig.yaml and run:

npx promptfoo@latest eval

Configuration

The example compares AI responses with human feedback:

providers:
  - openai:gpt-4o-mini # AI model
  - slack:C_YOUR_CHANNEL_ID # Human feedback

When you run the evaluation:

The AI model generates responses
The Slack provider posts prompts to your channel
Humans respond in Slack
Results are collected and compared

Finding Your Channel ID

In Slack:

Click the channel name at the top
Click "About" tab
Look for "Channel ID" at the bottom

Advanced Usage

For more advanced configurations (specific user feedback, timeout settings, etc.), see the Slack provider documentation.

Discussion