Collect Human Feedback via Slack

An example collecting human feedback via Slack to evaluate AI responses - posting a prompt to a Slack channel via Promptfoo's slack provider alongside an AI provider, so a human's Slack reply becomes a comparison point.

Use it when you want to compare an AI model's responses against real human judgment collected directly in Slack, rather than relying solely on automated grading. It requires creating a Slack App with specific Bot Token Scopes (chat:write, channels:history, channels:read, groups:history, groups:read, im:history, im:read), installing it to your workspace, and inviting the bot to your test channel.

Setup: install the optional @slack/web-api@^8 dependency (requires Node.js 20+, and supports HTTP_PROXY/HTTPS_PROXY/NO_PROXY through Promptfoo's Slack provider), create a Slack App, add the bot scopes above, install to your workspace and copy the Bot User OAuth Token (starts with xoxb-), set it as SLACK_BOT_TOKEN, and invite the bot with /invite @your-bot-name. Configure providers to compare an AI model against Slack:

providers:
  - openai:gpt-4o-mini # AI model
  - slack:C_YOUR_CHANNEL_ID # Human feedback

At eval time: the AI model generates responses, the Slack provider posts prompts to your channel, humans respond in Slack, and results are collected and compared. Find your channel ID via the channel name, then the About tab, then Channel ID at the bottom.

Uses the optional @slack/web-api package as Promptfoo's Slack provider, run alongside any standard AI provider (the example uses openai:gpt-4o-mini) in the same providers list. More advanced configuration - specific user feedback, timeout settings - is covered in Promptfoo's Slack provider documentation.

Teams that want a human-in-the-loop comparison point for AI responses - collected asynchronously via Slack - rather than relying purely on automated LLM-as-judge grading.