Extract speaker-attributed insights from meeting recordings

Build a Speaker-Aware Meeting Intelligence Pipeline with Audio Diarization

Many organizations already record important conversations, but a plain transcript often is not enough for reliable follow-up. The missing layer is speaker attribution: knowing who raised a concern, who made a commitment, and where the evidence appears in the recording. A speaker-aware transcript lets you separate customer needs from seller follow-up, keep structured evidence references next to action items, and route sensitive commitments into a review workflow before they land in a CRM, ticketing system, or knowledge base.

This pattern is useful whenever the downstream workflow depends on who said what:

• Sales discovery and solution consulting: capture customer requirements, seller commitments, decision criteria, blockers, and next steps with evidence.
• Customer success and account management: turn QBRs, renewal calls, and onboarding sessions into sourced risks, product asks, and follow-up plans.
• Support escalations and incident reviews: preserve the timeline, reported symptoms, owner commitments, and unresolved questions before creating tickets or postmortems.
• Recruiting and interview loops: summarize candidate or interviewer feedback while keeping quotes tied to the right speaker.
• Regulated or high-stakes reviews: add redaction, evidence checks, and human review before storing notes from healthcare, financial services, legal, or compliance-heavy conversations.

For revenue teams, the impact is usually less about generating another summary and more about reducing leakage between the conversation and the system of record. A pipeline like this can help teams capture CRM-ready next steps faster, identify renewal or expansion risks earlier, preserve evidence behind forecast updates, and coach reps or support teams from sourced examples instead of anecdotal notes. The goal is not to automate judgment away; it is to make handoffs, reviews, and follow-up actions more complete and auditable.

This notebook shows how to build a production-style, post-call meeting intelligence pipeline with OpenAI audio diarization. You will:

1. Accept a recorded meeting audio file.
2. Optionally map known speakers using short reference clips.
3. Call gpt-4o-transcribe-diarize with response_format="diarized_json".
4. Normalize the speaker-labeled segments into JSON and Markdown with stable segment IDs.
5. Use structured outputs to extract a meeting brief, decisions, risks, explicit questions, suggested follow-ups, action items, evidence references, and a follow-up email draft.
6. Write reviewable artifacts and a local guardrail report.

The default cells run without an API key using a synthetic diarized transcript. Real audio calls are opt-in so the notebook is safe to review top-to-bottom.

Architecture

Layer	Responsibility	Output
Audio intake	Accept a call recording and optional known-speaker clips.	meeting.wav, Agent=agent.wav
Pipeline runner	Validate inputs, encode references as data URLs, call OpenAI, and write artifacts.	Run metadata and output directory
Diarization	Call gpt-4o-transcribe-diarize with response_format="diarized_json" and chunking_strategy="auto".	Speaker-labeled segments
Transcript normalization	Convert API output into consistent JSON and Markdown with stable segment IDs.	transcript_segments.json, speaker_labeled_transcript.md
Meeting intelligence	Extract summary, decisions, actions, risks, explicit questions, suggested follow-ups, quotes, and follow-up email with structured evidence references.	meeting_intelligence.json, meeting_brief.md
Guardrails and review gate	Redact sensitive fields, verify evidence references, optionally moderate content, and route risky outputs for review.	guardrail_report.json

This is intentionally request-based. The Realtime API is a better fit for live voice UX, browser capture, or telephony streaming. For durable post-call diarization, this pattern uses the Transcriptions API and then runs structured extraction over the speaker-labeled transcript.

Why speaker-aware transcripts matter

The first version of meeting intelligence is often "send a transcript to a model and summarize it." That works for demos, but it breaks down in customer workflows because it loses who said what. A customer may state a requirement, a seller may make a commitment, and a manager may need the difference to be explicit.

Speaker-aware diarization gives the rest of the application better structure:

• Action items can include the speaker who committed to them.
• Risks can quote the exact customer concern.
• Follow-up email drafts can avoid attributing seller commitments to the customer.
• QA reviewers can spot-check speaker attribution by segment ID and timestamp.
• CRM sync jobs can store mechanically verifiable evidence rather than opaque summaries.

Security and guardrails

Meeting intelligence should be treated as a sensitive-data workflow, not just a transcription or summarization task. Raw recordings can contain customer names, commercial terms, support details, health or financial information, and internal strategy. Speaker reference clips can also be sensitive because they are tied to a person's voice. Once the pipeline turns that audio into structured outputs, those outputs may flow into CRM records, support tickets, account plans, dashboards, or review queues.

Security and guardrails matter most when the output can influence a business process. A sales call summary may capture pricing or contractual commitments. A support escalation may include production-impacting incidents or customer credentials. A recruiting debrief may include candidate feedback. A regulated-industry meeting may contain data that needs retention, access-control, or redaction policies. For these workflows, the safest pattern is to minimize raw audio retention, redact sensitive content where appropriate, require evidence-backed outputs, and route risky or low-confidence outputs through human review before downstream writes.

The included regex redaction is intentionally illustrative: it masks basic email and phone patterns only. It is not a complete PII or DLP system. For names, addresses, account identifiers, credentials, health data, financial data, or regulated workflows, use a policy-approved PII/DLP detector and keep a human review gate before downstream writes.

For the structured extraction step, this notebook sets store=False on the Responses API call so the generated meeting intelligence response is not stored as application state. store=False is a useful request-level control, but it is not the same as enabling Zero Data Retention for an organization or project. If your workflow requires stricter retention guarantees, review OpenAI's data controls documentation and confirm the right retention configuration for your use case.

Risk	Guardrail
Recording or speaker-reference misuse	Require consent and policy approval before recording, diarization, or reference-clip use. Treat speaker references as sensitive biometric-adjacent data.
Over-retention of raw audio	Do not save the raw transcription response by default. Keep raw audio and reference clips only as long as needed. Encrypt and restrict access if retained.
Prompt injection inside transcripts	Treat transcript text as untrusted evidence. Keep instructions in the system message and require the model to use only transcript-backed facts.
Unsupported action items or decisions	Use strict structured outputs and require evidence references that point to real segment IDs and quotes.
Sensitive content in generated notes	Run redaction before summarization where possible, then run post-generation checks on the transcript and brief.
Harmful or policy-sensitive content	Optionally call the Moderation API with omni-moderation-latest on transcript text and generated brief text. Moderation detects harmful content; it is not a replacement for privacy review.
Unsafe downstream writes	Do not write directly to CRM, ticketing, or analytics systems from the model output. Put a human review gate in front of medium/high risks, missing evidence, moderation flags, or raw-response retention.
Silent quality drift	Log model versions, prompt versions, schema versions, audio duration, redaction state, moderation state, and reviewer decisions. Sample calls for evals.

Prerequisites

• Python 3.10 or later.
• An OpenAI API key in OPENAI_API_KEY for real audio runs.
• A meeting recording in a supported audio format for real audio runs.
• Audio uploads must be 25 MB or smaller. Supported input formats are mp3, mp4, mpeg, mpga, m4a, wav, and webm.
• Optional: up to four short, single-speaker reference clips. The speech-to-text guide recommends 2-10 second references, encoded as data URLs when sent with multipart form data.

Run the notebook locally

From a local clone of the Cookbook repository, create a virtual environment, install Jupyter and the OpenAI SDK, then launch this notebook:

git clone https://github.com/openai/openai-cookbook.git
cd openai-cookbook
python3 -m venv .venv
source .venv/bin/activate
python -m pip install jupyter "openai>=1.93.0"
export OPENAI_API_KEY="your-api-key"
jupyter notebook examples/audio/speaker_aware_meeting_intelligence/speaker_aware_meeting_intelligence.ipynb

The synthetic demo below uses only the Python standard library and does not call the API. For real audio, you can also install the OpenAI SDK from inside an existing notebook environment:

%pip install "openai>=1.93.0"

Core diarization request

The core API request is intentionally small:

client = OpenAI(timeout=30 * 60)

with open("meeting.wav", "rb") as audio_file:
    stream = client.audio.transcriptions.create(
        model="gpt-4o-transcribe-diarize",
        file=audio_file,
        response_format="diarized_json",
        chunking_strategy="auto",
        stream=True,
        extra_body={
            "known_speaker_names": ["Agent"],
            "known_speaker_references": [to_data_url(Path("agent_reference.wav"))],
        },
    )
    for event in stream:
        if event.type == "transcript.text.segment":
            print(event.speaker, event.text, event.start, event.end)

The important details are:

• Use response_format="diarized_json" when you need segment-level speaker metadata.
• Use chunking_strategy="auto" for audio longer than 30 seconds.
• Use stream=True for completed recordings when you want finalized diarized segments as they become available.
• The Python SDK defaults to a 10-minute read timeout; the helper below uses 30 minutes for longer recordings.
• Pass known speaker names and references together, in the same order.
• Keep reference clips short and single-speaker.

Diarization vs speaker identification

Diarization answers "which voice spoke each segment?" It separates voices inside one recording, but it does not create a permanent identity profile or remember that speaker_0 from one call is the same person as speaker_0 in a later call. Without references, generic labels are still useful because they preserve attribution: the pipeline can distinguish the speaker who raised a requirement from the speaker who made a commitment.

Known-speaker references add an optional identity hint for the current request:

Input	Result
Meeting audio only	The model separates voices, usually with generic labels such as speaker_0 and speaker_1.
Meeting audio plus a named reference clip	Matching segments can use the supplied name; unmatched speakers can remain generic.
A later or historical recording	Pass the reference clip again. Labels do not carry across recordings automatically.

Pass a reference clip with the meeting recording

The meeting recording and the reference clip are separate inputs in one transcription request. Do not concatenate the reference clip onto the meeting audio. The meeting is uploaded as file=...; each reference clip is encoded as a data URL and sent through known_speaker_references with a name in the same position in known_speaker_names.

The helper below wraps that request shape. For example, this passes a meeting recording plus a separate short clip of an internal rep speaking:

meeting_audio = Path("customer_call.wav")
known_speakers = [
    ("Internal rep", Path("internal_rep_reference.wav")),
]

raw_transcription = transcribe_with_diarization(
    audio_file=meeting_audio,
    known_speakers=known_speakers,
)

Use a clean, consented 2-10 second clip with one speaker and minimal background noise. For recurring internal speakers, a production application can keep an access-controlled reference registry and attach the appropriate clip on each request. For historical recordings, run the same flow per recording; a reference clip may come from an older consented call if it is clean and single-speaker. Treat references as sensitive data, evaluate match quality on representative audio, and keep human review for high-stakes downstream writes.

Step 1: Define the structured output schema

Meeting intelligence often feeds systems of record. Use strict structured outputs so downstream code gets a stable shape and unsupported fields are rejected rather than silently accepted. This schema also requires structured evidence_refs: each extracted item must cite a transcript segment_id and a quote from that segment, which lets guardrails verify the grounding mechanically.

Step 2: Build audio and transcript helpers

Known-speaker references are optional. Without them, diarization can still separate speakers, but labels may be generic, such as speaker_0 or speaker_1. With references, the API can map segments to the names you provide.

Use short, clean reference clips with one speaker and minimal background noise. Keep reference clips only when you have consent and a clear business need.

Step 3: Normalize the transcript

The normalized transcript is the contract between audio processing and meeting intelligence. It helps you rerun summarization without retranscribing audio, inspect attribution quality, and keep raw audio retention short.

Each segment gets a stable segment_id such as seg_005. Later, the model must cite those IDs in evidence_refs, and the guardrail step verifies that each cited quote appears in the referenced segment.

The regex redaction helper below is intentionally illustrative: it masks basic email and phone patterns only. It is not a complete PII or DLP system; use a policy-approved detector and human review for sensitive or regulated workflows.

Step 4: Extract structured meeting intelligence

The model gets a speaker-labeled transcript and must use only that transcript as evidence. The safest default is to produce empty arrays instead of plausible but unsupported CRM notes. The schema also uses required-but-nullable fields, such as due_date_or_trigger, inferred_role, and directed_to_speaker, so unknown values stay null instead of being filled with guesses.

For every extracted fact, action item, risk, question, or recommendation, the model returns evidence_refs with a segment_id and quote. This gives reviewers a readable source trail and gives code something concrete to validate.

Step 5: Render a reviewable meeting brief

The review artifact keeps speaker, segment ID, timestamp, and quote evidence next to decisions, risks, and action items so humans can spot-check before anything is written downstream.

Step 6: Add guardrails and write artifacts

The sample writes a guardrail_report.json with local checks for:

• normalized transcript segments;
• basic email and phone PII patterns;
• evidence references that point to real segment IDs and matching quotes;
• medium/high risk outputs;
• optional moderation flags;
• raw transcription response storage.

Step 7: Run the deterministic demo fixture

This section is a deterministic no-network demo, not a model-quality eval. It uses a fixed synthetic diarized transcript and a fixed expected meeting-intelligence object so reviewers can run the notebook without an API key.

What this fixture checks

The fixture exercises the same artifact and guardrail path used by real audio: transcript rendering, JSON writing, Markdown brief rendering, PII redaction helpers, evidence-reference validation, nullable fields, and review routing.

What this fixture does not check

It does not measure transcription quality, diarization accuracy, or model extraction quality on new meetings. The eval sections below add deterministic scoring and an optional LLM-as-judge pattern for that layer.

Step 8: Run with real audio

The next cell is intentionally opt-in. Set RUN_REAL_AUDIO = True, provide your local audio paths, and make sure OPENAI_API_KEY is set.

The Transcriptions API accepts files up to 25 MB. chunking_strategy="auto" segments a valid upload; it does not split an oversized file. For larger meetings, compress to a supported lower-bitrate format or split the recording into bounded files before transcription, then preserve or offset timestamps when combining results.
Long recordings can also outlast the Python SDK default read timeout. The helper streams finalized diarized segments by default and keeps a 30-minute timeout as a backstop. Set stream_transcription=False only when you specifically need one non-streamed response; split unusually long recordings when one request is not operationally reliable.

How readers supply their files

This cookbook is notebook-first; there is no separate .py command in the published artifact. Put the meeting recording and any optional reference clips somewhere the notebook kernel can read. In local Jupyter, that can be a folder beside the notebook or an absolute path on disk. In a hosted notebook, upload the files into the notebook session first.

For example, a reader might have:

audio/
  customer_call.mp3
  internal_rep_reference.wav

Then point the configuration variables at those files:

AUDIO_FILE = Path("audio/customer_call.mp3")
KNOWN_SPEAKERS = {
    "Internal rep": Path("audio/internal_rep_reference.wav"),
}

AUDIO_FILE is the original meeting recording. KNOWN_SPEAKERS maps the label you want in the output to a separate 2-10 second reference clip; the helper sends the clip with the request rather than appending it to the meeting audio. In a production application, the same helper can receive a temporary file created from an upload or downloaded from object storage.

For the first production-style run, keep the setup simple:

• Use one meeting audio file.
• Use chunking_strategy="auto" for longer recordings.
• Add known-speaker references only when you have consent and a clear business need.
• Run redaction before storage.
• Run moderation when harmful-content classification is part of your review policy.

Step 9: Run deterministic smoke and regression checks

These checks are deterministic and do not call the API. Treat them as a smoke test and regression suite for the notebook mechanics, not as an eval of model quality.

Smoke-test checks

The first checks confirm that the notebook writes all expected artifacts and routes medium-risk outputs to review.

Regression checks

The remaining assertions catch regressions in schema nullability, evidence references, unsupported demo claims, response edge cases, redaction, and timestamp formatting.

Step 10: Run deterministic evals

This section runs a small deterministic eval against the labeled demo fixture. It is still not a broad production eval, but it shows how to score extraction quality with reproducible rules before adding model-graded judgments.

The scorers below measure action-item precision/recall, explicit-question precision/recall, unsupported decisions, nullable unknown fields, and evidence-reference validity. For production, replace GOLD_EVAL_LABELS with a larger labeled dataset and run the same scorers across every example.

Step 11: Add optional LLM-as-judge evals

LLM-as-judge evals are useful for grading qualities that deterministic scorers cannot fully capture, such as summary usefulness, missing follow-ups, and whether the brief would help a reviewer. Keep this optional because it calls the API and can vary by judge model. Use it alongside deterministic scorers, not instead of them.

The judge below receives the transcript, the structured output, and a rubric. It returns scores and review findings. Leave RUN_LLM_JUDGE_EVAL = False for the default no-network notebook run.

Production hardening checklist

Use this checklist before turning the sample into a customer workflow:

Concern	Recommendation
Consent	Make sure call recording, diarization, and known-speaker references are permitted in your product, policy, and region.
Raw audio retention	Store raw audio only as long as needed. Persist normalized transcript segments when possible.
Large recordings	Reject files over 25 MB before upload. Compress or split longer meetings, then preserve timestamp offsets when combining results.
PII and DLP	Treat the included email and phone regexes as illustrative only. Use a policy-approved PII/DLP detector and human review for sensitive or regulated workflows.
Speaker references	Treat reference clips as sensitive data. Store minimally, encrypt at rest, and rotate/delete when no longer needed.
Evidence	Require structured evidence references on decisions, risks, and action items. Validate that each reference points to a real segment ID and quote.
Human review	Route high-risk summaries, compliance promises, pricing claims, or contractual terms for review.
Moderation	Use the Moderation API for harmful-content classification when notes may contain unsafe content. Keep privacy and compliance checks separate.
Retry behavior	Retry transient API errors with backoff. Avoid duplicating downstream CRM writes by using idempotency keys.
Observability	Log model names, prompt versions, schema versions, audio duration, latency, redaction status, and reviewer decisions.
Evaluation	Sample calls weekly. Track speaker attribution accuracy, action-item precision, and unsupported-claim rate.

Evaluation guidance for production

The deterministic eval above is intentionally small: it proves that the scoring pattern works on a labeled fixture. Production teams should expand it into a representative eval set before writing outputs to downstream systems. Start with a small, consented set of recordings or transcript fixtures, create human-reviewed labels, and keep a holdout set for regression testing when prompts, schemas, or models change.

Area	Example metrics
Speaker attribution	Speaker-label accuracy, diarization error rate, speaker-turn boundary accuracy, known-speaker match rate.
Transcript grounding	Quote exactness, timestamp correctness, evidence-reference validity, unsupported-claim rate.
Structured extraction	Precision and recall for action items, decisions, risks, explicit questions, suggested follow-ups, and customer requirements.
Safety and privacy	PII redaction recall, moderation flag recall, false-positive review rate, raw-audio retention compliance.
Workflow impact	Time-to-CRM-update, reviewer override rate, follow-up completion rate, renewal or escalation risk detection latency.

A useful first eval is simple: ask reviewers to mark each extracted action item as correct, partially correct, unsupported, or missing from the output. Track precision for generated items and recall against the human-labeled gold set. For quotes and evidence, prefer exact-match or near-exact-match checks against the transcript segment text so that helpful-sounding but unsupported summaries do not pass unnoticed. LLM-as-judge can help grade usefulness and completeness, but keep deterministic grounding checks in the loop because they are easier to reproduce.

Next steps

You can adapt the same pipeline for:

• Customer success handoffs after quarterly business reviews.
• Support escalations where accountability and exact quotes matter.
• Sales discovery calls that feed CRM next steps.
• Recruiting interview debriefs where each interviewer needs sourced notes.
• Healthcare or financial-services workflows with stronger review and retention controls.

For live scenarios, use Realtime for the in-call experience and still run this post-call diarization pipeline when you need durable, evidence-backed meeting intelligence.

Useful docs:

• Audio and speech guide
• Speech-to-text and speaker diarization
• Structured outputs
• Moderation
• Safety best practices
• Realtime guide