Guide Whisper Transcription with Custom Prompts
Steer Whisper transcription style and spelling with fictitious prompts, including GPT-generated ones.
Why it matters
Enhance audio transcription accuracy and style consistency by leveraging custom prompts with OpenAI's Whisper API. This guide demonstrates how to steer transcription output for specific spellings, capitalization, and stylistic nuances.
Outcomes
What it gets done
Generate custom prompts for Whisper to control transcription style.
Correct misspellings of proper nouns (names, products, companies) in transcripts.
Use GPT to create fictitious prompts for advanced style steering.
Understand Whisper's prompt limitations and best practices.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-whisperpromptingguide | bash Steps
Steps in the chain
Overview
Whisper prompting guide
An OpenAI guide to steering Whisper transcription style and spelling using fictitious prompts - transcript generation via GPT and spelling glossaries - within Whisper's 224-token prompt limit. Use fictitious prompts to fix spelling or nudge style, not to give instructions - Whisper follows the prompt's style, not commands inside it, and cannot override what it actually hears.
What it does
This guide explains OpenAI Whisper's optional prompt parameter, normally meant to pass a prior segment's own transcript so the model maintains context and consistent style across a sequence of chained audio segments. It shows that prompts don't need to be genuine prior transcripts at all - fictitious prompts can steer Whisper toward particular spellings or writing styles, via two techniques: using GPT to generate a fictitious transcript for Whisper to emulate, and supplying a spelling glossary for names of people, products, or companies.
When to use - and when NOT to
Don't treat Whisper prompting like GPT prompting: an instruction like "Format lists in Markdown" is ignored, since Whisper follows the prompt's writing style rather than any instructions written inside it. The prompt is capped at 224 tokens (using Whisper's own multilingual tokenizer) - if you exceed that, only the final 224 tokens are kept and everything earlier is silently dropped, so front-load anything that must survive. Short prompts are less reliable at shifting style than longer ones, and Whisper is less likely to follow rare or atypical styles.
Inputs and outputs
Examples in the guide show a baseline NPR podcast transcription capitalizing "President Biden" by default; a short lowercase prompt only partially shifts the casing, while a longer prompt succeeds reliably. A separate example passes a glossary of product and company names to correct spellings Whisper would otherwise guess at, and another corrects character names (getting "Aimee" and "Shawn" instead of Whisper's guessed "Amy" and "Sean") the same way.
Integrations
GPT itself is a practical way to generate the fictitious transcripts used as prompts - give it instructions and have it write a long, stylistically representative passage for Whisper to imitate. Prompting has real limits: it only shapes ambiguous stylistic choices and spelling, and cannot override what the model actually hears - a prompt won't make a transcript sound like a Southern accent if the speaker isn't actually speaking with one.
Who it's for
Developers transcribing audio with unfamiliar names, brand terms, or specific style requirements (like capitalization or spelling conventions) who need Whisper's output to match a house style without post-processing every transcript by hand. These techniques are explicitly not fully reliable - they work often enough to be useful, especially for common, plausible styles, but aren't a guarantee, so results are still worth spot-checking rather than trusted blindly on high-stakes transcripts.
Source README
Whisper prompting guide
OpenAI's audio transcription API has an optional parameter called prompt.
The prompt is intended to help stitch together multiple audio segments. By submitting the prior segment's transcript via the prompt, the Whisper model can use that context to better understand the speech and maintain a consistent writing style.
However, prompts do not need to be genuine transcripts from prior audio segments. Fictitious prompts can be submitted to steer the model to use particular spellings or styles.
This notebook shares two techniques for using fictitious prompts to steer the model outputs:
- Transcript generation: GPT can convert instructions into fictitious transcripts for Whisper to emulate.
- Spelling guide: A spelling guide can tell the model how to spell names of people, products, companies, etc.
These techniques are not especially reliable, but can be useful in some situations.
Comparison with GPT prompting
Prompting Whisper is not the same as prompting GPT. For example, if you submit an attempted instruction like "Format lists in Markdown format", the model will not comply, as it follows the style of the prompt, rather than any instructions contained within.
In addition, the prompt is limited to only 224 tokens. If the prompt is longer than 224 tokens, only the final 224 tokens of the prompt will be considered; all prior tokens will be silently ignored. The tokenizer used is the multilingual Whisper tokenizer.
To get good results, craft examples that portray your desired style.
Setup
To get started, let's:
- Import the OpenAI Python library (if you don't have it, you'll need to install it with
pip install openai) - Download a few example audio files
As a baseline, we'll transcribe an NPR podcast segment
Our audio file for this example will be a segment of the NPR podcast, Up First.
Let's get our baseline transcription, then introduce prompts.
Transcripts follow the style of the prompt
Let's explore how prompts influence the style of the transcript. In the previous unprompted transcript, 'President Biden' is capitalized.
Let's try and use a prompt to write "president biden" in lower case. We can start by passing in a prompt of 'president biden' in lowercase and see if we can get Whisper to match the style and generate the transcript in all lowercase.
Be aware that when prompts are short, Whisper may be less reliable at following their style. Long prompts may be more reliable at steering Whisper. Let's try that again with a longer prompt.
That worked better.
It's also worth noting that Whisper is less likely to follow rare or odd styles that are atypical for a transcript.
Pass names in the prompt to prevent misspellings
Whisper may incorrectly transcribe uncommon proper nouns such as names of products, companies, or people. In this manner, you can use prompts to help correct those spellings.
We'll illustrate with an example audio file full of product names.
To get Whisper to use our preferred spellings, let's pass the product and company names in the prompt, as a glossary for Whisper to follow.
Now, let's switch to another audio recording authored specifically for this demonstration, on the topic of a odd barbecue.
To begin, we'll establish our baseline transcript using Whisper.
While Whisper's transcription was accurate, it had to guess at various spellings. For example, it assumed the friends' names were spelled Amy and Sean rather than Aimee and Shawn. Let's see if we can steer the spelling with a prompt.
Success!
Let's try the same with more ambiguously spelled words.
Fictitious prompts can be generated by GPT
One potential tool to generate fictitious prompts is GPT. We can give GPT instructions and use it to generate long fictitious transcripts with which to prompt Whisper.
Whisper prompts are best for specifying otherwise ambiguous styles. The prompt will not override the model's comprehension of the audio. For example, if the speakers are not speaking in a deep Southern accent, a prompt will not cause the transcript to do so.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.