Optimize Realtime API for Data-Intensive Function Calls
Handle data-heavy function calls in OpenAI's Realtime API without overwhelming speech-to-speech agents.
Why it matters
Maximize the effectiveness of OpenAI's Realtime API for data-intensive function calls, focusing on scenarios common in speech-to-speech agents.
Outcomes
What it gets done
Enhance performance and reliability of real-time conversational agents.
Address challenges of handling large data volumes in real-time.
Implement strategies for efficient data retrieval and processing.
Break down complex functions into smaller, manageable units.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-data-intensive-realtime-apps | bash Steps
Steps in the chain
Overview
Practical guide to data-intensive apps with the Realtime API
A practical guide for handling data-intensive function calls in OpenAI's Realtime API, covering function decomposition, context-window management, data filtering and formatting, and hint prompts. Use it once a Realtime API agent's function calls return large or deeply nested payloads causing latency or degraded responses. Assumes a working Realtime API setup already exists.
What it does
This cookbook is a practical guide for AI engineers building speech-to-speech and other real-time conversational agents on OpenAI's Realtime API, specifically addressing data-intensive function calls. Since agents often wrap existing APIs that weren't designed with agents in mind, function calls can return large payloads that overwhelm the Realtime API, causing slow responses or failed requests. Using a worked example - an NBA Scouting Agent with a searchDraftProspects function returning a hefty, deeply-nested player-data payload - it walks through four guiding principles for keeping data-heavy calls fast and reliable.
When to use - and when NOT to
Use this guide once your Realtime API agent's function calls are returning large or deeply nested payloads and you're seeing degraded response quality or latency - it assumes you already have a working Realtime API setup and doesn't cover initial setup basics. It's specifically about data volume inside function-call responses, not general Realtime API usage.
Inputs and outputs
The four principles: (1) break unwieldy functions into smaller, sharply-scoped ones - splitting the monolithic searchDraftProspects into a lean search returning only general stats plus a separate getProspectDetails for expanded data; (2) manage context as the conversation unfolds, since sessions can run 30 minutes but the rolling context window holds only about 16,000 tokens - periodically summarize the conversation state to cut cost and latency, and periodically remind the model of its system prompt and available tools via session.update if it starts losing track; (3) filter, flatten, and reformat data - trim responses to only the fields the question needs, flatten repeated nested labels like "ProspectInfo" without losing key information, and prefer clear key-based formats like JSON or YAML over large Markdown tables, which tend to overwhelm the model; (4) follow a data-heavy function call with a targeted hint prompt that tells the model exactly how to interpret and phrase its answer - the example prompt instructs the model to speak conversationally, round stats, skip IDs and URLs, and answer strictly from the provided data.
Integrations
The hint-prompt pattern is implemented by appending the function's result to the conversation as a function_call_output item, then emitting a response.create event carrying the hint as its instructions - this lets the model process the raw data once with the follow-up guidance already in hand, rather than guessing how to summarize it. For context management specifically, the guide points to a companion cookbook on automatic conversation summarization in Realtime sessions.
Who it's for
AI engineers building speech-to-speech or other real-time conversational agents on the Realtime API whose function calls return large or deeply nested payloads and need concrete techniques to keep responses fast and accurate.
Source README
Practical guide to data-intensive apps with the Realtime API
This cookbook serves as a practical guide to help AI Engineers maximize the effectiveness of OpenAI's Realtime API, specifically when dealing with data-intensive function calls. We'll focus on scenarios common in speech-to-speech agents, where vast amounts of data must be handled smoothly and efficiently.
This post won't cover the basics of setting up a Realtime API solution. Instead, you'll gain clear insights and actionable strategies to enhance the performance and reliability of your real-time conversational agents. It addresses specific challenges unique to handling large amounts of data in real-time conversational contexts.
What is the Realtime API?
Before we dive in, let’s quickly recap the API for those who are new. The OpenAI Realtime API is a recent offering that supports low-latency, multimodal interactions-such as speech-to-speech conversations and live transcription. Picture scenarios like real-time voice-based customer support or live movie transcriptions.
What is a data-intensive function call?
Agents need access to tools and relevant data to perform their tasks. For instance, a financial analyst agent might pull real-time market data. In many cases, services already exist in your environment that expose this information through APIs.
Historically, APIs weren’t designed with agents in mind and often return large volumes of data, depending on the service. As engineers, we frequently wrap these APIs with function calls to accelerate agent development-which makes perfect sense. Why reinvent what already exists?
If not carefully optimized, these data-intensive function calls can quickly overwhelm the Realtime API-leading to slow responses or even failures to process user requests.
Setting the stage
Our example centers on an NBA Scouting Agent that calls multiple functions to deliver in-depth analysis of upcoming draft prospects. To demonstrate practical guidelines for Realtime API interactions, we use large, realistic payloads inspired by NBA draft prospects. Below, you’ll find a monolithic searchDraftProspects function defined in the Realtime session to set the stage.
// "Hey, pull up point guards projected in the top 10 in the 2025 draft"
{
"type": "session.update",
"session": {
"tools": [
{
"type": "function",
"name": "searchDraftProspects",
"description": "Search draft prospects for a given year e.g., Point Guard",
"parameters": {
"type": "object",
"properties": {
"sign": {
"type": "string",
"description": "The player position",
"enum": [
"Point Guard",
"Shooting Guard",
"Small Forward",
"Power Forward",
"Center",
"Any"
]
},
year: { type: "number", description: "Draft year e.g., 2025" },
mockDraftRanking: { type: "number", description: "Predicted Draft Ranking" },
},
"required": ["position", "year"]
}
}
],
"tool_choice": "auto",
}
}
The searchDraftProspects function call returns a hefty payload. The example’s structure and size are drawn from real-world scenarios we’ve encountered.
// Example Payload
{
"status": {
"code": 200,
"message": "SUCCESS"
},
"found": 4274,
"offset": 0,
"limit": 10,
"data": [
{
"prospectId": 10001,
"data": {
"ProspectInfo": {
"league": "NCAA",
"collegeId": 301,
"isDraftEligible": true,
"Player": {
"personalDetails": {
"firstName": "Jalen",
"lastName": "Storm",
"dateOfBirth": "2003-01-15",
"nationality": "USA"
},
"physicalAttributes": {
"position": "PG",
"height": {
"feet": 6,
"inches": 4
},
"weightPounds": 205
},
"hometown": {
"city": "Springfield",
"state": "IL"
}
},
"TeamInfo": {
"collegeTeam": "Springfield Tigers",
"conference": "Big West",
"teamRanking": 12,
"coach": {
"coachId": 987,
"coachName": "Marcus Reed",
"experienceYears": 10
}
}
},
"Stats": {
"season": "2025",
"gamesPlayed": 32,
"minutesPerGame": 34.5,
"shooting": {
"FieldGoalPercentage": 47.2,
"ThreePointPercentage": 39.1,
"FreeThrowPercentage": 85.6
},
"averages": {
"points": 21.3,
"rebounds": 4.1,
"assists": 6.8,
"steals": 1.7,
"blocks": 0.3
}
},
"Scouting": {
"evaluations": {
"strengths": ["Court vision", "Clutch shooting"],
"areasForImprovement": ["Defensive consistency"]
},
"scouts": [
{
"scoutId": 501,
"name": "Greg Hamilton",
"organization": "National Scouting Bureau"
}
]
},
"DraftProjection": {
"mockDraftRanking": 5,
"lotteryPickProbability": 88,
"historicalComparisons": [
{
"player": "Chris Paul",
"similarityPercentage": 85
}
]
},
"Media": {
"highlightReelUrl": "https://example.com/highlights/jalen-storm",
"socialMedia": {
"twitter": "@jstorm23",
"instagram": "@jstorm23_ig"
}
},
"Agent": {
"agentName": "Rick Allen",
"agency": "Elite Sports Management",
"contact": {
"email": "rallen@elitesports.com",
"phone": "555-123-4567"
}
}
}
},
// ... Many thousands of tokens later.
]
}
Guiding principles
1. Break down unwieldy functions into smaller ones with clear roles and responsibilities
It almost goes without saying-when building function calls, your top priority is to design clear, well-defined functions. This makes it easy to trim response sizes and avoid overwhelming the model. Each function call should be straightforward to explain, sharply scoped, and return only the information needed for its purpose. Overlapping responsibilities between functions inevitably invites confusion.
For example, we can limit the searchDraftProspects function call to return only general details-such as player stats-for each prospect, dramatically reducing the response size. If more information is needed, the new getProspectDetails function call provides expanded details. There’s no universal solution; the right approach depends on your use case and data model.
{
"tools": [
{
"type": "function",
"name": "searchDraftProspects",
"description": "Search NBA draft prospects by position, draft year, and projected ranking, returning only general statistics to optimize response size.",
"parameters": {
"type": "object",
"properties": {
"position": {
"type": "string",
"description": "The player's basketball position.",
"enum": [
"Point Guard",
"Shooting Guard",
"Small Forward",
"Power Forward",
"Center",
"Any"
]
},
"year": {
"type": "number",
"description": "Draft year, e.g., 2025"
},
"maxMockDraftRanking": {
"type": "number",
"description": "Maximum predicted draft ranking (e.g., top 10)"
}
},
"required": ["position", "year"]
}
},
{
"type": "function",
"name": "getProspectDetails",
"description": "Fetch detailed information for a specific NBA prospect, including comprehensive stats, agent details, and scouting reports.",
"parameters": {
"type": "object",
"properties": {
"playerName": {
"type": "string",
"description": "Full name of the prospect (e.g., Jalen Storm)"
},
"year": {
"type": "number",
"description": "Draft year, e.g., 2025"
},
"includeAgentInfo": {
"type": "boolean",
"description": "Include agent information"
},
"includeStats": {
"type": "boolean",
"description": "Include detailed player statistics"
},
"includeScoutingReport": {
"type": "boolean",
"description": "Include scouting report details"
}
},
"required": ["playerName", "year"]
}
}
],
"tool_choice": "auto"
}
2. As conversations unfold, optimize the context
Realtime conversations allow for generous 30-minute sessions-but the rolling context window only supports ~16,000 tokens (depending on the model snapshot, context window limitations are improving). As a result, you may notice performance gradually decline during extended exchanges. As conversations progress and more function calls are made, the conversation state can expand quickly with both important information and unnecessary noise-so it’s important to focus on keeping the most relevant details. This approach helps maintain strong performance and reduces cost.
i) Periodically summarize the conversation state
Periodically summarizing the conversation as it unfolds is an excellent way to reduce context size-cutting both cost and latency.
See @Minhajul's' epic guide on implementing automatic summarization in Realtime conversations (link).
ii) Periodically remind the the model of its role and responsibilities
Data-heavy payloads can quickly fill the context window. If you notice the model losing track of instructions or available tools, periodically remind it of its system prompt and tools by calling session.update-this keeps it focused on its role and responsibilities.
3. Data processing and optimization
i) Use filtering in your function calls to trim data-heavy responses down to only the essential fields needed to answer the question
Generally, fewer tokens returned by function calls lead to better quality responses. Common pitfalls occur when function calls return excessively large payloads spanning thousands of tokens. Focus on applying filters in each function call, either at the data-level or function-level, to minimize response sizes.
// Filtered response
{
"status": {
"code": 200,
"message": "SUCCESS"
},
"found": 4274,
"offset": 0,
"limit": 5,
"data": [
{
"zpid": 7972122,
"data": {
"PropertyInfo": {
"houseNumber": "19661",
"directionPrefix": "N ",
"streetName": "Central",
"streetSuffix": "Ave",
"city": "Phoenix",
"state": "AZ",
"postalCode": "85024",
"zipPlusFour": "1641"
"bedroomCount": 2,
"bathroomCount": 2,
"storyCount": 1,
"livingAreaSize": 1089,
"livingAreaSizeUnits": "Square Feet",
"yearBuilt": "1985"
}
}
}
]
// ...
}
ii) Flatten hierarchical payloads-without losing key information
Hierarchical payloads from API calls can sometimes include repeated level titles-like "ProspectInfo" or "Stats"-which may add extra noise and make things harder for the model to process. As you explore ways to make your data more efficient, you might try flattening these structures by trimming away some of the unnecessary labels. This can help improve performance, but consider what information is important to keep for your particular use case.
// Flattened payload
{
"status": {
"code": 200,
"message": "SUCCESS"
},
"found": 4274,
"offset": 0,
"limit": 2,
"data": [
{
"prospectId": 10001,
"league": "NCAA",
"collegeId": 301,
"isDraftEligible": true,
"firstName": "Jalen",
"lastName": "Storm",
"position": "PG",
"heightFeet": 6,
"heightInches": 4,
"weightPounds": 205,
"hometown": "Springfield",
"state": "IL",
"collegeTeam": "Springfield Tigers",
"conference": "Big West",
"teamRanking": 12,
"coachId": 987,
"coachName": "Marcus Reed",
"gamesPlayed": 32,
"minutesPerGame": 34.5,
"FieldGoalPercentage": 47.2,
"ThreePointPercentage": 39.1,
"FreeThrowPercentage": 85.6,
"averagePoints": 21.3,
"averageRebounds": 4.1,
"averageAssists": 6.8,
"stealsPerGame": 1.7,
"blocksPerGame": 0.3,
"strengths": ["Court vision", "Clutch shooting"],
"areasForImprovement": ["Defensive consistency"],
"mockDraftRanking": 5,
"lotteryPickProbability": 88,
"highlightReelUrl": "https://example.com/highlights/jalen-storm",
"agentName": "Rick Allen",
"agency": "Elite Sports Management",
"contactEmail": "rallen@elitesports.com"
},
...
}
iii) Experiment with different data formats
The way you structure your data has a direct impact on how well the model processes and summarizes API responses. In our experience, clear, key-based formats like JSON or YAML help the model interpret data more accurately than tabular formats such as Markdown. Large tables, especially, tend to overwhelm the model-resulting in less fluent and less accurate outputs. Still, it’s worth experimenting with different formats to find what works best for your use case.
status:
code: 200
message: "SUCCESS"
found: 4274
offset: 0
limit: 10
data:
- prospectId: 10001
data:
ProspectInfo:
league: "NCAA"
collegeId: 301
isDraftEligible: true
Player:
firstName: "Jalen"
lastName: "Storm"
position: "PG"
heightFeet: 6
heightInches: 4
weightPounds: 205
hometown: "Springfield"
state: "IL"
TeamInfo:
collegeTeam: "Springfield Tigers"
conference: "Big West"
teamRanking: 12
coachId: 987
coachName: "Marcus Reed"
Stats:
gamesPlayed: 32
minutesPerGame: 34.5
FieldGoalPercentage: 47.2
ThreePointPercentage: 39.1
FreeThrowPercentage: 85.6
averagePoints: 21.3
averageRebounds: 4.1
averageAssists: 6.8
stealsPerGame: 1.7
blocksPerGame: 0.3
Scouting:
strengths:
- "Court vision"
- "Clutch shooting"
areasForImprovement:
- "Defensive consistency"
DraftProjection:
mockDraftRanking: 5
lotteryPickProbability: 88
Media:
highlightReelUrl: "https://example.com/highlights/jalen-storm"
Agent:
agentName: "Rick Allen"
agency: "Elite Sports Management"
contactEmail: "rallen@elitesports.com"
4. After data-heavy function calls, follow up with hint prompts
Underlying models often struggle to transition smoothly from data-heavy responses to accurate answers. To improve fluency and accuracy when working with complex data, provide a function call hint immediately after the function call. These hints guide the model on the specific task-teaching it how to interpret key fields and domain-specific values.
The following example illustrates an effective hint prompt.
// Function call hint
let prospectSearchPrompt = `
Parse NBA prospect data and provide a concise, engaging response.
General Guidelines
- Act as an NBA scouting expert.
- Highlight key strengths and notable attributes.
- Use conversational language.
- Mention identical attributes once.
- Ignore IDs and URLs.
Player Details
- State height conversationally ("six-foot-eight").
- Round weights to nearest 5 lbs.
Stats & Draft Info
- Round stats to nearest whole number.
- Use general terms for draft ranking ("top-five pick").
Experience
- Refer to players as freshman, sophomore, etc., or mention professional experience.
- Location & TeamMention hometown city and state/country.
- Describe teams conversationally.
Skip (unless asked explicitly)
- Exact birth dates
- IDs
- Agent/contact details
- URLs
Examples
- "Jalen Storm, a dynamic six-foot-four point guard from Springfield, Illinois, averages 21 points per game."
- "Known for his clutch shooting, he's projected as a top-five pick."
Important: Respond based strictly on provided data, without inventing details.
`;
In practice, we first append the function call result to the conversation. Then, we emit a response from the Realtime API with the hint prompt. Voilà-the model gracefully handles all the information.
// Add new conversation item for the model
const conversationItem = {
type: 'conversation.item.create',
previous_item_id: output.id,
item: {
call_id: output.call_id,
type: 'function_call_output',
output: `Draft Prospect Search Results: ${result}`
}
};
dataChannel.send(JSON.stringify(conversationItem));
// Emit a response from the model including the hint prompt
const event = {
type: 'response.create',
conversation: "none",
response: {
instructions: prospectSearchPrompt # function call hint
}
};
dataChannel.send(JSON.stringify(event));
Wrapping up
Building effective agents with the Realtime API is an ongoing process of exploration and adaptation.
Summary of Key Recommendations
- Filter data: Only include fields and details that are directly relevant to the user’s request or the model’s next step. Trim the rest.
- Flatten and simplify structures: Reduce deeply nested or redundant data. Present information in a way that’s easy for both models and humans to scan.
- Prefer clear, structured formats: Use JSON (or YAML) with consistent field names and minimal noise. Avoid large tables or markdown for data-heavy responses.
- Guide the model with hint prompts: After returning lots of data, follow up with a targeted prompt that explains exactly what the model should extract or summarize.
Remember-experimentation is essential. Realtime models keep improving, and we’ll continue sharing tips to help you get the most out of the Realtime API.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.