Prepare and Analyze Chat Data for Fine-Tuning
Validate chat fine-tuning datasets, check token counts against the 16,385-token limit, and estimate fine-tuning cost.
Why it matters
Streamline your AI development by preparing and analyzing chat datasets for fine-tuning large language models. Ensure your data is correctly formatted and understand its characteristics before training.
Outcomes
What it gets done
Validate chat dataset format against API requirements.
Identify and categorize data errors for debugging.
Calculate token counts for cost estimation.
Analyze message distribution and token limits.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-chatfinetuningdataprep | bash Steps
Steps in the chain
Overview
Data preparation and analysis for chat model fine-tuning
An OpenAI Cookbook notebook that validates chat fine-tuning dataset format, computes token and message statistics, flags oversized examples, and estimates fine-tuning cost. Use it before submitting a chat dataset to the fine-tuning API. Built for the current chat message format - legacy completion-style fine-tuning needs a separate notebook.
What it does
This OpenAI Cookbook notebook preprocesses and analyzes a chat dataset before fine-tuning a chat model such as gpt-3.5-turbo, using the current chat fine-tuning method (a separate legacy fine-tuning path exists for models like babbage-002 and davinci-002). It loads a dataset from a JSONL file, runs a battery of format-validation checks, computes token-count statistics, and estimates the resulting fine-tuning cost.
When to use - and when NOT to
Use this notebook before submitting a chat dataset to the fine-tuning API, to catch format errors and cost surprises ahead of time rather than after a failed or unexpectedly expensive job. It's specifically built around the current chat fine-tuning message format (system/user/assistant roles) - datasets meant for legacy completion-style fine-tuning need the separate legacy notebook instead.
Inputs and outputs
Format validation checks each conversation for seven distinct error types: data_type (entry isn't a dict), missing_messages_list, message_missing_key (missing role or content), message_unrecognized_key (keys beyond role, content, weight, function_call, name), unrecognized_role (not system/user/assistant), missing_content, and example_missing_assistant_message (no assistant turn at all) - the notebook prints a count per error type for debugging.
Beyond validation, a second analysis pass warns on missing system or user messages specifically - since those are what define the assistant's behavior and kick off the conversation, their absence is a distinct concern from the structural checks above. It also summarizes the messages-per-example distribution (a proxy for dialogue complexity), the total-tokens-per-example distribution (the main driver of fine-tuning cost), and tokens specifically in assistant messages (a proxy for how verbose the assistant's responses are), then flags any example over the 16,385-token limit, since oversized examples get truncated during fine-tuning with possible data loss rather than simply rejected outright.
Integrations
The final section estimates total tokens across the dataset to approximate fine-tuning cost - noting that job duration also scales with token count, so a larger dataset means both a bigger bill and a longer wait - and points to openai.com/pricing for current rates. A dedicated set of token-counting utilities is defined early in the notebook specifically so the same counting logic can be reused consistently across the token-limit checks, the per-example distributions, and the final cost estimate, rather than each section implementing its own counting method.
Who it's for
Teams preparing a chat dataset for fine-tuning who want to catch format errors, oversized examples, and cost estimates before submitting the job, rather than discovering problems from a failed or expensive fine-tuning run.
Source README
Data preparation and analysis for chat model fine-tuning
This notebook serves as a tool to preprocess and analyze the chat dataset used for fine-tuning a chat model.
It checks for format errors, provides basic statistics, and estimates token counts for fine-tuning costs.
The method shown here corresponds to the current fine-tuning method for gpt-3.5-turbo.
See legacy fine-tuning for models like babbage-002 and davinci-002.
Data loading
We first load the chat dataset from an example JSONL file.
Format validation
We can perform a variety of error checks to validate that each conversation in the dataset adheres to the format expected by the fine-tuning API. Errors are categorized based on their nature for easier debugging.
- Data Type Check: Checks whether each entry in the dataset is a dictionary (
dict). Error type:data_type. - Presence of Message List: Checks if a
messageslist is present in each entry. Error type:missing_messages_list. - Message Keys Check: Validates that each message in the
messageslist contains the keysroleandcontent. Error type:message_missing_key. - Unrecognized Keys in Messages: Logs if a message has keys other than
role,content,weight,function_call, andname. Error type:message_unrecognized_key. - Role Validation: Ensures the
roleis one of "system", "user", or "assistant". Error type:unrecognized_role. - Content Validation: Verifies that
contenthas textual data and is a string. Error type:missing_content. - Assistant Message Presence: Checks that each conversation has at least one message from the assistant. Error type:
example_missing_assistant_message.
The code below performs these checks, and outputs counts for each type of error found are printed. This is useful for debugging and ensuring the dataset is ready for the next steps.
Token Counting Utilities
Lets define a few helpful utilities to be used in the rest of the notebook.
Data Warnings and Token Counts
With some lightweight analysis we can identify potential issues in the dataset, like missing messages, and provide statistical insights into message and token counts.
- Missing System/User Messages: Counts the number of conversations missing a "system" or "user" message. Such messages are critical for defining the assistant's behavior and initiating the conversation.
- Number of Messages Per Example: Summarizes the distribution of the number of messages in each conversation, providing insight into dialogue complexity.
- Total Tokens Per Example: Calculates and summarizes the distribution of the total number of tokens in each conversation. Important for understanding fine-tuning costs.
- Tokens in Assistant's Messages: Calculates the number of tokens in the assistant's messages per conversation and summarizes this distribution. Useful for understanding the assistant's verbosity.
- Token Limit Warnings: Checks if any examples exceed the maximum token limit (16,385 tokens), as such examples will be truncated during fine-tuning, potentially resulting in data loss.
Cost Estimation
In this final section, we estimate the total number of tokens that will be used for fine-tuning, which allows us to approximate the cost. It is worth noting that the duration of the fine-tuning jobs will also increase with the token count.
See https://openai.com/pricing to estimate total costs.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.