Automate Agent Retraining for Production Readiness
OpenAI cookbook building a self-evolving agent retraining loop, grounded in FDA regulatory-document summarization.
Why it matters
Enable autonomous AI agents to evolve beyond proof-of-concept by establishing a repeatable, self-healing retraining loop that captures edge cases and improves performance over time.
Outcomes
What it gets done
Instrument agents with measurable feedback signals for diagnosing failures.
Implement prompt optimization strategies, from manual iteration to automated loops.
Assemble self-healing workflows combining human review, LLM-as-judge evaluations, and iterative prompt refinement.
Adapt reusable prompts, configurations, and evaluation templates for custom environments.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-autonomousagentretraining | bash Steps
Steps in the chain
Overview
Self-Evolving Agents: A Cookbook for Autonomous Agent Retraining
OpenAI cookbook (with Bain) building a self-evolving agent retraining loop - human/LLM-as-judge feedback, automated graders, and GEPA reflective prompt evolution - grounded in a two-agent FDA regulatory-document summarization and compliance-checking use case. Use when an agent needs to keep improving past its proof-of-concept plateau without constant manual prompt rewrites, especially in accuracy-critical regulated domains.
What it does
This OpenAI cookbook (with Bain) introduces a repeatable retraining loop for autonomous agents that have plateaued after proof-of-concept because they depend on humans to diagnose edge cases. It grounds the pattern in a regulated healthcare use case - drafting regulatory documents for pharmaceutical companies - using a simplified two-agent baseline: a summarizer (using file search over an uploaded CMC PDF, evaluated on ~70 sections from a public Hyperpolarized Pyruvate (13C) Injection sample document) and a compliance checker verifying FDA 21 CFR Part 11 compliance, both built on GPT-5 with low reasoning effort. The self-evolving loop itself has four steps: a baseline agent produces outputs, human reviewers or an LLM-as-judge score them, an aggregated eval score is compared against a target threshold (e.g. 0.8) up to a retry limit (e.g. 10), and once a candidate clears the threshold it becomes the new baseline. Section 2 walks through the OpenAI Evals platform UI end to end - upload a dataset, configure an initial prompt (starting as simple as "summarize"), generate outputs, add rating/feedback annotation columns, click Optimize to get an improved prompt, and iterate until quality plateaus. Section 3 automates this with four graders - a chemical-name-preservation check, a summary-length deviation check, a cosine-similarity check against the source, and an LLM-as-judge rubric score (all with 0.8-0.85 pass thresholds) - wired through run_eval/poll_eval_run/parse_eval_run_output helpers, a VersionedPrompt tracking class, and a metaprompt agent that proposes revisions; the loop keeps the prompt with the highest cumulative grader score across attempts. It extends this to compare candidate model versions within the same family (e.g. gpt-5, gpt-5-mini, gpt-5-nano) via a compare_model_candidates function, and finally demonstrates Genetic-Pareto (GEPA) - a reflective prompt-evolution framework that samples agent trajectories, reflects on them in natural language, and evolves the system prompt against separate training/validation sets - via a custom GEPAAdapter. Beyond the self-evolving loop itself, it recommends wiring in a continuous-monitoring step for production: a cron job or lightweight scheduler that periodically checks the data source (e.g. new PDF uploads or database entries) and automatically re-triggers the evaluation-and-optimization loop whenever fresh data appears, so the agent keeps adapting rather than drifting stale between manual runs. The appendix shows the actual evolved prompts from each method (Platform Optimizer, static metaprompt loop, GEPA) for direct comparison.
When to use - and when NOT to
Use this cookbook when an agent needs to keep improving after initial deployment without constant manual prompt rewrites - especially in domains demanding accuracy and auditability, like regulatory or compliance documentation. It applies the OpenAI Platform Optimizer for fast manual iteration, the static-metaprompt loop for lightweight full automation, or GEPA when you need a more generalized, less overfit prompt with empirical validation. It is not a guide to building the underlying summarization or compliance-checking agents themselves - the baseline agent is deliberately simplified - and production use requires adding guardrails and human-in-the-loop approval that this simplified example omits.
Inputs and outputs
Input is a baseline agent, a labeled or gradeable dataset (here, CMC regulatory document sections), and either human feedback or grader definitions. Output is an evolved system prompt (and optionally a chosen model version) selected by aggregate grader score, plus eval run logs, agent traces, and version history for rollback.
Integrations
Built on the OpenAI Evals platform and API (python, text_similarity, and score_model graders), GPT-5 model family (gpt-5, gpt-5-mini, gpt-5-nano), AgentBuilder for constructing the baseline agent visually, the Agents SDK ModelSettings class for model parameter comparison, and the external GEPA framework/GitHub repository for reflective prompt evolution.
You are a summarization assistant. Given a section of text, produce a summary.
Who it's for
ML/AI engineers and solution architects who need agents to keep improving past a proof-of-concept plateau - particularly in regulated domains like pharmaceutical or healthcare documentation where accuracy and auditability are non-negotiable.
Source README
Self-Evolving Agents: A Cookbook for Autonomous Agent Retraining
Overview
Agentic systems often reach a plateau after proof-of-concept because they depend on humans to diagnose edge cases and correct failures. This cookbook introduces a repeatable retraining loop that captures those issues, learns from the feedback, and promotes improvements back into production-like workflows. We ground the approach in a regulated healthcare documentation task, but the patterns generalize to any domain that demands accuracy, auditability, and rapid iteration.
What You Will Learn
- Diagnose why an autonomous agent falls short of production readiness and instrument it with measurable feedback signals.
- Compare three prompt-optimization strategies-from quick manual iteration to fully automated loops-and understand when to reach for each.
- Assemble a self-healing workflow that combines human review, LLM-as-judge evals, and iterative prompt refinement.
Who This Notebook Is For
- ML/AI engineers and solution architects who need to move beyond toy demos.
- Product and delivery teams looking for executable artifacts they can adapt into internal tooling or production pipelines.
How to Work Through This Notebook
- Start with Section 1 to understand the healthcare use case, baseline agent, and system architecture.
- Use Section 2 to practice prompt optimization within the OpenAI Evals interface and collect structured feedback.
- Run Section 3 to automate the optimization loop with graders, evals, and retraining logic.
- Reference the appendix for reusable prompts, configurations, and evaluation templates as you tailor the workflow to your environment.
The notebook is modular-feel free to run sections independently or sequentially as you adapt the retraining loop to your own agents.
1. Use Case Overview: Self-Evolving Agents in Healthcare
Problem Definition
For this cookbook, we focus on a real-world use case: drafting regulatory documents for pharmaceutical companies. These organizations must prepare and submit extensive documentation to regulatory authorities (e.g., the U.S. Food and Drug Administration) to obtain approval for new drugs. The accuracy and speed of these submissions are critical, as they directly impact how quickly life-saving treatments can reach patients.
Regulatory document drafting is a highly complex, iterative, and precision-driven process that requires deep scientific, medical, and compliance expertise. Despite the availability of advanced authoring tools, it remains labor-intensive and prone to human error. Agentic systems offer substantial leverage by assisting with research synthesis, content generation, and document structuring, yet human experts are still needed to ensure factual accuracy and regulatory compliance.
The key challenge is to design a feedback loop that enables these agentic systems to learn iteratively and refine model behavior over time. Such a system can gradually shift human effort from detailed correction to high-level oversight, improving efficiency while maintaining the rigorous standards required for regulatory submissions.
Self-evolving Agent
The diagram below illustrates the iterative process for continuously improving an AI agent through feedback, meta prompting, and evaluation. The loop combines human judgment or automated feedback using an LLM-as-a-judge to iteratively enhance performance.
Figure 1 - Diagram showing the self-evolving loop for automated agent improvement.
The process consists of the following steps:
Baseline Agent
The process begins with a baseline agent. In this notebook, we use a deliberately simple example (an agent that summarizes sections of a document) to illustrate the iterative improvement loop. In real-world or enterprise settings, the baseline agent could be much more complex. The summaries it produces serve as the initial benchmark for subsequent evaluation and refinement.Human Feedback (or LLM-as-judge)
The baseline agent’s outputs are then evaluated either by human reviewers (e.g., for production environments) and/or by an automated LLM-as-judge system. This step gathers both quantitative and qualitative feedback that indicates how well the agent meets its goals - for instance, if we are testing the length of the summary, the feedback might be “the summary is too long” or a numerical score (generally between0and1) generated by eval when assessing if the summary is under 500 words.Evals and Aggregated Score
Based on the collected feedback, new prompts are generated and tested through evaluations (Evals). These tests measure performance against predefined criteria, and the outcomes are combined into an aggregated score that reflects the overall performance. The loop continues until the score exceeds a target threshold (e.g.,0.8) or the maximum number of retries is reached (e.g.,max_retry = 10). If the retry limit is hit, engineers are alerted that manual improvements are required.Updated Baseline Agent
Once an improved version achieves the target performance, it replaces the original baseline agent. This updated agent becomes the foundation for the next iteration, supporting a continuous cycle of learning, feedback, and optimization.
Dataset Overview
The dataset used for evaluation comprises ~70 sections extracted from the Sample CMC Section for Hyperpolarized Pyruvate (13C) Injection, publicly available here. This dataset provides realistic, domain-specific content suitable for testing both scientific summarization and regulatory compliance behavior.
Baseline Agent Overview
To keep this cookbook self-contained and easily reproducible, we simplified the regulatory drafting use case while retaining its essential complexity. In production, a typical regulatory authoring agent comprises multiple specialized sub-agents responsible for tasks such as drafting, data analysis, compliance checking, citation generation, and fact verification.
For this guide, we narrow the scope of the regulatory authoring agent to focus on the self-healing aspect of the system. Our regulatory authoring agent consists of two sub-agents:
- A summarizer creating scientific and concise summaries.
- A compliance checker: evaluating each summary against key regulatory requirements (e.g., FDA 21 CFR Part 11).
Figure 2 - The baseline agent as created in the AgentBuilder UI.
For the remainder of this cookbook, we implemented a simplified version of the Summarizer agent (see the section Agent Setup below). Alternatively, you can reuse the code for the agent created with AgentBuilder. If you’d like to reproduce the agent directly from the AgentBuilder UI, here are the key prompts and parameters used:
Summarizer agent: This agent used the file search tool, where the CMC PDF was uploaded to the vector store.
Prompt: "Summarize section {{workflow.input_as_text}} from {{state.cmc_pdf}} uploaded to the vector store."
Compliance Checker agent:
Prompt: "Verify that the summary below is compliant with FDA 21 CFR Part 11: {{input.output_text}}. If the summary is compliant, return Compliant. Otherwise, return This section needs to be manually summarized."
Both agents were configured with the default parameters - using GPT-5, low reasoning effort, and text as the output format.
Evaluation Approach
To evaluate the baseline agent, there are two main approaches:
Collecting Human Feedback. This approach involves gathering feedback from human users through the OpenAI Evals platform (or a custom UI built for a specific application). It is best suited for production settings or when piloting a tool where subject matter experts (SMEs) interact with the tool in real-world scenarios. This method helps uncover edge cases that may not have been identified during development. On the Evals platform, users can provide thumbs-up or thumbs-down ratings and share qualitative feedback about the summaries.
Using an LLM-as-a-Judge. This option is typically used during the development phase, enabling fast feedback loops without requiring SME's time. An LLM-as-a-judge uses an LLM to automatically evaluate and score the agent’s outputs based on predefined criteria. It can also be used for monitoring model drift (e.g., in production) or validating changes between model and model versions (e.g., switching between
gpt-5andgpt-5-mini).
This cookbook demonstrates both approaches:
- Section 2 shows the platform UI approach for manual prompt optimization
- Section 3 implements the fully automated API approach using LLM-as-a-judge
Note: The Evals platform does not yet provide an API to retrieve user feedback programmatically.
2. Using the OpenAI Evals Platform
The OpenAI Evals platform provides an intuitive interface for prompt optimization and evaluation. This section demonstrates the complete workflow from dataset upload through iterative prompt improvement, showing how you can leverage the platform's visual interface to optimize your prompts before implementing automated solutions.
Step 1: Upload Dataset
To begin using the OpenAI Evaluation platform, you'll first need to upload your dataset:
- Click the + Create button
- Define the dataset name
- Upload a CSV file and select the columns to keep
- Upload
Your dataset should contain the documents or document sections that need to be summarized. Each row represents one input that will be processed by your system.
Step 2: Explore Your Data
Once uploaded, you can explore your dataset. Click the dataset name to explore the uploaded data. This allows you to verify that your data is properly formatted and contains the expected content before proceeding with prompt configuration.
Step 3: Configure Initial Prompt
This is where you define your initial system prompt and configure how data flows through your model.
Figure 3 - The platform's "New prompt" interface showing model configuration, variables, and system message settings.
Configuration Steps
- System Prompt: Add the system message that defines the model's task and behavior (this prompt will be optimized)
- User Prompt Template: Add the prompt message template for user messages, using variables such as
{{<column_name>}}that get replaced with actual data from your dataset - Model Selection: Choose the model for generation (e.g., gpt-4.1, gpt-5)
- Temperature: Configure creativity vs. determinism
You can start with a very simple prompt to demonstrate the power of the optimization process. For example, beginning with just "summarize" shows how the system can evolve from a minimal starting point.
Step 4: Generate Outputs
Once your prompt is configured, you're ready to generate outputs across your dataset. The prompt will run once per row and output will be generated on a new output column.
- Click "Generate Output"
- The platform runs your prompt against all samples
- Results appear in a new Output column
The platform will process each row in your dataset, replacing template variables with actual values and calling the model with your system prompt. This creates a baseline of outputs that you can evaluate.
Step 5: Review and Evaluate
Evaluation is where you provide structured feedback to guide prompt improvement.
Review Outputs
Add Evaluation Columns if not automatically added - Click "Columns" → "Annotations" → "Add":
- Rating - Binary (good/bad) or numeric ratings
- Feedback - Text describing what needs improvement
Provide Rating and Feedback - Add your assessment for each output.
Depending on the quality of the output, you may select a good or bad rating and explain your score based on how you would like the answer to be improved. For example:
(Rating) | Feedback
- (Good) Good, but only the answer should be provided. The output should not include headers or any text other than the answer.
- (Bad) The information is good, but it should be presented as bullet points.
- (Good) Good summary; it is clear.
- (Bad) Use bullet points when answering to improve readability. Summarize each sub-section individually.
Save Annotations - Your feedback is saved with the evaluation run
Figure 4 - The evaluation interface showing generated outputs with rating and feedback columns for annotation.
This structured feedback becomes the foundation for automatic prompt optimization.
Step 6: Optimize Prompt
After collecting feedback, the platform can automatically generate an improved prompt.
- Click "Optimize"
- A new prompt version is generated in a new tab
- Click "View Prompt" to see the improved version
Figure 5 - The improved prompt generated by the platform, showing detailed instructions and requirements.
Step 7: Iterate and Compare
With your improved prompt ready, start a new iteration to measure improvement.
- Click "Generate Output"
- Review the new results and provide feedback on any remaining issues
- Click "Optimize" again if needed
- Repeat until satisfied
The platform's tab structure allows you to compare performance across iterations. You can easily see how outputs evolved from your initial prompt to the optimized versions.
Figure 6 - Feedback and evaluation results for the optimized prompt, showing improvements in output quality.
When to Stop Iterating
Continue the optimization cycle until:
- Quality threshold reached: >80% of outputs receive positive feedback
- Diminishing returns: New iterations show minimal improvement
- Specific issues resolved: All identified failure modes are addressed
This platform-based approach provides an excellent foundation for understanding prompt optimization before moving to automated implementations. The visual interface makes it easy to see the impact of changes and understand the optimization process.
3. Self-evolving Loop with LLM-as-a-Judge
This section introduces a fully automated evaluation workflow using an LLM-as-a-Judge through the OpenAI API, eliminating the need for any user interface. This approach enables scalable, programmatic assessment of agent performance, supporting rapid iteration and continuous model monitoring in production.
Eval Creation
To evaluate the baseline summarization agent, we use four complementary graders that balance deterministic checks with semantic judgment.
| Grader | Type | Pass threshold | What it checks | Why |
|---|---|---|---|---|
| Chemical string name | python |
0.8 | If any exact chemical names in the section appear in the summary. | Forces preservation of critical domain entities so summaries don’t omit chemically meaningful terms. |
| Summarization length | python |
0.85 | Inverse deviation from an expected 100-word length. | Keeps summaries concise and comparable, reducing verbosity that can mask poor content. |
| Cosine similarity | text_similarity |
0.85 | Cosine similarity between section and summary texts. | Ensures the summary stays anchored to the source content rather than drifting semantically. |
| LLM-as-judge | score_model |
0.85 | A rubric-driven score from a model acting as an evaluator. | Captures nuanced quality signals that rule-based metrics miss, improving overall robustness. |
Notes
- The two Python graders catch domain fidelity and length discipline early, which stabilizes optimization before semantic tuning.
- Text similarity guards against superficial rephrasing that strays from the source.
- The LLM judge provides a holistic failsafe when edge cases slip past deterministic checks.
You should see an eval ID in the output, e.g. eval_.... This is the ID of the eval we just created (as shown below)
Figure 7 - The platform's Eval interface showing data source configuration, and test criteria settings.
Grader Scoring and Parsing
Next we'll need run the evals on the summarization agent's output and parse the results for the eval's grader scores. To do this we'll use a few helper functions:
run_eval: Simple runner to call the evals API with proper formattingpoll_eval_run: A polling utility to wait for the scheduled eval run to completeparse_eval_run_output: Parses the eval run and returns a structured output for the feedback loop
Now we can use the created eval ID from earlier and run the graders against an arbitrary input section and summary output. This forms the backbone of the feedback loop which will kick off the prompt optimization routine.
Eval execution run
Let's test our evals by providing a section and a generated summary directly.
You should see a list of grader scores in the output, e.g.
[{'grader_name': 'chemical_name_grader-<uuid>', 'score': 0.5, 'passed': False, 'reasoning': None}, {'grader_name': 'word_length_deviation_grader-<uuid>', 'score': 0.8, 'passed': True, 'reasoning': None}, {'grader_name': 'cosine_similarity-<uuid>', 'score': 0.9104484223477793, 'passed': True, 'reasoning': None}, {'grader_name': 'llm_as_judge-<uuid>', 'score': 0.8, 'passed': True, 'reasoning': 'The summary needs to include specific details from the section. Part of the essential information is captured. Key pieces of information are missing. Not all relevant structural information is included.'}]
Running this script we can see that most of our graders are passing except the chemical_name_grader. Next we'll programmatically recognize this opportunity to improve the summarization agent.
Note: When you run it locally, graders other than chemical_name_grader may fail at first. This is normal, as graders can initially fail, but the results should improve through the feedback loop. Early failures simply reflect the model adjusting its responses before converging on more accurate results.
Dashboard Observability
Eval runs and results can also be seen in the OpenAI Dashboard:
Figure 8 - Eval dashboard showing evaluation runs and results.
We can also drill down into a specific eval run:
Figure 9 - Detailed eval run results showing grader scores and performance metrics.
Agent Setup
Now that we have our evals and graders set up, we can go back to our summarization agent.
For simplicity, we will provide the code for a simple agent below. You could also use AgentBuilder, as shown in Figure 2, and export the code from the UI.
We will also need a metaprompt optimization agent, to optimize our prompt, as well as some simple utilities to handle prompt versions:
PromptVersionEntry: A pydantic model used to track the prompt and metadata as it changes in productionVersionedPrompt: A utility class to track prompt versions, this will be important in production when analyzing the evolution of the prompt as well as ensuring there is a fallback history in case of a regression
Next we'll create the starting summarization and prompt optimization agents.
Note: We created a wrapper to track prompt changes in the summarization agent since it is expected to evolve in production, the metaprompt agent's prompt will stay static for the purposes of this cookbook.
Orchestration and Monitoring
This is what we've done so far - we've created:
- Evals with 4 graders that will assess the outputs and produce a score for each grader
- A summarization agent with a versioned prompt class to track changes to the prompt and model
- A metaprompt optimization agent that will attempt to update the prompt based on a set of reasoning
Now these different functionalities can be composed to orchestrate the self-evolving loop with Agent tracing in the OpenAI dashboard.
Keep in mind that this is a simplified example. In a real-world scenario, you'd want to ensure you have guardrails for optimization attempts and that an alert notifies a human when a guardrail is triggered.
Note: Due to practical limitations of the cookbook we are simulating a stream of data by feeding in a static dataset and using print statements in place of true observability.
Orchestration Utilities
As in previous sections we'll create some utilities to manage the orchestration logic of the feedback loop.
Self-evolving loop
Now to simulate a stream of requests for summarization we'll feed in a prepared dataset and observe the optimization evolve from a naive prompt.
The referenced dataset.csv can be found in the Github repository.
How the final prompt is chosen
- Every evaluation logs the average grader score, the total score across graders, and whether the attempt passed the lenient criteria.
best_candidatetracks the most recent lenient pass (for transparency), but the final selection uses the aggregate totals to ensure we keep the top-performing prompt overall.- When the loop ends,
apply_best_candidate_if_neededrestores the prompt with the highest cumulative grader score (ties favor the latest version), guaranteeing that the surfaced prompt is the strongest performer observed.
Here is an example (abridged) output for the code above.
Inspecting the output shows that the self evolving prompt worked. There are a few takeaways to account for:
- The optimization is not always successful, so being able to roll back the prompt version is important
- The fidelity of the information from the graders is crucially important to ensuring a quality optimization
Agent Logs & Tracing
We can view optimization workflow runs in the dashboard under logs:
Figure 10 - Agent log traces showing optimization workflow runs in the dashboard.
And drill down into the different agent calls:
Figure 11 - Detailed agent trace showing individual agent calls and execution flow.
Continuous Monitoring
Once the evaluation loop is complete, the system should continue to monitor new incoming data and periodically re-evaluate model performance on blind datasets. This ensures the model remains accurate and compliant as the data distribution evolves.
To enable continuous monitoring, you can integrate a cron job or a lightweight scheduler loop that periodically checks for updates in your data source (e.g., new PDF uploads or database entries). When new data is detected, the system automatically triggers the evaluation and optimization loop described earlier.
For example (pseudo code):
This approach allows the model to continuously learn and adapt, improving over time as it processes fresh data - a key requirement for maintaining high-quality, real-world performance.
4. Going Further
a. Model Evaluation
We now have a fully automated loop improving our prompt with evals and accepting the new prompt when the rating is over the defined threshold.
In production, you could use a similar framework to monitor the performance of your agents as new user requests come in.
As mentioned above, this is a simplified example, and in a real-world scenario you'd want to have additional guardrails and a human-in-the-loop approach to approve new prompts.
Taking this concept further, we can also use evals to test different model parameter candidates such as the model version, verbosity, and reasoning. To see the full available set of parameters that could considered, check the ModelSettings class in the Agents SDK
The compare_model_candidates function is an example of how to:
- Optimize the prompt
- Generate candidate outputs from the optimized prompt using two or more different models
- Use evals to grade the candidate outputs and select the best candidate
It can be worked into the self_evolving_loop function with minimal refactoring.
NOTE: Production testing of model versions should be limited to versions within the same family version (e.g. gpt-5, gpt-5-mini, gpt-5-nano). It is recommended to conduct cross family version selection pre-production deployment.
And the final self_evolving_loop with model comparison code:
Here we can see a very similar output with additional information on the model version scores:
b. Prompt Optimization with Genetic-Pareto (GEPA)
We've demonstrated that the self-evolving loop works and that a prompt can be improved autonomously using Evals. However, we relied on a relatively straightforward, static metaprompt to improve our system prompt. In this section, we explore a more dynamic and reflexive method by using Genetic-Pareto (GEPA) [1] - a framework that samples agent trajectories, reflects on them in natural language, proposes prompt revisions, and evolves the system through iterative feedback loops.
The GEPA method, described in the paper available here, offers an compelling blueprint for continuous, self-improving prompt optimization. The code below draws generously on the GEPA Github repository available here.
We’ll reuse our graders and helper functions by adding a small adapter so that our setup works with GEPA. GEPA’s GEPAAdapter makes it easy to plug into our eval framework. We defined three hooks
evaluate: runs the summarization and grades with graders defined in the previous section (i.e., chemical_name_grader, word_length_deviation_grader, cosine_similarity, llm_as_judge).get_components_to_update: gets the text fields GEPA should evolve (here, system_prompt).make_reflective_dataset: packages inputs, outputs, and feedback for reflection.
Now that the adapter is ready, we can run GEPA using the same starting prompt ("You are a summarization assistant. Given a section of text, produce a summary.") and model (here, gpt-5) as in the earlier self-evolving loop for comparison. We provide our adapter instance, seed candidate, and training/validation sets to gepa.optimize(...). During the optimization, GEPA repeatedly invokes the adapter to score candidates, reflects on feedback, and ultimately produces the best evolved prompt.
Note: GEPA might take ~10-15 minutes to complete.
Here is an example (abridged) output for the code above:
In this cookbook, we explored three distinct approaches to prompt optimization:
OpenAI Platform Optimizer: using the Optimize button with a dataset containing manually entered human feedback (thumbs up/down and textual comments), we quickly produced a strong prompt with minimal configuration. This method excels at rapid iteration, but does not provide the automation needed for production environments.
Optimization using a static metaprompt: Our loop, incorporating four different graders,enabled automated exploration and iterative self-improvement without manual intervention. However, its exploration space was limited by a single static meta-prompt, and evaluation was performed section by section. Consequently, this approach risked overfitting to immediate grader feedback instead of achieving broader generalization.
GEPA optimization: Offering a more structured search process, reflective updates were informed by both quantitative scores and textual feedback, while candidates were trained on one dataset and validated on another. This method produced a more robust, generalized prompt and provided clearer empirical evidence of its performance.
Note: Examples of prompts generated by each method are available in the Appendix.
Depending on your use case, you may prioritize speed (OpenAI optimizer), lightweight automation (static metaprompt), or systematic generalization (GEPA). In practice, combining these methods by starting with rapid iteration and progressing toward reflective optimization can deliver both agility and performance.
Happy coding!
Citations
[1] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning by Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, Omar Khattab - https://arxiv.org/abs/2507.19457
Appendix
Examples of output prompts:
- Initial prompt:
You are a summarization assistant. Given a section of text, produce a summary.
- OpenAI Platform Optimizer:
You are a summarization assistant.
Task: Summarize the provided text concisely and accurately.
Output requirements:
- Output only the summary. Do not add titles, labels (e.g.,
"Summary:"), prefaces, or commentary.
- Preserve the document's structure. If multiple sections/subsections appear, summarize each one.
- Use a numbered list for sections/subsections (use their numbers/titles when present).
- Under each, use short dash bullets for key points.
- If there is only a single short section, return a brief bullet list or 1-2 concise sentences.
- Split any inline lists into separate bullets.
- Use plain, simple language. Keep bullets tight (ideally one line each). Remove redundancy.
- Include important quantitative details (values, units, conditions) and constraints. Do not invent information.
- Keep formatting simple: plain text, "1." numbering and "-" bullets only. No tables or special markup.
- Retain exact technical terms/notation from the source (e.g., chemical names, isotopic labels).
- If a section is explicitly marked "Not applicable," include that status; otherwise do not add it.
- Static metaprompt:
You are a technical summarization assistant for scientific and regulatory documentation. Your task is to generate a concise, comprehensive, and fully detailed summary of any scientific, technical, or regulatory text provided. Strictly adhere to the following instructions:
---
**1. Complete and Exact Information Inclusion**
- Capture *every* explicit fact, technical value, specification, quantity, measurement, regulatory reference, entity, process, site, and contextual detail verbatim from the source text.
- Do not omit or generalize any explicit information, no matter how minor.
**2. Precise Terminology and Named Entity Retention**
- Reproduce all names of chemicals, drugs, mixtures, buffer components, devices, companies, institutions, regulatory standards, section numbers, and procedural labels *exactly as stated*.
- Report all quantities, measurements, concentrations, ratios, masses, volumes, compositions, pH values, and units precisely as given.
- Do not paraphrase, rename, substitute, or simplify any term or value.
**3. All Procedural Details and Justifications**
- Explicitly include all described procedures, technical processes (e.g., terminal sterilization, aseptic processing), operational constraints, process justifications, compliance requirements, and standards references.
- Clearly state all reasons provided for choosing or omitting particular methods or processes.
**4. Regulatory and Compliance References**
- Accurately cite all regulations, standards (e.g., USP <797>), compliance statements, section numbers, and cross-references as in the original.
- Include all explicit mentions of compliance, applicability, and site location details.
**5. Explicit Statements of Absence, Limitations, and Applicability**
- Clearly state any declarations of absence, inapplicability (“Not applicable”), or limitations exactly as written in the source.
**6. Structural and Organizational Fidelity**
- Precisely reflect the original document’s section and subsection hierarchy, using clear section labels and indentation.
- Present all enumerations, lists, and tabulated data in structured bullet-point or numbered format, organized in accordance with the source document’s arrangement.
**7. No Paraphrasing, Summarizing, or Reinterpretation**
- Do *not* paraphrase, summarize contextually, reinterpret, or alter the meaning or sequence of any content.
- Remove only literal repetitions or redundant phrasing; otherwise, preserve all explicit statements, technical details, and contextual notes.
---
**Summary Output Objective:**
Produce a summary that delivers the full technical, factual, and regulatory content and structure of the original text, reformatted by eliminating only redundant language. The summary must enable audit, regulatory review, or peer reference without loss of any explicit information or terminology from the source.
---
*Apply these instructions rigorously to every provided document section to ensure scientific and regulatory accuracy and completeness.*
- GEPA optimizer:
You are a domain-aware summarization assistant for technical pharmaceutical texts. Given a “section” of text, produce a concise, single-paragraph summary that preserves key technical facts and exact nomenclature.
Length and format
- Write 1–3 sentences totaling about 45–70 words (target ~60; never exceed 90).
- Use one paragraph; no bullets, headings, tables, or heavy formatting.
Exact names and notation
- Include every chemical name that appears in the section at least once, using the exact original spelling, capitalization, punctuation, isotopic labels, brackets, hyphens, salts, buffer names, and parenthetical qualifiers. Treat distinct case/format variants as distinct names (e.g., [1-13C]pyruvic acid and [1-13C]Pyruvic acid are separate and each must appear once).
- Examples you must preserve verbatim when present: Hyperpolarized Pyruvate (13C) Injection; non-polarized Pyruvate Injection; Pyruvate (13C) Injection; hyperpolarized [1-13C]pyruvate; Mixture of [1-13C]pyruvic acid and 15 mM AH111501 sodium salt; TRIS/EDTA buffer solution; TRIS; NaOH; Na2EDTA; [1-13C]pyruvic acid; AH111501 sodium salt.
- Also preserve exact study identifiers, batch codes, section numbers, regulatory citations, and instrument parameters as written (e.g., GE-101-001, GE-101-003, USP <797>, 3.2.P.5.2.5, FFF106/140-806, FFF106/142-806, 3T MRI, 5 degree RF pulse, TR=3s, 90 degree pulse, 64 averages, TR=10s, 10 μl Gd/ml solution).
Content prioritization (if space is tight)
1) What the section is about (topic/purpose).
2) All named chemical entities and compositions (list all chemical names at least once; include concentrations/amounts if given).
3) Critical process/handling facts (e.g., aseptic processing vs terminal sterilization; ISO classifications; filtration specs; compounding/filling steps; temperatures/times/volumes; storage/administration limits).
4) Container/packaging specifics (e.g., cryovials, “sterile fluid path”).
5) Microbiological/testing/regulatory details (e.g., sterility/pyrogenicity testing timing; USP <797>; state board compliance; site/manufacturer if stated).
6) Overages/single-dose formulas and key quantities.
Numerical fidelity
- Preserve all critical numbers and units exactly (e.g., 1.44 g, 27.7 mg, 15 mM, 18 mL, 1.47 g, two 0.2 μm filters, ISO 7, ISO 5, 38 mL).
- Include testing/analysis parameters when present (e.g., polarization/relaxation time (T1); number of spectra; pulse angles; TR values; MRI location relative to clean room).
Style and compression
- Be neutral and factual; do not infer unstated information.
- Consolidate repeated statements; compress lists with commas/semicolons to save words.
- Mention tables/figures only to convey key data; do not reproduce them.
- If many chemicals are present, ensure each distinct name appears once; group them succinctly.
- Avoid symbols or special formatting not in the source text.
Common domain cues to include when present
- Aseptic processing vs terminal sterilization and the rationale/timing (e.g., “tested for sterility and pyrogenicity subsequent to patient administration”).
- Environmental/processing controls (ISO 7/ISO 5; LAF unit; filtration; filling/weight targets per cryovial).
- Site/regulatory context (e.g., USP <797>; California State Board of Pharmacy; University of California, San Francisco Department of Clinical Pharmacy).
- Study/kit equivalence statements (e.g., equivalence to GE-101-001/GE-101-003 formulations).
- QC/measurement methods (e.g., capacitive threshold at Administration syringe nominal 38 mL).
Self-check before finalizing
- Does the paragraph contain every distinct chemical name exactly as written in the section (including case and notation variants)?
- Is the summary 45–70 words (≤90), in a single paragraph?
- Are the most critical process/regulatory/testing details and all key numbers preserved without unnecessary verbosity?`
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.