Optimize configurable systems against measurable objectives
A skill that turns vague tuning requests into bounded, evidence-backed optimization experiments against a scalar objective.
Why it matters
Turn vague tuning requests into bounded optimization experiments with defined search spaces, trial budgets, baselines, and evidence-backed recommendations for hyperparameters, prompts, inference settings, or control policies.
Outcomes
What it gets done
Define optimization targets with tunable parameters, valid ranges, and baseline measurements
Run iterative trials within budget constraints while recording parameters, metrics, and failures
Compare best results against baselines and simple search strategies to validate improvements
Report recommended configurations with measured gains, tradeoffs, and validation requirements
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/ag-optim-agent | bash Overview
Optim Agent
A skill that turns vague tuning requests into bounded optimization experiments - defining a scalar objective, search space, budget, and baseline, then reporting an evidence-backed recommendation. Use it when the objective can be scored consistently and the user has approved consuming compute/API budget - not for one-shot implementation tasks with no real search space.
What it does
This skill optimizes configurable systems against a measurable scalar objective, turning a vague tuning request into a bounded experiment with a defined search space, budget, baseline, and evidence-backed recommendation. It applies to tuning hyperparameters, prompts, inference settings, simulation parameters, quantitative strategies, or RL/control policies - whenever the objective can be scored as a scalar score, loss, accuracy, cost, latency, reward, or risk-adjusted metric, and the user needs a small-budget loop with trial history, comparisons, and stop criteria.
Its nine-step process: define the optimization target as a single maximize-or-minimize scalar metric; list tunable parameters with valid ranges, types, defaults, and forbidden combinations; establish at least one baseline before any agent-guided trials; set the budget up front (trials, time, compute, money, or dataset subsample); run trials one at a time unless the user explicitly approves parallel execution; record every trial's parameters, metric value, notes, and failure status; compare the best result against both the baseline and a simple search strategy where possible; stop when the budget is exhausted, improvement plateaus, or the next trial can't be justified from evidence; and report the recommended configuration, measured gain, tradeoffs, and any validation still needed before production use.
Worked examples span hyperparameter tuning (learning rate, regularization, tree depth for a credit-default model, tracked by validation AUC against a default baseline), inference tuning (retrieval depth, temperature, and reranker threshold for a RAG workflow, optimized under a latency or cost ceiling), and simulation/control tuning (controller gains or environment parameters, optimizing reward or error while logging failed trials separately so they don't bias the recommendation).
When to use - and when NOT to
Use it when the objective can be scored consistently and the user has given permission to run experiments and consume compute or API budget. It explicitly steps back when the objective is purely subjective and can't be scored consistently, when that permission hasn't been given, or when the task is a one-shot implementation, debugging, or code review request with no actual configurable search space. Best practice keeps the first run small, prefers parameters with clear operational meaning over arbitrary knobs, treats failed trials as data worth recording rather than discarding, and validates the final configuration on held-out data or a fresh seed before calling it robust - it does not guarantee a global optimum, and results depend on objective quality, noise, search-space design, and reproducibility.
Inputs and outputs
Input is the scalar objective, the tunable parameter space with ranges and defaults, and a trial budget. Output is a trial history with recorded parameters and outcomes, a recommended configuration compared against baseline, and a report of measured gain, tradeoffs, and outstanding validation needs.
Who it's for
Engineers and researchers tuning a configurable system - ML hyperparameters, RAG inference settings, or simulation/control parameters - who need a disciplined, budget-bounded experiment loop rather than ad hoc trial-and-error, with domain review still required before applying tuned configurations to production, financial, safety-critical, or user-impacting systems.
Source README
Optim Agent
Overview
Use this skill to optimize configurable systems against a measurable scalar objective. It helps an agent turn vague tuning requests into bounded experiments with a defined search space, budget, baseline, and evidence-backed recommendation.
When to Use This Skill
- Use when tuning hyperparameters, prompts, inference settings, simulation parameters, quantitative strategies, or RL/control policies.
- Use when the objective can be measured as a scalar score, loss, accuracy, cost, latency, reward, or risk-adjusted metric.
- Use when the user needs a small-budget optimization loop with trial history, comparisons, and stop criteria.
Do not use this skill when
- The objective is purely subjective and cannot be scored consistently.
- The user has not provided permission to run experiments or consume compute/API budget.
- The task is a one-shot implementation, debugging, or code review request with no configurable search space.
Instructions
- Define the optimization target in one sentence: maximize or minimize one scalar metric.
- List the tunable parameters, valid ranges, types, defaults, and any forbidden combinations.
- Establish at least one baseline before proposing agent-guided trials.
- Set the budget up front: number of trials, time, compute, money, or dataset subsample.
- Run or request trials one at a time unless the user explicitly approves parallel execution.
- Record every trial with parameters, metric value, notes, and failure status.
- Compare the best result against the baseline and a simple search strategy when possible.
- Stop when the budget is exhausted, the improvement plateaus, or the next trial cannot be justified from evidence.
- Report the recommended configuration, measured gain, tradeoffs, and any validation still needed before production use.
Examples
Example 1: Hyperparameter optimization
Tune learning rate, regularization, and tree depth for a credit-default model. Track validation AUC for each trial, compare against the default configuration, and recommend the best setting only if it improves the baseline under the agreed trial budget.
Example 2: Inference tuning
Tune retrieval depth, temperature, and reranker threshold for a RAG workflow. Optimize answer quality under a latency or cost ceiling, then report the best configuration with quality, latency, and cost tradeoffs.
Example 3: Simulation or control
Tune controller gains or environment parameters for a simulator. Optimize reward or error while logging failed trials separately so unstable configurations do not bias the recommendation.
Best Practices
- Keep the first run small; expand only after the loop produces useful signal.
- Prefer parameters with clear operational meaning over arbitrary knobs.
- Treat failed trials as data and record why they failed.
- Validate the final configuration on held-out data, a fresh seed, or a separate scenario before calling it robust.
- Ask before running expensive, long, or externally billed experiments.
Limitations
- This skill does not guarantee a global optimum.
- Results depend on objective quality, noise, search-space design, and experiment reproducibility.
- Use domain review before applying tuned configurations to production, financial, safety-critical, or user-impacting systems.
Additional Resources
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.