Verifiable Targets
What autonomous agents can search - problems with programmatic, trusted scoring.
Autonomous agents can only search for models when the goal is verifiable: a target is any problem whose candidate solutions can be scored programmatically, by evaluation machinery the agents cannot influence. If a number can be computed that says "this model is better than that one" — and the agents can't game how it's computed — coding agents can hillclimb it.
Two kinds of verifiable targets work with hillclimb searches today.
emflow Problems
Problems from emflow's registry, addressed as emflow://<name>:
rebase hillclimb start emflow://gefcom2014:solar --budget 2hemflow problems are the strongest form of verifiable target because leakage protection is structural, not procedural:
| Property | How |
|---|---|
| Point-in-time data | Every observation is served through a bitemporal DataFeed — a model physically cannot read data that wasn't knowable at the forecast origin. |
| Validation / holdout splits | Agents iterate against the validation split; the shipped model is selected on a holdout split the agents never see. |
| Objective scoring | The problem declares its objective (pinball loss, MAE, ...); the evaluator computes scores from returned predictions — nothing a candidate prints is trusted. |
| Reference baselines | Benchmarks ship a reference model that becomes the search's scored floor, and official leaderboards rank the result. |
Candidates are emflow Predictor classes exposing get_model(); the evaluator fits each candidate on the official training view and scores it per forecast origin. Sandboxed evaluations run offline and credential-free, so generated code cannot reach holdout data or secrets.
Ingested benchmarks include GEFCom2014 (load/price/wind/solar), GEFCom2017, and HEFTCom2024, plus tutorial problems. Any problem added to emflow's registry is immediately a valid search target.
Problem Folders
A self-contained directory whose problem is its verifier. No emflow is required:
problems/my-problem/
├── problem.yaml # problem_id, metric, higher_is_better, time_budget_s
├── description.md # what agents read
├── verifier.sh # executable: runs solution.py, writes the score to $HILLCLIMB_RESULT
└── data/ # optional runtime inputsCandidates are solution.py files. The engine runs only verifier.sh: exit code 0 means the candidate is valid, and the score is whatever it writes to $HILLCLIMB_RESULT ({"score": <float>} or a bare number). Anything the solution prints is ignored. Set holdout: true in problem.yaml and the engine also runs verifier.sh --holdout for selection, so selection doesn't overfit the validation signal.
Hosted searches take emflow registry problems. Problem folders run with rebase hillclimb start --local.
This form suits competition-style tasks (Kaggle datasets, optimization puzzles) and anything where "run the verifier" fully defines success.
What Makes a Good Target
- The metric must mean what you want. Agents optimize exactly what is scored — a proxy metric produces models that excel at the proxy.
- Held-out evaluation is non-negotiable. With dozens of candidates, selecting on the same signal agents iterate against overfits it; hillclimb selects on a blend of validation and hidden-holdout ranks so a lucky validation score gets vetoed.
- Evaluation cost bounds search depth. Every candidate pays a full evaluation; a target that takes an hour to score yields few candidates per budget.
- Selection pressure is recorded. Finished searches log how many candidates were tried (
n_trials) alongside the final score, keeping performance claims honest.