Verifiable Targets
What autonomous agents can search - problems with programmatic, trusted scoring.
Autonomous agents can only search for models when the goal is verifiable: a target is any problem whose candidate solutions can be scored programmatically, by evaluation machinery the agents cannot influence. If a number can be computed that says "this model is better than that one" — and the agents can't game how it's computed — coding agents can hillclimb it.
Two kinds of verifiable targets work with hillclimb searches today.
emflow Problems
Problems from emflow's registry, addressed as emflow://<name>:
rebase hillclimb start emflow://gefcom2014:solar --budget 2hemflow problems are the strongest form of verifiable target because leakage protection is structural, not procedural:
| Property | How |
|---|---|
| Point-in-time data | Every observation is served through a bitemporal DataFeed — a model physically cannot read data that wasn't knowable at the forecast origin. |
| Validation / holdout splits | Agents iterate against the validation split; the shipped model is selected on a holdout split the agents never see. |
| Objective scoring | The problem declares its objective (pinball loss, MAE, ...); the evaluator computes scores from returned predictions — nothing a candidate prints is trusted. |
| Reference baselines | Benchmarks ship a reference model that becomes the search's scored floor, and official leaderboards rank the result. |
Candidates are emflow Predictor classes exposing get_model(); the evaluator fits each candidate on the official training view and scores it per forecast origin. Sandboxed evaluations run offline and credential-free, so generated code cannot reach holdout data or secrets.
Ingested benchmarks include GEFCom2014 (load/price/wind/solar), GEFCom2017, and HEFTCom2024, plus tutorial problems. Any problem added to emflow's registry is immediately a valid search target.
Problem Folders
A self-contained directory with a verifier script — no emflow required:
problems/my-problem/
├── problem.yaml # metric, direction, budget
├── description.md # what agents read
├── verify.py # scores submission.csv, prints `val_score: <float>`
├── sample_submission.csv
└── data/ # optional runtime inputsCandidates are scripts that write submission.csv; the orchestrator runs verify.py and trusts only its final val_score: line. An optional hidden holdout is carved from the training data so selection doesn't overfit the validation signal.
This form suits competition-style tasks (Kaggle datasets, optimization puzzles) and anything where "run the verifier" fully defines success.
What Makes a Good Target
- The metric must mean what you want. Agents optimize exactly what is scored — a proxy metric produces models that excel at the proxy.
- Held-out evaluation is non-negotiable. With dozens of candidates, selecting on the same signal agents iterate against overfits it; hillclimb selects on a blend of validation and hidden-holdout ranks so a lucky validation score gets vetoed.
- Evaluation cost bounds search depth. Every candidate pays a full evaluation; a target that takes an hour to score yields few candidates per budget.
- Selection pressure is recorded. Finished searches log how many candidates were tried (
n_trials) alongside the final score, keeping performance claims honest.