Backtesting

Backtesting on Rebase is powered by emflow, the open-source sequential-decision framework the toolkit builds on (rebase-toolkit[modeling]). A backtest runs a model through the sequential decision loop on historical data. This page explains what happens inside — the machinery behind verifiable targets and hillclimb evaluations, and the same loop you can run yourself.

The Core Idea: Point-in-Time Data

The number-one failure mode of energy-forecasting backtests is leakage — a model accidentally reading data that wasn't knowable at forecast time, producing scores that evaporate in production. emflow makes leakage structurally impossible rather than procedurally avoided:

• Every dataset field is bitemporal: rows carry both their target time and the knowledge time at which they became available (actuals arrive with delays; forecasts have issue times).
• Models never touch raw data. They receive a view frozen at a moment: train (everything knowable at the training cutoff) and, per forecast origin, an Observation with obs.history(...), obs.forecasts(...), and the obs.target_index to predict.
• A model that tries to peek simply finds the future absent — there is nothing to read.

The Evaluation Loop

emflow problems are gym environments (ForecastEnv): the action is the forecast. An Experiment drives the loop:

import emflow as ef

problem = ef.load_problem("gefcom2014:solar")
result = ef.evaluate(problem, MyPredictor(), split="validation")
print(result.score, result.n_scored)

Under the hood: the environment resets, the model is fit() on the training view, and for every forecast origin in the split the model predict(obs)s; rewards settle only when the targets become knowable, and the problem's objective (pinball loss, MAE, ...) is computed by the environment from the returned predictions — nothing the model prints is trusted.

Models subclass emflow.Predictor (fit/predict), or FeaturePredictor for a declarative-feature fast path that evaluates all origins vectorized — orders of magnitude fewer model calls on large problems.

Splits and Honest Scores

Problems declare official splits in target time: a training period, a validation period to iterate against, and a holdout period for final selection. Iterating against the holdout invalidates it — this is why hillclimb lets agents see only validation scores and selects on holdout ranks computed by the orchestrator.

For untrusted submissions there is a Verifier: it takes a fresh, untrained model, fits it itself on the official training view, scores the holdout, and records results (with metadata like the number of candidates tried) on a leaderboard. Benchmark problems ship reference baselines and published competition leaderboards, so a score comes with a rank.

Where You Meet It