Data Warehouses

Read training and prediction data from Snowflake, Databricks, BigQuery, and Microsoft Fabric, and write forecasts back.

Use the data-warehouse sources when your data already lives in Snowflake, Databricks, BigQuery, or Microsoft Fabric. rebase.sources gives every warehouse the same read/write surface and a leakage-safe bitemporal mapping, so a Predictor can train on warehouse data and be backtested honestly, then write forecasts back.

The connector runs inside your deployed function, step, or model — the same place your fit/predict code runs. Rebase never sees your warehouse credentials or data: it only injects the credentials you configure as environment variables (via secrets=), and your code queries the warehouse directly.

Install

Each backend has its own optional extra so your image only carries the driver it uses:

uv pip install "rebase-toolkit[snowflake]"
uv pip install "rebase-toolkit[databricks]"
uv pip install "rebase-toolkit[bigquery]"
uv pip install "rebase-toolkit[fabric]"

Install all four with rebase-toolkit[sources]. Declare the extra on the image you deploy:

import rebase as rb

image = rb.Image.python("3.12").uv_pip_install("rebase-toolkit[snowflake]")

Credentials

Credentials resolve in this order, highest priority first:

  1. explicit keyword arguments to the factory,
  2. REBASE_SOURCE_<CONNECTION>_<FIELD> environment variables when you pass connection="<name>",
  3. provider-standard environment variables (e.g. SNOWFLAKE_ACCOUNT).

Supply credentials as a secret: a named bundle of environment variables, created once and attached to any function, model, or asgi_app. Create the bundle with the CLI — values go straight to Secret Manager and are never stored by Rebase:

$ rebase secret create acme-snowflake \
    SNOWFLAKE_ACCOUNT=xy12345.eu-central-1 \
    SNOWFLAKE_USER=REBASE_SVC \
    SNOWFLAKE_PRIVATE_KEY=-  < rsa_key.p8
Created secret acme-snowflake with keys: SNOWFLAKE_ACCOUNT, SNOWFLAKE_PRIVATE_KEY, SNOWFLAKE_USER
Workspace:   <workspace-id>
Use it with: secrets=[rebase.Secret.from_name("acme-snowflake")]

KEY=- reads that value from stdin; --from-dotenv .env.acme imports a dotenv file; --force overwrites an existing bundle. Inspect with rebase secret list (names and key names only — never values) and remove with rebase secret delete.

Attach the bundle in code — every key becomes an environment variable at run time:

class DemandForecast(rb.Predictor):
    name = "demand-forecast"
    image = rb.Image.python("3.12").uv_pip_install("rebase-toolkit[snowflake]")
    secrets = [rb.Secret.from_name("acme-snowflake")]

    def fit(self, train) -> None: ...
    def predict(self, obs) -> dict: ...

secrets= is accepted on functions, models (including Predictor), asgi_apps, and workflows, and takes a list of secrets — attach several bundles and their keys merge (later bundles win). For workflows, pass it to @project.workflow(...) or the rebase.Workflow(...) constructor; a scheduled workflow resolves its secret references when its schedule is synced, so cron-triggered runs carry the credentials too. (The module-level @rb.workflow(...) decorator does not accept secrets= — use the project-scoped form.) Besides from_name, you can build secrets from local values at deploy time:

rb.Secret.from_dict({"SNOWFLAKE_ACCOUNT": "xy12345"}, name="acme-snowflake")
rb.Secret.from_dotenv(".env.acme", name="acme-snowflake")

Both create (or update) the workspace bundle when you deploy, then attach it — handy in CI. Prefer from_name + rebase secret create for credentials shared across a team.

For non-sensitive config there is also env={...} with literal values, but env= values are stored in plaintext on the version record — never put real credentials there.

Secrets are workspace-scoped: each bundle's keys are namespaced to your workspace in Secret Manager, so one workspace cannot read another's. Values are injected by Cloud Run at container start and never pass through the deploy payload or source snapshot.

Read

Every source exposes read(query, *, params=None) returning a pandas DataFrame:

src = rb.sources.bigquery(connection="acme")
df = src.read("SELECT ts, load_mw FROM `acme.energy.demand` WHERE site = @site",
              params={"site": "site-001"})

Leakage-safe reads with read_bitemporal

The number-one failure mode of an energy-forecasting backtest is leakage — a model reading data that was not knowable at forecast time. A raw warehouse table usually has only a target timestamp, so you declare how to derive the knowledge time (when each row became available) with a BitemporalSpec:

from datetime import timedelta

from rebase.sources import BitemporalSpec

# Best: the table records when each row became knowable.
spec = BitemporalSpec(valid_time="ts", knowledge_time="issued_at")

# Or: availability lags the target time by a fixed, known delay.
spec = BitemporalSpec(valid_time="ts", knowledge_delay=timedelta(hours=1))

df = src.read_bitemporal(
    "SELECT ts, issued_at, load_mw FROM `acme.energy.demand` WHERE site = @site",
    spec,
    params={"site": "site-001"},
)

src here is the BigQuery source from the previous example. The parameter style follows the warehouse's driver: BigQuery takes a dict with @name markers, Snowflake takes a list with %s, Databricks takes a dict with %(name)s. Each warehouse page shows its form.

read_bitemporal returns a frame with canonical valid_time and knowledge_time columns, ready to feed to emflow for a backtest where leakage is structurally impossible.

If you set neither knowledge_time nor knowledge_delay, knowledge_time falls back to ingestion time and the connector warns you. Only rely on that for data that is genuinely known at its valid_time; otherwise your backtest scores will not hold up in production.

Write forecasts back

Every source also exposes write(df, table, *, mode="append") (mode="replace" truncates first), so a deployed model can push forecasts back to the warehouse:

src = rb.sources.snowflake(connection="acme", schema="ANALYTICS")
src.write(forecast_df, "FORECASTS", mode="append")

For Snowflake, table is a single identifier: set database and schema on the source rather than writing a dotted name.

BigQuery and Snowflake use native bulk load paths. Fabric writes use batched INSERT statements suited to modest result sets such as forecasts. Databricks writes run one parameterized INSERT per row and currently fail with the pinned databricks-sql-connector (see Databricks). For large bulk loads, stage Parquet and use COPY INTO (Databricks) or the native bulk loader (Fabric).

The energy data model (BigQuery)

For energy time series, the BigQuery source goes beyond generic SQL: it stores data in the canonical layout of rebase's open data model (energydatamodel / timedatamodel) — the same schema rebase uses internally — so warehouse data is structurally interchangeable with platform data.

The layout is an append-only series_values table with three time axes plus a series catalog:

ColumnMeaning
valid_timethe timestamp the observation is about (always present)
knowledge_timewhen the value became knowable / was issued — the forecast-issue axis; horizon = valid_time − knowledge_time
change_timewhen the row was written or corrected — corrections are new rows, never updates

A series is keyed the energydb way — (path, data_type, name), e.g. ("portfolio/site-1/t01", "forecast", "electricity.supply") — with unit, FLAT/OVERLAPPING type, retention (short, medium, long or forever), and an optional description in the catalog. series_id is derived deterministically from the key, so registration is idempotent.

src = rb.sources.bigquery(connection="acme")
src.ensure_energy_schema("energy")   # creates series_values + series (idempotent)
src.register_series("energy", path="portfolio/site-1/t01", data_type="forecast",
                    name="electricity.supply", unit="MW", timeseries_type="OVERLAPPING")

# write a forecast issue: VERSIONED shape = knowledge_time + valid_time + value
src.write_series("energy", forecast_frame, path="portfolio/site-1/t01",
                 data_type="forecast", name="electricity.supply")

Reads replicate timedb's point-in-time semantics — the properties that make backtests honest:

key = ("portfolio/site-1/t01", "forecast", "electricity.supply")

src.read_series("energy", key)                          # latest: newest issue, newest correction
src.read_series("energy", key, as_of=cutoff)            # only what was knowable at `cutoff`
src.read_series("energy", key, overlapping=True)        # every forecast issue kept
src.read_series("energy", key, include_updates=True)    # full audit trail (change_time, changed_by)

Frames follow the timedatamodel conventions: UTC microsecond timestamps and float64 values. write_series accepts a TimeSeries or a frame indexed by valid_time or (knowledge_time, valid_time), or with those as columns. read_series returns a flat frame with path, data_type, name, valid_time and value columns, plus knowledge_time when overlapping=True and change_time, changed_by and annotation with include_updates=True. Only SIMPLE and VERSIONED shapes are writable; AUDIT and CORRECTED come back from reads, matching TimeSeries.validate_for_insert.

The physical layout mirrors rebase's ClickHouse store: partitioned by valid_time month, clustered by (series_id, valid_time, knowledge_time, change_time). The read SQL uses standard QUALIFY, so the same builder will back Snowflake and Databricks next.

Per-warehouse settings

WarehouseFactoryAuth (recommended)Key settings
Snowflakerb.sources.snowflakeKey-pair (private_key/private_key_path) or passwordaccount, user, warehouse, database, schema, role
Databricksrb.sources.databricksOAuth M2M (client_id+client_secret) or access_tokenserver_hostname, http_path, catalog, schema
BigQueryrb.sources.bigqueryApplication Default Credentials / service accountproject, location, credentials_path
Microsoft Fabricrb.sources.fabricAzure AD service principalserver, database, client_id, client_secret, tenant_id

Each warehouse page covers the full setup: driver install, auth walkthrough, dialect-correct parameter binding, and write-path behavior.

Any setting can be passed to the factory directly, or supplied through the environment variables described in Credentials.

Use in a model

A production Predictor reads its training view from a source and stays fully backtestable:

import rebase as rb
from rebase.sources import BitemporalSpec


class DemandForecast(rb.Predictor):
    name = "demand-forecast"

    def fit(self, train) -> None:
        ...

    def predict(self, obs) -> dict:
        ...

Point the model's training loader at rb.sources.<warehouse>(connection=...).read_bitemporal(...), and the class you backtest is the class you deploy.

A deployed Predictor receives warehouse credentials through secrets= (see Credentials). Locally, fit/predict and emflow backtests read from your own environment, so you can iterate without deploying.

On this page