Workflow Execution

Rebase Workflows are pipeline-style compositions of steps. They use Prefect's DAG and task execution model under the hood.

Workflows default to mode="interactive", isolation="shared", which executes runs on a warm Cloud Run service worker backed by a Cloud Run-hosted Prefect API. Use mode="job" for a fresh Cloud Run Job per workflow run. Dedicated interactive workflows are not supported.

For the full overview across functions, models, and workflows, see Execution Modes.

Execution modes

Set the mode on the workflow decorator:

import rebase as rb

project = rb.project("energy-forecasting")

@project.step()
def load_a(a: int = 0) -> dict:
    return {"value": a}

@project.step()
def load_b(b: int = 0) -> dict:
    return {"value": b}

@project.step()
def add_values(left: dict, right: dict) -> dict:
    return {"sum": left["value"] + right["value"]}

@project.workflow(name="add-workflow", mode="job")
def add_workflow(a: int = 0, b: int = 0) -> dict:
    left = load_a(a)
    right = load_b(b)
    return add_values(left, right)

project.deploy()
print(add_workflow.remote(a=40, b=2))

@project.step(...) registers step bodies as internal workflow graph nodes. Step decorators do not take mode or isolation: steps execute in-flow inside the workflow run, so the workflow's mode covers every step.

Deprecated run_type="quick" maps to interactive/shared and run_type="long" maps to job. The GKE-Prefect and Prefect Cloud paths no longer exist; see Execution Modes.

Infrastructure mapping

Both modes are orchestrated by Rebase's self-hosted Prefect API, which itself runs on Cloud Run. Run records report the resolved infrastructure in the internal execution_backend field.

Scheduled runs use the workflow's configured mode, just like manual and triggered runs.

Workflow runs in both modes are cancellable: rebase run cancel sets the Prefect flow run to cancelling and the worker stops the flow. rebase run logs reads Prefect's log API for both modes.

Platform Configuration

The mapping is configured by the platform, not by user code. The API service needs:

There are two Prefect work pools: the Cloud Run service pool runs mode=interactive workloads on the warm worker, and the Cloud Run jobs pool (default name rebase-workflows-cloud-run-jobs) runs mode=job workloads as one Cloud Run Job per run. Each keeps whatever image it was last registered with, and both execute the same runner code — so refreshing only one leaves half the workloads on stale code while the deploy reports success.

Deploy with scripts/deploy.sh dev or scripts/deploy.sh prod. It loads the target's deploy/<target>.env and runs scripts/deploy_public_cloud_run_api.sh, which deploys the API and refreshes both pools. Pass SKIP_JOBS_POOL=1 to leave the jobs pool alone; the script then prints which pool it did not touch and the command that does.

Refreshing the jobs pool is still not enough for scheduled mode=job workflows: Prefect pins the runner image into each scheduled workflow's deployment when the workflow is synced, so those keep the image of their last sync. The deploy therefore ends by re-syncing every scheduled workflow through the API's own image (the resync-schedules Cloud Run job), which pins the new image. Pass SKIP_SCHEDULE_RESYNC=1 to skip it; the summary then prints the command to run by hand. The image tag identifies both repositories the image is built from — <toolkit_sha>-sdk<sdk_sha>, with a -dirty suffix when the vendored SDK checkout has uncommitted edits — so a deployed revision always says which SDK it carries.

The jobs-pool script creates the Cloud Run Prefect API service, Cloud Run Jobs worker service, work pool, and runner deployment:

PROJECT_ID=rebase-agents \
REGION=europe-west3 \
IMAGE=europe-west3-docker.pkg.dev/rebase-agents/rebase-workflows/workflow-mvp:TAG \
WORKFLOWS_DATABASE_URL=postgresql+asyncpg://... \
PUBLIC_API_URL=https://... \
scripts/deploy_prefect_cloud_run_jobs.sh

The Prefect server's own database connection is not passed to any deploy script. It lives in the prefect-database-url Secret Manager secret (created by toolkit/terraform) and is mounted into rebase-prefect-api with --set-secrets, so the credential never appears in a shell, a process list, or the service's plaintext environment.

PUBLIC_API_URL is required: it is written into the pool's job template, and the runner mints a run-scoped API key only when the template carries it. Registering a pool without it lets every flow start and then fail on user code's first self-call with missing bearer token. The API-deploy script resolves it automatically from the deployed service; the jobs script refuses to run without it.

Workspace compute limits (workspace_compute_policies) are read with rebase workspace compute-policy show and changed with rebase workspace compute-policy set. Reading is open to any workspace member; changing requires a superadmin, because these are vendor-side ceilings rather than a tenant's own setting. scripts/set_workspace_compute_policy.py remains the break-glass path — it needs only a database URL, so it still works when the API is unavailable.

For administering every workspace at once, use rebase admin: a TUI listing each workspace with its members, quota and this month's spend, with the quota and the monthly credit grant editable in place. It is gated by a profile-level superadmin check — the caller's session email must be in the API's SUPERADMIN_EMAILS and under the company domain — so it needs a session credential, not an API key. That gate is also what makes it work on workspaces the superadmin is a member of: the principal-level rebase workspace compute-policy set refuses those by design, because a real membership row always wins over the superadmin fallback.

SUPERADMIN_EMAILS is a list[str] setting and is parsed as JSON. Set it as SUPERADMIN_EMAILS='["name@rebase.energy"]'; a bare address fails Settings() at import and the API container will not start.

The two timeout ceilings mirror Cloud Run's own. max_run_timeout_seconds bounds what a service answers in one request — ASGI apps, quick functions, interactive workflows — and goes up to Cloud Run's 3600 s request maximum. max_job_timeout_seconds bounds a mode="job" workflow's own Cloud Run Job task and goes up to 86400 (24 h). Both are ceilings, not defaults: a job workflow opts in to a longer run with its own timeout_seconds (or by declaring one on each of its steps), an ASGI app with its timeout_seconds. A longer timeout costs proportionally more credits, because a run is reserved and charged on elapsed runtime.

Sizing a job workflow

A mode="job" workflow gets its own Cloud Run Job, so it can declare what that container gets. Without this it takes the backend default, which is deliberately small:

@project.workflow(name="sync", mode="job", memory="2Gi", cpu="1")
def sync() -> dict:
    ...

memory takes MiB as a number (2048) or a Cloud Run string ("2Gi"); cpu takes cores (2) or milli-vCPU ("2000m"). Both require mode="job" — an interactive workflow runs inside the shared Prefect worker and has no container of its own to size, so a limit there is refused rather than silently ignored. Not to be confused with resources=, which annotates the workflow's steps and does not size the workflow's own container.

Both are bounded by the workspace's max_cloud_run_cpu_milli / max_cloud_run_memory_mib, and asking for more fails the deploy rather than the run:

409 cloud_run_memory cannot exceed 1024 MiB for this workspace

A superadmin raises that ceiling with --max-memory-mib / --max-cpu-milli on compute-policy set. Cloud Run also couples the two — above 4 GiB needs at least 2 vCPU, above 8 GiB at least 4 — which is checked at deploy time rather than by Cloud Run at launch:

409 cloud_run_memory of 8192 MiB requires cloud_run_cpu of at least 4 vCPU on Cloud Run

Memory is not free: credits are charged per GiB-second of what the container reserves, for the whole run, whether or not the workflow uses it.

Bounding a run

Every workflow run has a hard bound, timeout_seconds, and the platform enforces it at three layers: inside the container (the run fails with a failure_reason of run_timeout or step_timeout naming the step and the number), at the Cloud Run task timeout a minute behind it (the SIGKILL, for a step that cannot be interrupted), and in the reaper behind that. A Cloud Run task is never retried: a run that hits its bound is over.

You rarely set the workflow bound yourself. Declare a timeout_seconds on each step — the bound for one attempt of that step — and the workflow's is derived at deploy: the sum of every step's timeout_seconds × (retries + 1), plus a minute of startup.

@project.step(timeout_seconds=120, retries=1)
def fetch() -> dict: ...

@project.step(timeout_seconds=600)
def land(payload: dict) -> dict: ...

@project.workflow(mode="job", schedule=rb.Cron("2,17,32,47 * * * *"))
def collect() -> dict:
    return land(fetch())
# effective timeout_seconds: 120 × 2 + 600 + 60 = 900

Set timeout_seconds on the workflow to override the sum in either direction; a step longer than its workflow is refused at deploy, naming both numbers. If any step declares no timeout there is nothing to sum, the run falls back to the platform default, and deploy() warns which step is why. rebase workflow get shows timeout_seconds (what you declared), effective_timeout_seconds (what runs) and timeout_source (explicit, derived or default).

A step that times out is not retried, since its abandoned attempt cannot be reclaimed; a step that raises is re-run up to retries more times. The derived bound budgets for every attempt, so a workflow with retries reserves a little more than it usually needs.

The bound is capped by the workspace's max_job_timeout_seconds for a mode="job" workflow and by max_run_timeout_seconds for an interactive one (see above); asking for more fails the deploy, not the run. A superadmin raises either with rebase admin set <workspace> --max-job-timeout-seconds / --max-run-timeout-seconds.

Benchmark

Use the workflow benchmark when comparing the two workflow execution paths:

uv run python scripts/benchmark_workflow_backends.py \
  --backends prefect_cloud_run_jobs,prefect_cloud_run_service \
  --cases no_deps,boltons \
  --repetitions 2 \
  --output-json benchmark-results/workflows.json \
  --output-chart benchmark-results/workflows.png

The output separates registration time, backend provision time, provider submit time, result polling, SDK/API overhead, first run latency, and warm p50/p90/p95. For workflows, backend provision is normally zero per workflow because the Prefect infrastructure is deployed ahead of time; the queue, worker, and Cloud Run job startup costs appear in the run latency categories.