Skip to main content

Training runs

A training run turns a succeeded dataset build into a checkpointed, resumable, fully tracked behavioral model. Runs execute as queued GPU jobs; you interact with them entirely through the /runs API.

Starting a run​

POST /runs with a dataset id and a model size:

{
  "dataset_id": "ds_01j9zx…",
  "model_config_tier": "S",
  "labels": ["lbl_01j9zy…"],
  "seed": 0
}
  • The dataset build must have succeeded with an empty violation report (leakage gates fail closed; failed_precondition otherwise).
  • model_config_tier picks a preset (S default, M, L); overrides fine-tunes it within validated bounds (invalid_argument when out of bounds or unknown).
  • labels selects which of the dataset's labels get prediction heads; omitted means all of them, [] trains self-supervised only.
  • trainer overrides the maximum epoch budget, batch token budgets, checkpoint/validation cadence, GPU count, validation-driven early stopping, and the fingerprinted learning-rate schedule.

Everything above is assembled into a content-addressed run config (schema https://spec.tessera.dev/schemas/run_config.json): identical requests reuse the in-flight run, and re-running an identical config after completion reproduces evaluation metrics within a tight tolerance — training is seeded and deterministic by construction.

Duration is a budget​

total_steps is the immutable maximum derived from the epoch cap and the dataset's train-token count. With the defaults (epochs: 10, early_stopping: true, patience: 5, min_delta: 0.0001, cooldown_ratio: 0.2, lr_schedule: wsd), validation may trigger a deterministic cooldown and finish successfully below that maximum. Set early_stopping: false and lr_schedule: cosine for fixed-horizon behavior. With early stopping enabled, omitting val_every_steps resolves a fingerprinted nominal cadence capped at 500 steps and validates at each actual packed epoch boundary; explicit cadence remains exact, and fixed-horizon mode retains the legacy default. Newly assembled short WSD runs pin a proportional cooldown minimum, while legacy configs keep their original 50-step minimum and resume curve. Early stopping requires WSD; cosine remains the explicit fixed-horizon schedule.

Monitoring​

  • GET /runs/{id} — status (queued → running → succeeded | failed | cancelled), pinned fingerprints, retained checkpoint URIs, the resolved trainer echo, best_val_loss, adaptive stop facts, and the created model_id once succeeded.
  • GET /runs/{id}/metrics — multi-series step time series ({series: {name: [[step, value], …]}, cursor}); poll incrementally with ?after_step=<cursor> for live charts. The additive epoch series supplies the terminal epoch without client-side reconstruction.
  • GET /runs/{id}/logs — legacy-compatible timestamp, severity, and message fields by default. Pass ?structured=true to include available step/epoch, validation, throughput, checkpoint, and early-stopping context.

Checkpoints and resume​

Runs checkpoint on a configurable cadence, on a new best validation result, and at the end: safetensors weights + optimizer state + data-position bookkeeping, each fingerprinted in a manifest. The last 3 checkpoints plus the best-by-validation-loss are retained. The registered model always uses that best checkpoint (the newest complete checkpoint is a flagged fallback only if validation never ran). A preempted or restarted run resumes exactly — the loss curve and adaptive stop step continue as if uninterrupted.

Cancelling​

POST /runs/{id}:cancel stops a queued run immediately; a running run stops gracefully and leaves a final valid checkpoint when it reaches its next checkpoint before its training Job is stopped. Either way the run ends cancelled, keeping the checkpoints it has. Terminal runs emit a run.completed webhook and meter training_gpu_seconds usage.

A succeeded run registers a model version in status training; evaluation and promotion gates take it from there.