Training runs

POST /runs trains a behavioral model on a succeeded dataset build: self-supervised next-event modeling plus one head per label. Runs are content-addressed by their full config (dataset fingerprint, model config, trainer settings, seed) — the same request twice is one run.

The default way to train a model is a training study: a fixed grid of eight ordinary runs, the winner chosen on validation data only, and one evaluation of that winner. POST /runs remains for training exactly one configuration, and a study's children are ordinary runs created through the same path. Everything below applies to both, with one exception: POST /runs never returns a run that a training study created. A request for the same configuration (a clone of a study child included) starts its own run instead.

The request

{
  "dataset_id": "ds_00000000000000000000000000",
  "model_config_tier": "S",
  "seed": 7
}

• model_config_tier — S (default, ≈30M params), M (≈120M), or L (Phase 3). Tier presets fix the architecture; overrides may adjust dimensions within bounds (out-of-bounds or unknown keys are invalid_argument — the quickstart's CI profile is a worked example of legitimate overrides).
• labels — subset of the dataset's labels to train heads for; omitted = all; [] = self-supervised only.
• trainer — the maximum epoch budget, token batch sizes, checkpoint/validation cadence, learning rate, and adaptive-duration settings. Every resolved value lands inside the fingerprinted run config and is echoed on the run; there are no invisible settings.
• seed — training randomness is seeded and recorded: the same (dataset, config, seed) reproduces metrics within tight tolerance.

Batch size, schedule and label weights

When trainer.global_batch_tokens is omitted, Tessera sizes the batch from the dataset so that every epoch has about 100 optimizer steps: clamp(pow2_floor(train_tokens / 100), 16384, 1048576). A 3.8 million-token train split gets a 32,768-token batch, 117 steps per epoch and about 1,170 updates over ten epochs. Datasets of 104,857,600 train tokens or more keep the 1,048,576-token batch. When max_tokens_per_microbatch is omitted as well on a single-process run, the microbatch cap follows the batch (min(131072, max(ceil(batch / 2) + 1024, longest train row))), so each optimizer step really holds about one batch of tokens. That coupling needs a dataset built by dataset builder 1.2.0 or later, which records each split's longest row in the manifest (token_stats.split_max_row_tokens). A dataset built earlier keeps the 131,072 cap; rebuild it to get the coupled cap (the rebuild is a new dataset version, because the builder version is part of a dataset's identity). Explicit values are always honored exactly, and the resolved pair is echoed on the run. A study sets the batch on each child itself and gives each child the microbatch cap that follows its batch; it needs a dataset built by dataset builder 1.2.0 or later.

Single-process runs on the wsd schedule compute their schedule horizon from the batches they will really pack, once at start. The learning rate then decays to zero on the last update the data supports, and a run that completes it reports stop_reason: budget_cap.

Unless overrides.label_weights is given, every binary label weighs 10 in the training loss; other labels weigh 1. An explicit map, {} included, is used as given.

After training, a single-process run with at least one binary label refits each binary head on the frozen representation of the checkpoint it ships. The refit uses training rows only, with a regularised logistic fit whose strength is chosen by five-fold cross-validation. The refit checkpoint becomes the model version, and it is listed last in the run's checkpoint_uris. Multiclass and regression heads keep their trained weights. The refit is deterministic: the same checkpoint and training rows always give the same head. A head whose refit would not fit in the trainer's memory keeps its trained weights, and the run still completes. Cancelling a run during the refit cancels it with the checkpoints it already has.

Adaptive duration

New runs treat epochs as a maximum, not a promise to execute every epoch. The defaults are epochs: 10, early_stopping: true, patience: 5, min_delta: 0.0001, cooldown_ratio: 0.2, and lr_schedule: wsd. When early stopping is enabled and val_every_steps is omitted, Tessera validates at every actual packed epoch boundary and fingerprints the nominal derived cadence, capped at 500 optimizer steps. This remains correct when whole-microbatch packing produces fewer updates than the nominal budget math. Short WSD runs also pin a proportional cooldown minimum in their immutable config; long runs retain the 50-step quality floor. Explicit validation cadence is always honored exactly, and fixed-horizon mode retains its legacy cadence. Early stopping requires the WSD schedule; use early_stopping: false with lr_schedule: cosine for fixed horizon.

Each full validation pass compares val_loss with the best result so far. If five consecutive checks do not improve it by at least min_delta, the run enters a deterministic learning-rate cooldown and then succeeds. The best validation checkpoint—not simply the newest checkpoint—becomes the model version. A run with label heads picks that checkpoint by its labels, not by the total loss: the validation with the greatest mean primary label metric (AUC for binary labels, macro-F1 for multiclass, R² for regression). An exact tie, such as an AUC of 1.0 at several checks on a small validation split, goes to the lower total loss, then to the earlier check. The total loss also counts the next-event terms, which can keep improving after a label has started to overfit. best_step names that checkpoint, while best_val_loss stays the lowest total validation loss seen, so the two can belong to different steps. A run without validation label targets selects by the total loss. To request the former fixed-horizon behavior, set early_stopping: false, lr_schedule: cosine, and the desired epoch cap.

total_steps is therefore an immutable budget cap. A stopped run reports final_step, stop_reason, best_step, and cooldown_start_step; it remains in the normal succeeded state.

Following a run

• GET /runs/{id} — status (queued → running → succeeded | failed | cancelled), the resolved trainer echo, budget/stop facts, checkpoints, and the resulting model_id on success.
• GET /runs/{id}/metrics?after_step= — incremental metric series (loss components, learning rate, throughput, and served epoch) for dashboards and polling. Fetch one final page after observing a terminal run so the final epoch point cannot lag the run record.
• GET /runs/{id}/logs — paginated timestamp, severity, and message entries. Pass ?structured=true to include available step/epoch, loss, throughput, validation, checkpoint, and early-stopping fields. The opt-in keeps older strict API clients compatible with the default response.
• POST /runs/{id}:cancel — graceful stop; the run keeps its checkpoints.

A run stays queued until its compute class has a free node. When the class has no capacity (its pods cannot be scheduled for five minutes, for example while a GPU pool is out of capacity), the run's logs carry a WARNING entry that names the class: the run starts automatically when capacity frees up, or you can cancel it and re-post it on another compute class. The entry is repeated periodically while the run waits; it does not change the run.

trainer.compile is off by default on every compute class, so runs train eagerly. On GPU, compilation stays off until a gpu-l4 run shows that it is faster than eager training. Set trainer.compile: true to opt in: the run then compiles the model's dense encoder kernels, and the number of compiled variants stays small and fixed however the batches vary. If a run ever compiles more variants than that bound, it continues without compilation instead of slowing down: the logs carry a torch.compile fell back to eager warning with the reason, and the metrics carry one compile_fallback point (with compile_graphs) at that step.

Checkpointing is atomic and fingerprinted; a resumed run reproduces the loss curve and stop step exactly, including a resume during cooldown. The last checkpoints and the selected best checkpoint are retained, plus the head-refit checkpoint the model version uses when the run refits its binary heads.

Run logs

Run logs are kept for 30 days. GET /runs/{id}/logs and the console's run page read the trainer's log lines from the platform's log store: a Cloud Logging bucket whose default retention is 30 days, which the platform does not change (see Google Cloud's logs retention periods). After that the endpoint returns an empty page for the run, and the console shows "No logs are available for this run. Logs are kept for 30 days." The run record and its metrics are stored separately and stay available.

Evaluation and the gates

A succeeded run produces a model version and evaluation runs automatically on the dataset's test split. GET /models/{id}/eval-report serves the six-gate report:

Gate	Question it answers
G1 Label quality vs baselines	does each head beat the previous accepted baseline (or the trivial baseline), and is it within 0.01 of the internal reference model with a paired 95% CI lower bound of the difference of at least -0.03?
G2 Score calibration	are predicted probabilities honest (ECE ≤ 0.05)?
G3 Count invariance	do embeddings encode behavior, or just activity volume?
G4 Train/serve tokenizer parity	do the training path and the serving path produce byte-identical tokens?
G5 Label-leakage audit	independently recomputed masks: zero targets from the future?
G6 Drift baseline	were complete reference distributions produced for production drift monitors?

Failing gates come with actionable diagnostics (the offending labels, attributes or dimensions). G1-G3 are quality judgments and are advisory by default: they require an explicit, reasoned acknowledgement before bundling. G4-G6 are integrity checks and always block. A project's strict policy or a raise-only override can make quality failures blocking too. Read the model's server-resolved promotion verdict rather than deriving a decision from the unchanged gates_passed report fact. Promotion gates explains the policy, acknowledgement, audit trail, and G1 anti-ratchet.