Skip to main content

Pre-registered protocols and verdicts

A comparison you design after seeing the results proves little. A protocol fixes the terms of the comparison before the data it judges exists: which labels, which metrics, which internal quality safeguards and comparators your trained model must beat, and by what margin. Tessera records it immutably, then judges every evaluation against exactly those terms. The verdict can say fail. Verdicts are advisory evidence: promotion gates G1–G6, serving and deliveries never read them.

1. Declare a protocol before building the dataset​

POST /v1/protocols with this body:

{
  "name": "churn model against the current churn score",
  "provenance": "production",
  "labels": ["lbl_…"],
  "metrics": ["auc", "lift@10pct"],
  "comparator_ids": ["cmp_…"],
  "margins": {"auc": 0.005},
  "banding_policy": {
    "bands": [{"band": "high", "min_score": 0.8}, {"band": "low", "min_score": 0.0}],
    "list_size": 500,
    "alert_budget": 50
  }
}
  • Labels are your label definitions (lbl_…).
  • Metrics:
    • each label's primary metric: auc for binary, macro_f1 for multiclass, r2 for regression;
    • the decision metrics lift@{1,5,10}pct, capture@{1,5,10}pct, precision@list, recall@list and alert_precision@budget, for binary labels.
  • Arms (optional) select the platform's internal quality safeguards from Decision metrics; the accepted values are listed in the API reference. Comparators are your uploaded scores. A protocol needs at least one internal safeguard or comparator.
  • Margins are optional, per metric, and default to 0. A criterion passes only when the lower bound of the paired 95% confidence interval of (model − reference) is strictly above its margin.
  • Provenance is your claim about the data the protocol judges: fixture, design-partner or production. It is copied into every verdict.

The response carries protocol_fingerprint (the canonical-JSON fingerprint of what you declared), declared_at (the server time of the protocol's audit event) and audit_id. There is no update or delete. To change the terms, declare a new protocol.

Order matters. A protocol can judge a dataset build only when it was declared strictly before the build's manifest created_at. Otherwise evaluations fail with failed_precondition, reason protocol_after_data. Declare first, then build the dataset.

2. Evaluate a model against the protocol​

POST /v1/evaluations with this body:

{"model_id": "mdl_…", "protocol_id": "prt_…"}
  • The request names the model and the protocol, plus an optional seed. It never restates criteria: extra fields are rejected.
  • The answer is 202 with the evaluation's index row (status: running).
  • Poll GET /v1/evaluations/{report_id} until status is succeeded or failed. The Job retries a failed attempt; the evaluation reads failed (reason job_failed:<reason>) only after the last attempt. A run still running after the Job's 6-hour deadline reads as failed with reason timeout. A pod that dies without reporting (out of memory, evicted, deadline exceeded) is only visible through that timeout. Posting the same request again relaunches a failed evaluation.
  • Repeating the identical request returns the same evaluation.

The evaluation runs the full model evaluation with the protocol's internal safeguards, banding policy, margins and comparators. Its report is the model's latest evaluation report. Gates G1–G6 are recomputed from the same inputs, so they equal those of the model's other reports for the same seed.

3. Read the verdict​

GET /v1/evaluations/{report_id} returns the index row and, once succeeded, the stored report verbatim, re-hashed against its recorded fingerprint before it is served. The report adds two blocks:

  • verdict: one row per declared criterion (label, metric, arm), in sorted order:

    • pass when the margin verdict is true;
    • fail when it is false;
    • not_evaluated with a reason when the report has no evidence for it, for example a reference that failed or skipped the label, or a decision metric on a multiclass label.

    The overall verdict is pass only if every criterion passes, fail if any fails, and incomplete otherwise. Missing evidence is never shown as a win or a loss. The block also carries the protocol fingerprint, the provenance, and a recompute appendix: every input fingerprint, the seeds, and the exact API calls that reproduce the numbers.

  • evaluation_basis: what was judged. It lists the protocol, the dataset build and its created_at, and the target_set_fingerprint of the evaluated test targets. Evaluations of the same dataset under the same protocol share this fingerprint, so their verdicts compare like for like.

GET /v1/evaluations?protocol_id=prt_… lists every evaluation of a protocol, newest first.

4. Backtest: how early would each scorer have alerted?​

POST /v1/backtests with this body:

{"protocol_id": "prt_…", "model_id": "mdl_…"}

Instead of model_id you can name a deployment slot. The server backtests the model of the slot's active bundle at request time. Or name a bundle_id to backtest that bundle's model. A backtest is identified by the bundle it resolved to, not by how it was addressed: the same bundle by slot or by id is one backtest. A later activation starts a new one, and rolling the slot back to an earlier bundle returns the earlier backtest. The report records the bundle, its fingerprint and the slot named by the request that created the backtest (a later request for the same bundle, by another slot or by id, returns it unchanged).

A backtest replays a ladder of past cutoffs over the test split of the model's dataset, for the model, the protocol's internal safeguards and its comparators:

  • Scores at a cutoff. A scorer's score for an entity at cutoff c is its output at the entity's latest test anchor at or before c. A comparator's score is its latest row with as_of at or before c.

  • Alerts. At each cutoff, the top alert_budget entities (from the protocol's banding policy) are alerted. Ties go to the higher score, then the lower entity key.

  • Outcomes. A positive's outcome time is its first matured positive observation after the first cutoff, as recorded in the dataset build. A derived label's positive is timed when it happened: a presence rule (a conversion or a repeat purchase) at the event that satisfied it, an absence rule (churn) at the end of its horizon. Positives whose outcomes all fall at or before the first cutoff are excluded and counted.

  • Lead time. For each lead day d in 90:

    • a positive is eligible when its outcome is at least d days after the first cutoff. Eligibility is the same for every scorer.
    • a scorer succeeds when its first counted alert came at least d days before the outcome.

    The curve is share_alerted_at_least_d_days: successes over eligible positives, with an entity-bootstrap 95% confidence interval. n_eligible is reported per lead day. The median lead days of alerted positives is reported too.

  • Silence. Scores of prefix_only scorers (every comparator, a model without observation time, and the internal safeguards) change only when the entity has new events. In a full-book backtest of a model with observation time, the internal safeguards score as of each cutoff and are graded as observation_time, like the model. A first alert in an interval where the entity had no event came from other entities' scores moving, not from new evidence, so it does not count. Only alerts at the first cutoff, or after the entity had an event since the previous cutoff, count. The denominator never changes: every scorer divides by the same eligible positives. first_alert_in_silence counts the positives whose first alert did not count.

Without cutoffs, the server uses 6 evenly spaced cutoffs over the dataset's time window and records them. You can pass 2–12 ascending RFC 3339 instants inside that window instead.

A backtest never refreshes a slot or writes served scores. The same request always produces byte-identical reports. Read results with GET /v1/backtests/{backtest_id}. The report's recompute appendix lists every fingerprint, the bootstrap seed and the API calls behind the numbers.

The full-book out-of-time backtest​

Add "engine": "full_book_out_of_time" and a bundle_id (or a slot) to backtest a deployed bundle out of time. The server rescores the bundle's whole entity book at each cutoff, exactly as a refresh of that bundle would, over the event-store snapshot pinned by the model's dataset. It never writes served scores.

  • Window. Cutoffs fall after the model's training cutoff (the end of the dataset's time window) and no later than the dataset's censor clock minus the protocol labels' longest maturation, as the dataset pinned them. The censor clock is the earlier of the snapshot instant and the time of the dataset's last event: a finite export ends before its snapshot is taken, and no outcome is graded after the data ends, so a churn rule never reads the missing data as a lapse and a conversion rule never reads it as a miss. Without cutoffs, the server uses 6 evenly spaced cutoffs, the last at the end of that window. Cutoffs outside it are refused, and a dataset with no matured out-of-time window gets failed_precondition (full_book_window_empty). A request without a bundle gets failed_precondition (bundle_required). A dataset built by builder 1.3.0 or later records where its data ends. For an older dataset, finding it reads the newest files of the dataset's snapshot. If a file's bytes no longer match what the snapshot recorded, the request gets failed_precondition (snapshot_file_mismatch), and a backtest that finds the mismatch when it runs reads failed (reason job_failed:snapshot_file_mismatch). Take a new snapshot and rebuild the dataset, then backtest a model trained on the rebuilt dataset: the backtest reads the snapshot pinned by the model's dataset.

  • Out-of-sample entities only. The dataset build attaches outcomes up to its snapshot, so after the training cutoff, the outcomes of the dataset's train and validation entities were training targets. The full-book backtest grades only entities in the dataset's test split and entities the dataset does not hold at all. The number of in-sample entities left out is logged with the run.

  • Models trained on this dataset only. A shared model import or a model trained on a foundation corpus learned from other datasets, whose entities can overlap the window. Its in-sample entities are not known, so the request gets failed_precondition (full_book_model_not_dataset_trained).

  • Event time, not arrival time. A cutoff's window holds the snapshot's events up to the cutoff by event time, including events imported or received later, as a historical backfill is. A backtest therefore shows what the model would have scored with the data as it is now known, not only what had arrived by each cutoff.

  • Erased entities. Entities erased by the time the backtest runs are dropped from scores and outcomes, even though the pinned snapshot still holds their events. Their rows in the dataset's train and validation splits are left out before any internal safeguard is fitted, and no internal safeguard scores them. The model rescores the whole book, as a refresh would, but an erased entity's scores are never graded.

  • Outcomes. Outcomes are read as of the last cutoff plus the longest maturation, a time fixed at request time and recorded as outcomes_as_of. A positive counts only when it matured under the same rule the dataset build uses: at least the label's maturation after the entity's latest event at or before it. A derived label is computed from the snapshot's events after the training cutoff, with the same rule as the dataset build.

  • Internal safeguards. Each declared internal safeguard is fitted exactly as the evaluation fits it, on the dataset's train split. It scores every graded entity at each cutoff from the same events the model sees, at the entity's latest event (its score changes only when the entity has new events). For a model with observation time, the safeguards score as of each cutoff, as they were trained. A safeguard reports skipped for a label when the training data cannot support it, and the backtest still grades the other scorers:

    • insufficient_label_support: the label has fewer than 50 labeled targets in the dataset's train split (labeled target events, so one entity can contribute several), or all of its labeled training targets have the same value (one class only). Only the train split is checked; the validation split is not.
    • family_not_applicable: the train split gives that safeguard nothing to fit for the label.

    A safeguard that cannot be fitted or scored reports failed with a typed reason, and the other scorers and the model are graded as usual. A model whose inputs are semantic.v2 gets no internal-check grading (the typed reason is zoo_not_supported_for_semantic_inputs), so G1 reports baseline_unavailable for it: acknowledge that with a reason, or use v1 or descriptions.v1 inputs.

  • Earlier reports. Full-book backtests requested since the internal safeguards are graded get new ids, so a report computed before then is never returned for a new request. It stays readable by its own id. A backtest whose run started before then and had not finished reads failed (job_failed:failed_precondition); request it again. A full-book report requested before the censor clock bounded the window (2026-10-15) may also read outcomes after the dataset's data ends: its outcomes_as_of is later than the dataset's last event, and churn-style labels may count positives that never happened. To recompute such a report, request the backtest again: the new request resolves the censored window, so it gets new cutoffs and a new backtest id.

Full-book backtests are not metered, like evaluations.

5. Detection-quality thresholds and how Tessera verifies them​

A protocol can declare detection_quality thresholds, per label, over alert_precision@budget, precision@list or recall@list:

"detection_quality": [{"label": "lbl_…", "metric": "alert_precision@budget", "min_value": 0.8}]

The thresholds are part of the protocol fingerprint. No API operation runs them against your data yet. Tessera's own verification uses them in an injection test harness. Today its acceptance tests run the harness; next, the phase driver and the evaluator's real-image check will run it too. The harness checks both directions over two books of entities built from one seeded stream. Both books hold the same entities with identical histories. In the planted book, a fixed set of entities also carries a planted sequence, for example a burst of failed external logins followed by a privilege escalation and a large egress. The control book has no plant. The model under test scores both books at the same cutoff.

  • Alert. An entity is alerted when it ranks within the protocol's alert_budget (higher score first, then the lower entity key) and its score reaches the highest band of the banding policy (score >= bands[0].min_score). The budget alone always selects someone; the band is what lets a quiet book raise no alert.
  • Planted direction. Each declared threshold must hold, with the planted entities as positives and every other entity as negatives, and at least one planted entity must be alerted. With no thresholds declared, every planted entity must be alerted. alert_precision@budget counts the top-alert_budget ranking, as in the decision metrics; the band only decides the alerted set used by the alerted-entity rules.
  • Control direction. No entity may be alerted.
  • Every label. Each declared label is judged and listed with its rule: its own thresholds if it declares any, otherwise every planted entity must be alerted. A label with no planted entities is not_evaluated, and the result is then incomplete, not pass.
  • Reachable bars only. If even a perfect model could not pass on the books (a threshold out of reach, more planted entities than the alert budget under the every-planted rule, or a highest band no score can reach or every score reaches), the harness stops with harness_fixture_cannot_satisfy:<bar> rather than reporting a model failure.

A label passes only when both directions pass. Catching the plant while the control also alerts is a fail, and so is a quiet control that misses the plant. Each failed direction lists typed reasons: threshold_not_met (with the metric, its value and min_value), no_planted_alert, planted_not_alerted or control_alerted (with the count and entity keys). Each result records the protocol id and fingerprint, the seeds and the fingerprints of both books, and the same inputs always produce byte-identical results.