Decision metrics, internal reference checks & comparators

Every evaluation report already answers "is your trained model better than a simple reference?" (the automated internal baseline check), and the platform checks every sequence model against its internal quality safeguards before it can be promoted. The proof engine extends that answer into the terms you act on — ranked lists, alert budgets and score bands — measured for the model, the internal reference checks and your own uploaded scores, all by one metric implementation (tessera_core.metrics). Every verdict here is advisory, with one exception: G1 reads the internal reference checks of the same report. For each gating label, the model must be within 0.01 of the internal reference model on the label's primary metric, and the lower bound of the paired 95% interval of that difference must be at least −0.03 (see Promotion gates). The decision metrics, the margin verdicts (beats_baseline_at_margin), the comparators and the banding policy are never consulted by a gate, and a customer comparator is never part of the internal reference set.

Availability

• Available now: comparators (POST /comparators, GET /comparators, GET /comparators/{comparator_id}), protocols (POST /protocols) and the evaluations that apply one: POST /evaluations for a model and POST /evaluations:baseline for the protocol's internal reference checks before any model exists. The proof center in the console is the screen for all of these.
• Proof-engine evaluations: the protocol supplies the internal reference checks, the banding policy, the margins and the comparators, so the report carries the blocks below plus a pre-registered verdict. See Protocols & verdicts and Baseline evaluations.
• The platform's automatic evaluations (after a training run, when preparing a promotion, and the evidence evaluation of POST /models/{model_id}/signals-evaluations) run the internal reference checks, because G1 needs them. They carry no decision_metrics or comparator gradings unless a protocol asks for them.

The rest of this page documents the semantics of those blocks.

Decision metrics

A protocol evaluation adds a decision_metrics block to the report, measured under the protocol's banding policy. The proof-engine inputs are declared in the protocol (POST /protocols); the internal reference checks run when it declares any:

{
  "banding_policy": {
    "bands": [
      {"band": "high", "min_score": 0.8},
      {"band": "medium", "min_score": 0.5},
      {"band": "low", "min_score": 0.0}
    ],
    "list_size": 500,
    "alert_budget": 50
  },
  "margins": {"auc": 0.005, "capture@10pct": 0.01},
  "comparator_ids": ["cmp_…"]
}

The policy is data, never code: bands are ordered by strictly descending min_score, list_size and alert_budget are at least 1, and the normalized policy is canonical-JSON fingerprinted. The report records the fingerprint (banding_policy_fingerprint), echoes the normalized policy, and the fingerprint joins the report's identity. The same fingerprinted policy object later drives serving-side band columns and deliveries — the bands you act on are the bands the bake-off measured.

The decision unit is the entity. Each test entity contributes one ranked row per label — its latest masked test target — so "top 10%", list_size, alert_budget and band counts are counts of entities, not of targets (decision_unit: "entity_latest_target"). For every binary label (and per class, one-vs-rest, for multiclass labels) each reference reports, with 95% entity-level bootstrap confidence intervals from the same resample engine as every other report metric:

• lift@k / capture@k at the 1% / 5% / 10% ranked fractions,
• precision/recall at list_size,
• alert precision at alert_budget,
• band-assignment matrices per pair of scorers: how many entities each pair places in the same or different bands, with per-band positive counts.

Ranking ties break deterministically (score descending, then entity key ascending). Regression labels report not_applicable.

When a binary label's decision entities hold no positive, lift@k, capture@k and recall@list are undefined, so they are left out (the label reads n_positive_entities: 0). precision@list, alert_precision@budget and the band-assignment matrices are still measured (precision is a real zero there). This happens when every entity's latest target is negative, for example a repurchase label where most entities' last purchase has no follow-up, even though earlier targets (and so AUC) do have positives. A protocol criterion on an undefined metric reads no_positive_entities instead of a value and is never judged, so the protocol verdict cannot pass on it; criteria on the measured metrics are graded as usual. Multiclass classes and comparators follow the same rule. Whenever positives exist, capture@k = lift@k × n_top / N. Confidence intervals for labels with very few positive entities include bootstrap resamples without a positive, which use the indifference values (lift 1, capture n_top / N); read such intervals as wide.

Margin discipline: beats_baseline_at_margin

A bare delta > 0 is not a win. For every label × metric × reference pair — internal reference checks and comparators alike — the report computes the paired entity-bootstrap CI of (model − reference) and claims a win only when the CI lower bound strictly exceeds your declared margin (margins: metric name → margin ≥ 0, default 0). A +0.001 delta whose CI spans zero is reported as not a win. The legacy model_beats_baseline sign check on the internal baseline check remains, byte-identical, for compatibility and is never cited by the new blocks.

This is what makes band erasure measurable: a model that beats a reference by +0.015 AUC while assigning exactly the same entities to the top band reports the AUC win and a band tie — an all-diagonal band-assignment matrix and beats_at_margin: false on every banded metric.

The internal reference checks

The platform runs its internal quality safeguards beside the model. Their results are used for gate G1 and for the margin verdicts above; this page does not describe how they are built.

Every check inherits the baseline harness discipline: point-in-time-safe features from the same causal prefix, identical masked test targets asserted before any delta, insufficient-support skips, and artifacts under the report's baseline/ prefix that are never bundled or served. A check that does not apply to a label is skipped with a typed reason, never fabricated. Each check's primary-metric verdict is computed once and reused wherever the check appears. A failure in any of these advisory stages is recorded as a typed failed block; it never prevents the report — and its G1–G6 results — from being written.

Comparators: grade your own scores (POST /comparators)

Comparator files — a Klaviyo predicted-CLV export, GA4 bands, a UEBA score, an in-house model — are evaluation inputs only: they are never ingested, never bundled and never served. Upload an uncompressed, comma-delimited CSV (with a header row) or a Parquet file of up to 64 MiB, then register it without completing the upload:

1. POST /uploads with the file name and format — no source_id.
2. PUT the bytes to the returned upload_url.
3. Register the session (do not call POST /uploads/{upload_id}:complete, which attaches a file to an ingestion source and syncs it):

POST /comparators
{
  "name": "klaviyo-predicted-clv",
  "upload_id": "upl_…",
  "column_mapping": {"entity_id": "customer", "as_of": "scored_at", "score": "clv"}
}

Registration claims the upload session: it can never be completed into an ingestion source afterwards (:complete answers 400 failed_precondition, upload_claimed_by_comparator). A session that was already completed, or created for a source, is refused (upload_bound_to_ingestion), as is one with no uploaded bytes or past its 24-hour expiry.

Registration answers 202 with the comparator in pending; an asynchronous validation job then reads the uploaded file and settles it to ready (row and entity counts, the as_of range and the file fingerprint) or failed (with a reason and up to 100 per-row violations). Poll GET /comparators/{comparator_id}. Entity ids in a text column are read as text (leading zeros are kept); a numeric column follows the entity id rule, so 12 and 12.0 are the entity 12 and a fractional or non-finite id is a violation. as_of must be RFC 3339 with a UTC offset, and score must be a finite number.

A protocol names up to four comparators (comparator_ids). Each must be ready when an evaluation runs, and each is graded on the identical masked test targets with the identical metric code, bootstrap and banding policy as the model and the internal reference checks. The file fingerprint joins the report's identity and is re-verified before grading.

Point-in-time discipline is fail-closed: for each target at entity-time t the grader uses your row with the greatest as_of ≤ t; any row usable only by leaking (as_of after every target time of its entity for a label) fails the comparator's block with per-row violations — leaked rows are never silently dropped. Targets without a qualifying row reduce coverage, and every paired comparison is computed on the covered subset under the covered target-set fingerprint, so comparisons are never apples-to-oranges. Multiclass labels are not graded against comparators yet (skipped).

Comparators and entity deletion

Deleting an entity (POST /entities/{entity_type}/{entity_id}:delete) also removes its rows from the project's comparator files: each row whose mapped entity_id column holds that id, read exactly the way validation reads it. In a CSV file only those lines are cut: every other byte stays as it was. A Parquet file is written again in the layout it was written in (decimals stored as integers, INT96 timestamps and the file's metadata stay so, and a categorical column keeps only the categories the remaining rows use). It is used only when every column keeps its type and every other row its exact values (an ordered category keeps its order), and when the new file keeps the entity's id in no identifier column, category, statistic, column name or metadata; otherwise the file is kept and the deletion reports it for review. Another entity's row that names the id in a different column (a referrer) stays as it is, and the deletion reports the file for review with the number of such rows, on every pass. The rewritten file is validated again (the comparator's counts, as_of range and fingerprint change), and the file's earlier versions are deleted.

• Shared uploads. When several comparators use one upload, the file is rewritten once, without the rows any of their mappings attributes to the entity, and every one of them is validated again.
• Entity type. A comparator file names no entity type. When every evaluation and backtest that used the comparator graded the same entity type, only a deletion of that type removes rows from it. Otherwise a row is removed whenever its id matches, whatever its type.
• Files validation cannot read. A comparator file is never deleted. A file that no mapping can interpret (it does not parse, its mapped column is missing, or its identifiers are not text) stays as it is, and the comparator keeps the reason validation gave it. When such a file still holds the entity's id (in any cell, also within a value such as 12345;678 or as the number 12345.0), the deletion completes and reports the file for review.
• Uploads after a deletion. Validation drops the rows of an entity that was deleted in the comparator's environment before it validates the file, and grading never reads such a row.

A run in flight that pinned the earlier file fails, and its reason is comparator_rewritten_by_erasure. A request made while the deletion is rewriting the file is refused with failed_precondition, reason comparator_rewritten_by_erasure, and the comparator_id in details[0]. After the rewrite, a request (including a replay of a recompute appendix) grades the rewritten file under a new report id. Reports recorded before the deletion keep their grades. A validation violation names the row and the column, never the cell's value.