Design-partner runbook

This runbook runs a comparison on a design partner's own data and publishes what it shows, whatever that is. The comparison is fixed in writing before the data it judges exists, so the outcome cannot be chosen afterwards. A failing verdict is a result and is published as one.

Three rules hold throughout:

The protocol comes first. It is registered before the partner's dataset is built. Tessera refuses to judge a build that is older than its protocol's declaration.
Thresholds come from the protocol. Margins, metrics, comparators and internal reference checks are read from the registered protocol. They are never restated on a command line or edited in a report.
The class never goes up. A run reports the provenance class of its protocol. A run on recorded or generated data is a fixture run and is never relabelled design-partner or production.

note

The steps below call the public API. They are linted with the rest of these docs but are not run by CI, because they need a partner's project and data.

Before you start

Need	Detail
A partner project	A project in the partner's organization with its own data connected and a contract approved. Not a copy of another project's data.
An API key	`read` and `write` scopes on that project, from the partner's organization.
Labels	The label definitions the comparison judges, already created.
Comparators	At least one ready comparator (`POST /comparators`): the partner's current scores. A protocol needs a comparator or an internal check, or it is refused with `invalid_argument`; the example below names one comparator.
An agreed protocol	A JSON body naming the labels, metrics, comparators, internal reference checks and margins, signed off by the partner before any dataset build.
A place to archive	Somewhere durable for the protocol, the reports and their fingerprints; the published document cites them.

The commands read TESSERA_API_BASE (for example https://api.staging.tessera.zone/v1) and TESSERA_API_KEY from the environment. They keep ids between steps in $TESSERA_STATE_DIR.

1. Register the protocol

The protocol body, with placeholder ids to replace with your label and comparator ids:

{
  "name": "partner comparison against the partner's current scores",
  "provenance": "design-partner",
  "labels": ["lbl_00000000000000000000000000"],
  "metrics": ["auc", "lift@10pct"],
  "comparator_ids": ["cmp_00000000000000000000000000"],
  "margins": {"auc": 0.005},
  "banding_policy": {
    "bands": [{"band": "high", "min_score": 0.8}, {"band": "low", "min_score": 0.0}],
    "list_size": 500,
    "alert_budget": 50
  }
}

Save the body to a file and point TESSERA_PROTOCOL_BODY at it:

01-register-protocol.sh
#!/usr/bin/env bash
# Design-partner runbook, step 1: register the protocol BEFORE the dataset is built.
# The protocol fixes the labels, metrics, comparators and margins the results are
# judged against. Set TESSERA_PROTOCOL_BODY to the path of the JSON body you
# agreed with the partner (see the runbook page for its shape).
#
# The Idempotency-Key is derived from the request body, so the same body is
# the same request (a safe retry) and a changed body is a new protocol. Set
# TESSERA_IDEMPOTENCY_SALT to register a second protocol with an identical body.
set -euo pipefail

: "${TESSERA_API_BASE:?e.g. https://api.staging.tessera.zone/v1}"
: "${TESSERA_API_KEY:?the tsk_ key of the partner project}"
: "${TESSERA_PROTOCOL_BODY:?path to the agreed protocol JSON body}"
: "${TESSERA_STATE_DIR:?set by the runner; ids are handed between steps}"
AUTH=(-H "Authorization: Bearer $TESSERA_API_KEY")

KEY="docs-partner-protocol-$(python3 -c '
import hashlib, os, sys
body = open(sys.argv[1], "rb").read()
salt = os.environ.get("TESSERA_IDEMPOTENCY_SALT", "").encode()
print(hashlib.sha256(body + salt).hexdigest()[:24])
' "$TESSERA_PROTOCOL_BODY")"

if ! PROTOCOL="$(curl -sS --fail-with-body -X POST "$TESSERA_API_BASE/protocols" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $KEY" \
  --data-binary "@$TESSERA_PROTOCOL_BODY")"; then
  echo "registering the protocol failed:" >&2
  printf '%s\n' "$PROTOCOL" >&2
  exit 1
fi
printf '%s' "$PROTOCOL" | python3 -c '
import json, sys
protocol = json.load(sys.stdin)
print("protocol_id:", protocol["protocol_id"])
print("provenance:", protocol["provenance"])
print("protocol_fingerprint:", protocol["protocol_fingerprint"])
print("declared_at:", protocol["declared_at"])
' | tee "$TESSERA_STATE_DIR/protocol.txt"
printf '%s' "$PROTOCOL" | python3 -c 'import json,sys; print(json.load(sys.stdin)["protocol_id"])' \
  > "$TESSERA_STATE_DIR/protocol_id"

Download playbooks/design-partner-runbook/01-register-protocol.sh — this exact file runs in CI on every change (hosts shown are for this environment).

Keep the printed protocol_fingerprint and declared_at. There is no update or delete: to change the terms, edit the body and run the step again; a new body is a new request and registers a new protocol, and the document says so. Running the same body again is a safe retry that returns the same protocol. To register a second protocol with an identical body, set TESSERA_IDEMPOTENCY_SALT to any new value.

2. Build the dataset and train

Only now build the partner's dataset, then train. See Datasets and Training runs. The build's created_at must be later than the protocol's declared_at; otherwise step 3 fails with failed_precondition, reason protocol_after_data. If it does, the order was wrong. Register a new protocol and rebuild, and say so in the published document.

3. Judge the model against the protocol

02-evaluate-and-read-verdict.sh
#!/usr/bin/env bash
# Design-partner runbook, step 3: judge the trained model against the protocol.
# Run after the partner dataset was built (strictly after the protocol was
# declared) and a model was trained on it. Set TESSERA_MODEL_ID.
#
# The Idempotency-Key is derived from the model and the protocol: repeating the
# same judgement is a safe retry, and a different protocol is a new request.
set -euo pipefail

: "${TESSERA_API_BASE:?e.g. https://api.staging.tessera.zone/v1}"
: "${TESSERA_API_KEY:?the tsk_ key of the partner project}"
: "${TESSERA_MODEL_ID:?the trained model, mdl_...}"
: "${TESSERA_STATE_DIR:?set by the runner; ids are handed between steps}"
AUTH=(-H "Authorization: Bearer $TESSERA_API_KEY")
PROTOCOL_ID="$(cat "$TESSERA_STATE_DIR/protocol_id")"
MAX_POLLS="${TESSERA_MAX_POLLS:-720}"   # 5 s apart: one hour by default

fail() { echo "$1" >&2; exit 1; }

KEY="docs-partner-evaluation-$(printf '%s:%s' "$TESSERA_MODEL_ID" "$PROTOCOL_ID" | python3 -c '
import hashlib, sys
print(hashlib.sha256(sys.stdin.buffer.read()).hexdigest()[:24])
')"
EVALUATION="$(curl -sS --fail-with-body -X POST "$TESSERA_API_BASE/evaluations" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $KEY" \
  -d '{"model_id": "'"$TESSERA_MODEL_ID"'", "protocol_id": "'"$PROTOCOL_ID"'"}')" \
  || fail "starting the evaluation failed: $EVALUATION"
REPORT_ID="$(printf '%s' "$EVALUATION" | python3 -c 'import json,sys; print(json.load(sys.stdin)["report_id"])')"

STATUS=""
for _ in $(seq 1 "$MAX_POLLS"); do
  DETAIL="$(curl -sS --fail-with-body "$TESSERA_API_BASE/evaluations/$REPORT_ID" "${AUTH[@]}")" \
    || fail "reading evaluation $REPORT_ID failed: $DETAIL"
  STATUS="$(printf '%s' "$DETAIL" | python3 -c 'import json,sys; print(json.load(sys.stdin)["evaluation"]["status"])')"
  if [ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ]; then break; fi
  sleep 5
done
echo "evaluation: $STATUS"
if [ "$STATUS" != "succeeded" ]; then
  printf '%s' "$DETAIL" | python3 -c '
import json, sys
row = json.load(sys.stdin)["evaluation"]
print("reason:", row.get("reason", "still running after the polling bound"), file=sys.stderr)
'
  exit 1
fi
printf '%s' "$DETAIL" | python3 -c '
import json, sys
verdict = json.load(sys.stdin)["report"]["verdict"]
print("verdict:", verdict["overall"])
print("provenance:", verdict["provenance"])
for criterion in verdict["criteria"]:
    print("  %s %s vs %s: %s" % (criterion["label"], criterion["metric"], criterion["arm"], criterion["status"]))
'
echo "$REPORT_ID" > "$TESSERA_STATE_DIR/report_id"

Download playbooks/design-partner-runbook/02-evaluate-and-read-verdict.sh — this exact file runs in CI on every change (hosts shown are for this environment).

The verdict has one row per criterion and an overall result: pass, fail or incomplete. Missing evidence is not_evaluated with a reason, never a pass or a fail. See Protocols & verdicts.

4. Run the lead-time backtest

Only for protocols that declare lead-time metrics:

03-backtest.sh
#!/usr/bin/env bash
# Design-partner runbook, step 4: the lead-time backtest under the same protocol.
# Optional: only for protocols that declare lead_time_metrics. Set TESSERA_MODEL_ID.
#
# The Idempotency-Key is derived from the model, the protocol and the seed: the
# same backtest is a safe retry, and another seed is another backtest.
set -euo pipefail

: "${TESSERA_API_BASE:?e.g. https://api.staging.tessera.zone/v1}"
: "${TESSERA_API_KEY:?the tsk_ key of the partner project}"
: "${TESSERA_MODEL_ID:?the trained model, mdl_...}"
: "${TESSERA_STATE_DIR:?set by the runner; ids are handed between steps}"
AUTH=(-H "Authorization: Bearer $TESSERA_API_KEY")
PROTOCOL_ID="$(cat "$TESSERA_STATE_DIR/protocol_id")"
SEED="${TESSERA_BACKTEST_SEED:-7}"
MAX_POLLS="${TESSERA_MAX_POLLS:-720}"   # 5 s apart: one hour by default

fail() { echo "$1" >&2; exit 1; }

KEY="docs-partner-backtest-$(printf '%s:%s:%s' "$TESSERA_MODEL_ID" "$PROTOCOL_ID" "$SEED" | python3 -c '
import hashlib, sys
print(hashlib.sha256(sys.stdin.buffer.read()).hexdigest()[:24])
')"
BACKTEST="$(curl -sS --fail-with-body -X POST "$TESSERA_API_BASE/backtests" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $KEY" \
  -d '{"protocol_id": "'"$PROTOCOL_ID"'", "model_id": "'"$TESSERA_MODEL_ID"'", "seed": '"$SEED"'}')" \
  || fail "starting the backtest failed: $BACKTEST"
BACKTEST_ID="$(printf '%s' "$BACKTEST" | python3 -c 'import json,sys; print(json.load(sys.stdin)["backtest_id"])')"

STATUS=""
for _ in $(seq 1 "$MAX_POLLS"); do
  DETAIL="$(curl -sS --fail-with-body "$TESSERA_API_BASE/backtests/$BACKTEST_ID" "${AUTH[@]}")" \
    || fail "reading backtest $BACKTEST_ID failed: $DETAIL"
  STATUS="$(printf '%s' "$DETAIL" | python3 -c 'import json,sys; print(json.load(sys.stdin)["backtest"]["status"])')"
  if [ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ]; then break; fi
  sleep 5
done
echo "backtest: $STATUS"
if [ "$STATUS" != "succeeded" ]; then
  printf '%s' "$DETAIL" | python3 -c '
import json, sys
row = json.load(sys.stdin)["backtest"]
print("reason:", row.get("reason", "still running after the polling bound"), file=sys.stderr)
'
  exit 1
fi
printf '%s' "$DETAIL" | python3 -c '
import json, sys
body = json.load(sys.stdin)["backtest"]
print("fingerprint:", body.get("fingerprint"))
'
echo "$BACKTEST_ID" > "$TESSERA_STATE_DIR/backtest_id"

Download playbooks/design-partner-runbook/03-backtest.sh — this exact file runs in CI on every change (hosts shown are for this environment).

5. Archive and publish

Archive the protocol, the evaluation report and the backtest report together with their fingerprints. Then write the results with the published-results template (apps/docs/templates/published-results.md in the Tessera repository):

the provenance class in the headline, the class of the protocol;
one fingerprint reference per number, pointing at the archived report that holds it;
the protocol id, fingerprint and declaration time;
the verdict as it came back, including a fail or incomplete;
a recompute section that repeats the calls above with the archived ids and seeds.

Run make docs-lint before publishing. The claims lint checks the headline, the fingerprints and the wording for the document's class.

Steps 3 to 5 in one command

tessera-milestone-partner, in tools/milestones, runs steps 3 to 5 against the partner project and stops at the first failure:

04-judge-archive-publish.sh
#!/usr/bin/env bash
# Design-partner runbook, steps 3 to 5 in one command: the packaged driver
# (tessera-milestone-partner) judges the model against the registered protocol,
# runs the lead-time backtest when the protocol declares lead-time metrics,
# archives the protocol and the reports by fingerprint and writes the
# published-results document. Set TESSERA_MODEL_ID and TESSERA_REPO_ROOT (a
# checkout of the Tessera repository).
#
# There is no flag for a threshold, a margin or a provenance class: thresholds
# come from the registered protocol, and a run against our hosted deployment
# reports at most the design-partner class.
set -euo pipefail

: "${TESSERA_API_BASE:?e.g. https://api.staging.tessera.zone/v1}"
: "${TESSERA_API_KEY:?the tsk_ key of the partner project}"
: "${TESSERA_MODEL_ID:?the trained model, mdl_...}"
: "${TESSERA_REPO_ROOT:?a checkout of the Tessera repository}"
: "${TESSERA_STATE_DIR:?set by the runner; ids are handed between steps}"
PROTOCOL_ID="$(cat "$TESSERA_STATE_DIR/protocol_id")"
REPORT_DIR="${TESSERA_PARTNER_REPORT_DIR:-$TESSERA_STATE_DIR/partner}"

uv run --project "$TESSERA_REPO_ROOT/tools/milestones" tessera-milestone-partner run \
  --target staging \
  --protocol-id "$PROTOCOL_ID" \
  --model-id "$TESSERA_MODEL_ID" \
  --report-dir "$REPORT_DIR"

Download playbooks/design-partner-runbook/04-judge-archive-publish.sh — this exact file runs in CI on every change (hosts shown are for this environment).

It writes nothing but the evaluation and the backtest. It never loads data.
It reads every threshold, margin, metric and the alert budget from the registered protocol. It has no flag for them.
It reports the class of the protocol, and refuses a protocol whose class is stronger than the run allows (provenance_exceeds_context): a run against our hosted deployment reports at most design-partner, a run against a local stack at most fixture.
It refuses a design-partner protocol when the project's data is Tessera's own fixture data (fixture_data_claims_partner_class): a project named like Tessera's harness runs, or a source carrying a generator's ground-truth manifest, the sample archive or a fixture bucket. Fixture data is judged only under a fixture protocol.
A refusal from the API is reported as it came back, for example protocol_after_data.
Next to report.json it writes archive/ (the protocol and the reports, each named by its fingerprint), published_results.json and rerun.json, the exact command that repeats the run. The evaluation and the backtest are content-addressed, so repeating the command reads the same reports.

A fail or incomplete verdict does not fail the command: it is archived and published as it came back.

If something goes wrong

Symptom	Cause	What to do
`protocol_after_data`	The dataset build is older than the protocol declaration.	Register a new protocol, rebuild, and state the order in the document.
`fixture_data_claims_partner_class`	The project's data is Tessera's own fixture data.	Run it under a `fixture` protocol; partner classes need the partner's own data.
`provenance_exceeds_context`	The protocol claims a stronger class than the run allows.	Judge it in the context its class belongs to; never re-declare a fixture run as `design-partner`.
Verdict `incomplete`	Evidence is missing for a criterion, such as a reference that failed.	Publish it as `incomplete` with the reasons. Do not rerun until it passes.
Verdict `fail`	The model did not clear a margin.	Publish it as `fail`. A new protocol is a new comparison, not a correction.
The partner's data is not ready	The project has no approved contract or too few labelled entities.	Stop. Nothing is run against a stand-in dataset under the partner's name.

Before you start​

1. Register the protocol​

2. Build the dataset and train​

3. Judge the model against the protocol​

4. Run the lead-time backtest​

5. Archive and publish​

Steps 3 to 5 in one command​

If something goes wrong​