Delivery integrity: purposes, suppression and holdouts
Three controls decide which entities an output may reach. One filter enforces them, and no API accepts a bypass.
In the console, Settings, then Suppression & purposes, manages the purpose registry and the
suppression sets. Saving the registry, creating or editing a set, and adding or removing entries each
open a confirmation that shows the current value beside the new one. Deleting a set instead asks you to
type its id, and states that its entities stop being suppressed within about 60 seconds. A purpose that
sets, experiments, slots, keys or roles still use cannot be removed, and the page names what uses it.
The page repeats which outputs apply suppression today, that exports, alerts, audiences and reads run
under the general purpose (so a set must cover general or all purposes to affect them), that a
change reaches outputs within about 60 seconds, and that the purpose registry is shared by the live and
test environments. An entry batch holds at most 1,000 entities with ids of at most 256 characters, and a
set at most 100,000 entries; the page warns when a batch would pass that limit. The holdout experiment
readout is not shown in the console yet.
What is enforced today
| Output | Suppression sets | Holdout experiments | If suppression state cannot be read |
|---|---|---|---|
State reads (GET …/state) and batch score reads (POST …/scores:batch) | A suppressed entity reads exactly like an entity with no state. | Held-out entities stay visible (analysts need the control group). | The request fails with 503 unavailable and is not billed. |
Exports (POST …/exports, every destination kind) | Entities in a set covering the slot's purpose, else general (or ["all"]), are left out, and the manifest counts them. | Held-out entities of active experiments on the export's slot for that purpose are left out, and the manifest counts them. | The export fails with unavailable and writes no manifest; parts already written stay in the bucket, unreferenced. |
Alert webhooks (alert.raised, alert.resolved, entity.went_quiet, score.band_changed) | The alert of a suppressed entity is withheld; its episode still exists for triage. Suppression is checked on every event of an episode, so an entity suppressed after its alert was sent has the resolve withheld. | The alert of a held-out entity is withheld when an experiment for the slot's purpose (else general) is active on the alert's slot. This is decided once per episode, when its opening webhook is published, and the episode's resolve follows that verdict, so an experiment that starts or ends mid-episode never leaves a receiver with a raised alert that is not resolved. | The alert stays queued and is retried with back-off (30 seconds, doubling to 15 minutes); nothing is sent undecided. |
| Audiences (preview counts and samples, materialized member artifacts, deliveries and lookalike artifacts) | Suppressed entities are left out, and the audience manifest counts them. | Held-out entities of active experiments on the audience's slot for the slot's purpose (else general) are left out. | The output fails with a typed suppression_unavailable (a preview answers 503); nothing unfiltered is written or delivered. |
Per-entity webhooks (score.updated, segment.migration, marker.fired) | The event of a suppressed entity is not sent. | The event of a held-out entity is not sent when an experiment for the slot's purpose (else general) is active on the slot, including entities in the control group. The withholding is counted, not recorded per entity. | The event is never sent unfiltered. score.updated and segment.migration are dropped: they are already best-effort (throttled; a dropped one is not queued, and it does not use up the entity's throttle window). marker.fired is durable, so it stays in its outbox and is retried until the filter can evaluate; only a suppressed or held-out verdict removes it. Every drop is counted and logged. The decision is made once, when the event is published: retries and replays of an event already sent are not filtered again (HD-05). |
| Destinations (the PostHog person-property writer) | Suppressed entities are left out of every batch, and the filter runs again before every send attempt. The delivery receipt records every set that covered the destination's purpose. | Held-out entities of active experiments on the destination's slot, for the destination's purpose (else general), are left out and counted on the receipt. | Nothing is sent: the batch waits and is retried, and the receipt records nothing undecided. |
| Other reads: entity listings and event timelines, segment members, feature views, trajectories | Not applied. | Not applied. | — |
Reads, exports, alerts, per-entity webhooks and audiences run under their slot's purpose, else general; a destination runs under the purpose it declares, else its slot's, else general. A project with
no suppression sets and no holdout experiments gets exactly the output it got
before suppression existed. Export, alert, audience and destination deliveries
do not record control manifests yet, so the control group they withheld is not
recorded. An experiment with purpose general therefore also withholds the alerts
of its control group; to scope an experiment away from a slot's alerts, give the
slot a different purpose (PUT /v1/slots/{slot}/purpose).
Purposes
A project's purpose registry (GET/PUT /v1/purposes) names why outputs
are produced: for example general, marketing or responsible-gaming.
Every project starts with general and may register up to 16 lowercase-kebab
names. The name all is reserved. The registry is shared by the project's
live and test environments.
An output's effective purpose is:
- the purpose its export or destination declares, otherwise
- its slot's purpose, otherwise
general.
Reads declare no purpose, so they run under their slot's purpose, else
general. Purpose-bound suppression sets and holdouts apply under the
effective purpose.
Slot purposes and purpose allow-lists
Three pairs of operations manage purpose bindings. Each uses its own small record, so the existing API key, custom role and slot responses are unchanged:
| Operations | Record | null means | Names checked against |
|---|---|---|---|
GET/PUT /v1/slots/{slot}/purpose | {"purpose": "marketing"} | the slot declares none: its outputs run under general | the project's registry |
GET/PUT /v1/projects/{project_id}/api-keys/{key_id}/purposes | {"purposes": ["risk"]} | unrestricted (every key starts this way) | the key's project registry |
GET/PUT /v1/orgs/{org_id}/roles/{role_id}/purposes | {"purposes": ["risk"]} | unrestricted (every custom role starts this way; built-in roles are always unrestricted) | name syntax only (a role spans the organization) |
Key and role allow-lists are managed from console sessions only. A person whose own role is restricted can only narrow an allow-list within their own purposes; they can never lift a restriction. Every change is audited with the previous and the new value.
What is enforced. The slot purpose sets the effective purpose of reads,
exports, alerts, per-entity webhooks, audiences and destinations. A changed
slot purpose reaches exports, alerts, audiences and destinations on their next
run, and serving reads and per-entity webhooks within 5 seconds (each serving
replica and scoring writer caches slot purposes for up to 5 seconds). If the slot purpose cannot be read, the output fails with
503 unavailable rather than running under a guessed purpose.
Every output also checks the caller's allow-list. An output whose effective
purpose is outside it fails with 403 permission_denied:
details[0].reason is purpose_not_allowed and details[0].purpose names
the purpose. Nothing is created, changed or returned, and no signed URL is
issued. The check covers:
- reads: entity state, markers, bands, batch scores, observations, evidence, ranked and history reads, feature views and segment members;
- exports, warehouse exports and export schedules, including reads of their records and signed artifact URLs;
- audiences (create, update, preview, materialize, materializations) and lookalike jobs, including their signed URLs;
- alert rules (create, update and the packaged went-quiet install) and alert episodes (list, get and disposition);
- destinations (create, update, enable and backfill);
- a holdout experiment's control manifest, under the experiment's purpose.
- the committed event feed, under the fixed purpose
event-feed: feed reads (a long-poll is checked again when it wakes), stats, event exports with their part anddeletionsdownload URLs, and the connected-application heads watch. A restricted allow-list needsevent-feedregistered in the project, like any other name. - the other reads that return raw event payloads, under the same fixed
purpose
event-feed: the entity timeline, the evidence timeline (which also needs its slot's purpose), dead-letter items, the source sample, sync rejections, snapshot-sampled mapping previews and requests for projection exports colored by event attributes. Contract profiles, an upload's schema preview, a failed sync page's keyset position, the trajectory inspector's original values and the artifact URL of a projection colored by event attributes are left out instead of refused (see Other reads of raw events). - segment cards and their comparison, under the slot's purpose (they carry exemplar entities).
An API key's allow-list reaches requests within 5 seconds, as other key changes do. A console session uses its effective role in the project (a project role assignment, else its organization role); a change to that role's allow-list also reaches requests within 5 seconds.
Background outputs act for the credential that set them up and check its allow-list every time they run, not only when they were created:
| Output | Acts for | Checked |
|---|---|---|
| Export and projection runs | the requester | before any data is read |
| Event exports | the creator (an export that names none is refused where the organization uses purpose restrictions) | before the snapshot is pinned, parts are written or rows are metered |
| Scheduled exports | the schedule's creator, or whoever last updated it | at every run |
| Manual audience materializations | whoever triggered the run | before any output |
| Scheduled (an audience's own interval schedule) and triggered audience materializations | the audience's creator, or whoever last updated it | before any output |
| Destination deliveries | the destination's creator, or whoever last enabled it | before every batch and every retry |
| Alert webhooks | the rule's creator or installer, or whoever last updated it | before every webhook is published |
Records created before allow-lists existed name no creator and keep running as before.
One environment per key. An API key holds exactly one environment, so an output that acts for a key runs only in that key's environment. An output of the other environment that acts for it (a configuration a key of one environment saved on the other environment's slot, which requests no longer allow) is refused as if its allow-list were empty, whether or not the organization uses purpose restrictions. Outputs that act for a console session are not affected, and an application is checked against the environments it is registered for.
Key rotation and offboarding. If an organization uses purpose
restrictions anywhere (any API key or custom role has an allow-list, or any
slot declares a purpose), a credential that no longer stands (a revoked key,
a removed or deactivated member, a deleted custom role) is refused as if its
allow-list were empty. Revoking a key or removing a member therefore stops
the outputs that act for it. Before you rotate or offboard, re-bind each
output to a current credential: update the schedule, audience or alert rule,
or call :enable on the destination, with that credential, then run a
backfill where needed. An organization that uses no purpose restrictions
keeps today's behavior: rotation and offboarding never stop its outputs.
Whether an organization uses purpose restrictions is re-read at most every
60 seconds.
A refusal:
- fails an export, projection or audience materialization run with
permission_denied(purpose_not_allowed), and an event export witherror.codepurpose_not_allowed; - disables a destination (
status: disabled,disabled_reason: null) and drops its queued and pending deliveries. The organization audit log records why: adestination.auto_disableevent on the destination withdetails{"reason": "purpose_not_allowed", "purpose": "<purpose>"}. Re-enable it with a credential that may use the purpose (that credential then acts for the destination) and run a backfill; - withholds an alert webhook. Its episode is still recorded.
When the allow-list cannot be read, an export, projection or audience
materialization run fails with unavailable; run it again. A destination
batch and an alert webhook wait and are retried. Nothing runs unchecked.
Not covered yet: lookalike jobs are checked when they are requested but
not again while they run; score.band_changed webhooks have no creator and
apply only suppression; audience KPIs and refresh jobs return aggregates
only and are not checked.
Per-entity webhooks (score.updated, segment.migration, marker.fired)
act for the credential that switched them on: the last writer of the slot's
monitor config (score.updated), the last creator or updater of an active
marker (marker.fired), and whoever requested the segment build behind the
slot's current registry (segment.migration). Switching one on needs the
slot's purpose (403 purpose_not_allowed, nothing saved). Before each event
leaves, that credential's allow-list is checked again (cached at most 5
seconds): an event it no longer allows is dropped. When the credential cannot
be read, score.updated and segment.migration are dropped and
marker.fired is held and retried; none is ever sent under general. An
event with nothing recorded behind it (for example a marker deleted after it
fired) is dropped. A configuration saved before this check existed has no
recorded credential, and it fails closed: in an organization that uses purpose
restrictions its events are dropped (purpose_not_allowed) until the configuration is saved
again by a credential that may use the slot's purpose. In an organization
that never configured purposes nothing changes, because nothing is
restricted.
The webhook endpoint that receives an entity-level event acts for its
subscriber too: the credential that created the endpoint, or that last updated
it with a body carrying url or event_types (changed or repeated
unchanged; a body with only description keeps the subscriber).
Entity-level events are score.updated, segment.migration, marker.fired,
alert.raised, alert.resolved, entity.went_quiet and
score.band_changed; the slot outputs audience.synced, audience.failed,
export.completed, export.failed and the by-name-only types are checked the
same way. Subscribing to one of them, or to "*", needs
general and every purpose a slot of the project declares (403
purpose_not_allowed, nothing saved). Each event is checked again against
the subscriber's current allow-list and the slot's purpose when it is queued
for the endpoint and before every send, retries and replays included: an
endpoint whose subscriber was narrowed, revoked, removed or deactivated gets
no new deliveries, and its pending ones are abandoned. An endpoint that an
API key subscribed receives only the entity-level events and slot outputs of
that key's environment (a tsk_test_ key's endpoint receives test events, a
live key's endpoint live ones), and the same holds for the events whose data
names an environment: audit.recorded, gate.failed (whose environment is
the model's), slot.activated, slot.rolled_back and refresh.*. One that
names none, such as a project-level audit record, reaches every endpoint,
except a gate.failed that names none (emitted by an earlier version), which
reaches no API key's endpoint.
The other environment's events, including those queued before the
endpoint's subscriber changed environment, are not delivered to it and do
not count as refusals: the environment is checked before the purpose. An
endpoint that a console session subscribed receives both environments'
events. To receive an environment's events again, save the endpoint's url
or event_types with a credential of that environment, or from the console.
When an endpoint
starts being refused, its organization gets one
webhook_endpoint.subscriber_refused audit event (at most once an hour per
endpoint; also delivered to audit.recorded subscribers), naming the endpoint
and the event type. An endpoint saved before this check existed has no
recorded subscriber: in an organization that uses purpose restrictions it is
refused until it is updated with its url or event_types (the same values
are enough) by a credential allowed those purposes; elsewhere nothing changes.
If the subscriber's purposes cannot be read, the event is held and retried
later, never sent unchecked.
Whether an organization uses purpose restrictions (any slot purpose, any key or custom role with a purpose list) is cached for up to 60 seconds, so for about a minute after an organization's first restriction an unrecorded configurer or subscriber can still be treated as unrestricted. Allow-lists themselves are read fresh for every endpoint check (and cached at most 5 seconds for the per-entity publish check).
Purposes are referenced by suppression sets, holdout experiments,
destinations, slots, live API keys and custom roles. A custom role's list
references the purposes in every project of its organization. A destination's
purpose must be a registered name: an unregistered one is 400
invalid_argument. A purpose that a reference still names cannot be removed:
the request fails with failed_precondition, and the error lists the
referencing resources. A revoked key no longer holds a reference.
Suppression sets
A suppression set (/v1/suppression-sets) excludes entities from every
output of the purposes it covers. Set purposes to ["all"] to cover every
purpose. Like the purpose registry, sets belong to the project and apply in
both its live and test environments. So a request acting in the test
environment (a tsk_test_ key, or the console switched to test) may only
add to them: it can create sets, add entries and purposes, edit a
description and delete an empty list set, but removing an entry, deleting any
other set, removing a purpose from a set or changing its predicate is refused
(permission_denied, environment_not_allowed), because it would lift a live
self-exclusion too. Use a live key, or switch the console to live. An empty
list set withholds nothing, so deleting one lifts no exclusion. The limit of
32 sets is per project, shared by both environments, and sets carry no
environment: deleting an empty list set from test can remove a set a live
integration just created, whose next entries:add then answers 404 (the
deletion is audited in the test environment).
-
List sets hold explicit entity refs. Change them with
entries:addandentries:remove, up to 1,000 refs per batch and 100,000 entries per set. In the console, a list set also loads entity ids from a file, with the entity type chosen once for the whole file. The operator says how the file names its ids:- One id per line (suggested for a text file): each whole line, without its line ending, is one id, kept exactly as written, commas and spaces included. A line of only spaces is refused.
- First column of a CSV (suggested for a
.csvfile): the first column of each row, quotes honored. The summary says how many rows had more columns and how many ids lost surrounding spaces.
A first row of
idorentity_idis treated as a header. Ids over 256 characters and quoted ids that are not closed on their line are refused, and repeated ids are sent once. The file is sent as consecutive batches of up to 1,000, one at a time. Each batch has its ownIdempotency-Key, so retrying a failed batch never double-counts. A failed batch stops the load with its reason and request id, and Resume sends it again. A refusal for the role offers no Resume, since it would be refused again. -
Predicate sets evaluate the shared predicate language against each entity's state (scores, signals and segment).
A project may have up to 32 sets. Every change is audited in the same transaction as the change itself. The audit record of an update shows the purposes before and after, and an entries batch records the entity-list fingerprint and count.
Example: a responsible-gaming team keeps a list set rg-self-excluded with
purposes ["all"]. A self-excluded player then reads exactly like an entity
that has no state, on state reads and batch score reads, while their
neighbours read normally. Exports, alert webhooks and audiences leave the player
out too, and so do per-entity webhooks and destinations.
If suppression state cannot be read, the filter fails closed:
- a read request fails with
503 unavailableand is not billed; - a delivery batch is blocked.
Unfiltered data is never returned. A change applies at once on the serving replica that handled it, and on every other replica usually within 30 seconds and always within 60.
Holdout experiments
A holdout experiment (/v1/slots/{slot}/experiments) withholds a
deterministic share of a slot's entities from deliveries for one purpose.
-
Assignment is a pure function of the experiment's salt and the entity. Take
sha256(salt ‖ "{project_id}#{type}#{id}"), read its first 8 bytes as an unsigned big-endian integer, and divide by 2⁶⁴. The entity is held out when the result is belowholdout_fraction(at most 0.5); the comparison is exact. The same entity always lands in the same group. -
The salt never leaves the platform; responses carry only its fingerprint.
-
Salt, fraction and purpose are immutable. There is no update operation: to change any of them, conclude the experiment and create a new one.
-
A slot holds at most one active experiment per purpose. Every withheld entity therefore belongs to exactly one control group. To start a new experiment for the same slot and purpose, conclude the active one first.
-
Reads still show held-out entities, because analysts need to see the control group.
-
Control manifests. Every delivery path that runs under an experiment's purpose records its cycles as control manifests: the entities it withheld (the control group) and the entities it confirmed it delivered (the treated group). Large cycles are recorded in parts of up to 5,000 entries, and a cycle is recorded once (a retry of the same cycle is a no-op). Read them, oldest first, with
GET …/experiments/{id}/control-manifest(up to 20 records per page; the list shows the parts as stored, so a cycle that is still being written, or whose writer stopped, shows fewer parts than itspartsfield, and the readout skips such a cycle). Each path defines its cycle:Path Cycle Treated means Recorded Export one export (cycle id = the export id, time = its start) the rows written when the export has delivered; a recording that cannot be written is retried, then counted and logged, and never fails the export Audience one materialization (id = the materialization id, time = its creation) the members delivered when every destination has succeeded, before the materialization turns terminal; a failed materialization records nothing, because one that delivered to some destinations and not others is a partial exposure. Lookalike jobs record nothing: they rank candidates and deliver only the top ones, so their candidates are not an exposed population Destination two moments per delivery receipt: the cut ( …:cut, the withheld entities) and the vendor accepting the batch (…:sent, the delivered entities plus any entity the attempt's re-filter withheld)the entities the vendor accepted before the transaction that makes each moment durable commits, so a transient storage fault rolls it back and it is retried (for …:sentthat is after the vendor accepted, so the same events are sent again), at most five consecutive times per cycle, after which the cycle is given up and counted; a permanent fault (no permission yet, no key) is counted and the batch is delivered; a batch that never succeeds records no treated entityAlert webhooks one drain window of the durable outbox (cycle id from the settled rows) the alerts published before the drain commits, so a fault aborts the window and the same rows are drained again; only an episode's opening is a verdict, its close adds nothing score.updated,segment.migration,marker.fireda best-effort recording window of at most 60 seconds or 5,000 entities per project, environment, slot and purpose the events that passed the filter for publication (the verdict is made before the event is published, and these events have no durable copy) window by window, see below The inline events have no durable copy, so their recording is best effort by design. Each window is written as ONE cycle that holds BOTH arms (the treated deliveries and the withheld entities), so a crash, or a window that cannot be written, loses a whole window symmetrically for both arms: the arms shrink together and nothing is biased.
control_manifest_cycles_recorded_total,control_manifest_entities_recorded_total(by path and arm) andcontrol_manifest_record_failures_totalcount what is recorded and what is not.A cycle's entries are spooled to local disk past 32 MiB (one spool per arm) and the records are built and digested one part at a time, so recording does not hold a cycle in memory. The two arms are interleaved across a cycle's parts, and a reader skips a cycle whose stored parts are fewer than the cycle has, so a cycle abandoned mid-write is never read half-way. The spool is bounded: a cycle holds at most 1,000,000 entries and 96 MiB, and all cycles of a process at most 192 MiB (the serving and scoring pods have a 256 MiB scratch disk, which other users share, so the bound is a ceiling and not a reservation); a cycle over a bound, or whose spool cannot be written, is dropped whole, both arms, and counted (
reasontoo_largeorspool).A recording that cannot be written never wedges a delivery. A permanent fault (the destination writer's storage role is not applied yet, the entity-hash key or its library is missing, a cycle already recorded with other content) is counted under
control_manifest_record_failures_totalwith itsreason(permission,key,config,conflict,unconfigured) and the delivery proceeds; the cycle's two arms are lost together. A transient storage fault (storage) is retried, bounded for the paths that record before they commit: after five consecutive failures of one destination (one project for alerts) the cycle is given up (abandoned) and recording for it cools off for five minutes (cooloff). Other reasons:too_large,spool,accumulate,partial_delivery(a materialization with a failed destination),droppedandadd. An empty readout can therefore mean that no cycle ran or that recording was unavailable: the counters tell which. -
Erasure. A control manifest holds an entity's ref and
entity_hash, so deleting the entity removes it from every recorded part: the entity-deletion worker's export-artifact sweep rewrites the parts that name it (every other entry and the part layout are kept byte for byte), deletes every superseded generation and re-seals each cycle's claim. The deletion stays open while an export or audience materialization that started before it is still running. A recording made after the erasure leaves the entity out: each recorder checks the erasure tombstones when it writes. A readout over a rewritten manifest is correct for the remaining entities. Suppression entries are kept on erasure on purpose, so that an exclusion outlives the person's data.
Incrementality readout
The incrementality readout (GET …/experiments/{id}/readout?label_id=&start=&end=)
reports the treated and held-out outcome rates of one binary label, their
difference and a bootstrap confidence interval. It is computed only from the
recorded control manifests, never from the salt, and it returns aggregates only:
no per-entity rows, scores or recommendations. Holdouts measure outcomes; no API
recommends or chooses a treatment. Like the control manifest, it is an output of
the experiment's purpose: a principal whose purpose allow-list excludes that
purpose gets 403 permission_denied (purpose_not_allowed).
Every step is deterministic, so a reader can reproduce a number from the
recorded facts. Let M be the label's maturation_days.
- Cycles. Only manifest records whose
computed_atlies in[start, end)count. - Arms. The treated arm is the union of those records'
treated_entries, the holdout arm the union of theirentries, restricted to the label's entity type and counted once per entity. An entity that appears in both arms is excluded from both; the exclusion is logged and counted in a metric, and the response has no field for it. - Anchor. An entity's anchor
Ais thecomputed_atof the earliest in-window record that names it. - Outcome: one fixed horizon per entity. The label's observations are read
point-in-time at
as_of = end. An observation's label-anchor time is itsobserved_atminusMdays (the realized quality join's rule:observed_at, neverreceived_at). An entity's outcome is the value, 0 or 1, of its FIRST binary observation (ordered byobserved_at, then observation key) whose label-anchor time is in[A, A + M days)and whoseobserved_atis beforeend. Every counted entity is therefore followed for the same one period whenever it was anchored, so a delivery retry that delays an anchor cannot shorten its follow-up. - Exclusion, not zero. An entity with no such observation has no matured
outcome in the window and is EXCLUDED from its arm. It is never counted as
0.
nin the response is the number of counted entities of the arm. A label whoseMis 0, or as long as the window or longer, matures nothing inside the window, and a label with no observations has no outcome for any entity, so those requests are refused (step 8), never reported as 0 against 0. - Rates.
treated.rateandholdout.rateare the mean outcome of the counted entities of each arm, anddelta = treated.rate - holdout.rate. - Interval.
ci95is the percentile 95% interval (2.5th and 97.5th percentiles) of 1,000 bootstrap resamples. Each resample draws the counted entities of each arm independently with replacement, the treated arm first and then the holdout arm, and takes the difference of the two resampled rates. The generator is seeded with the first eight bytes (big-endian) of the SHA-256 of the length-prefixed (8-byte big-endian length, then UTF-8) valuesproject_id, environment,experiment_id,label_id, andstartandendas microseconds since the Unix epoch. Entities are ordered by id before drawing. For the same recorded manifests and the same label observations, the same request therefore always returns the same interval. An observation that was received afterendbut whoseobserved_atis before it IS included, so the same request can return a different readout once such late observations arrive. - Minimum and cap. Each arm needs at least 30 COUNTED entities, that is
after the exclusion in step 5, and may hold at most 100,000 entities before
it. The manifests are listed from
start, read one record at a time and folded into one entity-to-anchor map per arm, and the bootstrap resamples in blocks sized to the arm, so memory follows the arm size. Measured on the test runner with one observation per entity (peak resident memory of the whole process, which includes about 115 MiB of interpreter and libraries, and time): 25,000 entities per arm 199 MiB and 2.6 s; 50,000, 232 MiB and 5.3 s; 100,000, 293 MiB and 10.7 s; 200,000, 413 MiB and 20 s; 250,000, 470 MiB and 25 s; 400,000, 711 MiB. The cap of 100,000 keeps the readout's own footprint near 180 MiB inside the 512 MiB serving pod. It is a limit of this endpoint, not a data limit: narrow the window to read a larger experiment.
The readout answers failed_precondition with reason readout_unavailable
when the label is not binary, no manifest was recorded in the window, an arm is
empty, an arm is above the cap, a recorded document or stored object cannot be
read (it is gone or does not parse), no label store is configured, or fewer
than 30 entities per arm have a matured outcome ("not enough entities with a
matured outcome in the window"). The message says which. A storage fault that
may heal on retry (a provider error or a transport fault, in the manifest store
or the label store alike) answers 503 unavailable. A missing or foreign slot,
experiment or label is not_found. A readout reports that no manifest was
recorded when no delivery cycle ran under the experiment's purpose in the
window, or when recording was unavailable (see control_manifest_record_failures_total).
A label whose point-in-time read would hold more than 400,000 observations or 64 MiB of stored data is refused with its own message before any of it is read: the readout holds the label's observations in memory, and this bound is sized like the arm cap (two observations per entity of the largest allowed arms). The bound covers the whole label, not the two arms; reading only the entities of the arms is a later label-store improvement.
See also: State reads and Exports.