Skip to main content

dataset_files

GET /dataset-file-retention​

Get the project's dataset build file retention

The project-wide dataset build retention setting (WP-O7 req 4), one per project and covering both environments, beside the model file retention. While retention_days is set, the files of every idle build of a registered kind are scheduled for deletion retention_days after it became idle; null is off. Before any setting exists it answers retention_days: null, version: 0, updated_by: null and updated_at: null. Sessions scope the project with X-Tessera-Project + X-Tessera-Environment; API keys are project-bound already. Requires the read scope.

Operation ID: getDatasetFileRetention · Service: gateway-api

Authentication​

  • apiKey (http (bearer)) — Project API key (003 §2): tsk_live_… or tsk_test_… in Authorization: Bearer. Scoped to exactly one project + environment; scopes are a subset of ingest, read, admin. Keys are stored hashed and shown once at creation.
  • consoleSession (http (bearer)) — Console session (003 §2): Identity Platform JWT. Data-plane calls add X-Tessera-Project + X-Tessera-Environment headers, validated against org membership.

Responses​

StatusSchemaDescription
200DatasetFileRetentionThe project's dataset build retention setting.
defaultErrorEnvelopeError response (003 §1). Every non-2xx response uses this envelope; the HTTP status matches error.code. Rate-limited requests get 429 resource_exhausted with retry_after_seconds and a Retry-After header; the X-RateLimit-* headers ride on EVERY response, success or error (003 §1; plan-driven per API key + endpoint class — WP-B2). A 503 unavailable carries retry_after_seconds too for a lock-wait timeout (SQLSTATE 55P03), a rule preview at capacity (preview_capacity_exhausted), or a label preview during which the erasure of every entity holding its censor clock began (label_preview_clock_entity_erased): how long to wait before retrying.

PUT /dataset-file-retention​

Set the project's dataset build file retention

Sets the project-wide dataset build retention (WP-O7 req 4), for both environments: retention_days from 1 to 365, or null to turn retention off. Anything else is 400 invalid_argument with a field_violation on body.retention_days. The put locks the project's setting, created on first use. With expected_version, another current version (0 before any setting) is 400 failed_precondition, details[0] a DatasetFileRefusalDetail with reason version_mismatch. An identical value writes nothing: the same version, no audit event. A changed value stores version current + 1, updated_by (the caller's actor id: a user, an API key or application:\<id\>) and updated_at, and records dataset_file_retention.update (retention_days, previous_retention_days, version) in the same transaction. The put itself deletes nothing: within one cycle the purge worker schedules every idle build of a registered kind retention_days after it became idle and moves the due time of rows that have not started deleting, and turning retention off returns the rows retention scheduled to retained (explicit purge requests stay). With the model file retention at the same value, a build is deleted in the same worker cycle as its last model, or the next. Requires an org role of admin or above, or the admin scope. The setting governs both environments, so a request acting in the test environment (a test API key, or a console session scoped to test) may only keep files longer: turning retention on or shortening it is refused with 403 permission_denied, details[0] \{type: permission_violation, reason: environment_not_allowed\}, and the setting is unchanged (HD-851); a longer value or null is accepted. An application acting in a project it owns keeps both directions (WP-A9; WP-O7 req 20's partner flow turns both retentions on).

Operation ID: putDatasetFileRetention · Service: gateway-api

Authentication​

  • apiKey (http (bearer)) — Project API key (003 §2): tsk_live_… or tsk_test_… in Authorization: Bearer. Scoped to exactly one project + environment; scopes are a subset of ingest, read, admin. Keys are stored hashed and shown once at creation.
  • consoleSession (http (bearer)) — Console session (003 §2): Identity Platform JWT. Data-plane calls add X-Tessera-Project + X-Tessera-Environment headers, validated against org membership.

Request body​

DatasetFileRetentionPut

Responses​

StatusSchemaDescription
200DatasetFileRetentionThe project's dataset build retention setting after the put.
defaultErrorEnvelopeError response (003 §1). Every non-2xx response uses this envelope; the HTTP status matches error.code. Rate-limited requests get 429 resource_exhausted with retry_after_seconds and a Retry-After header; the X-RateLimit-* headers ride on EVERY response, success or error (003 §1; plan-driven per API key + endpoint class — WP-B2). A 503 unavailable carries retry_after_seconds too for a lock-wait timeout (SQLSTATE 55P03), a rule preview at capacity (preview_capacity_exhausted), or a label preview during which the erasure of every entity holding its censor clock began (label_preview_clock_entity_erased): how long to wait before retrying.

GET /dataset-files​

List the dataset build file lifecycles of the environment

The dataset build file lifecycle rows of the resolved project and environment (WP-O7 reqs 3 and 12), newest updated_at first, cursor-paginated (003 §1). A build gets a row when it becomes idle or when its purge is requested; a build without a row is retained and is read with getDatasetFiles. The optional status filter takes one DatasetFileStatus. Requires the read scope.

Operation ID: listDatasetFiles · Service: gateway-api

Authentication​

  • apiKey (http (bearer)) — Project API key (003 §2): tsk_live_… or tsk_test_… in Authorization: Bearer. Scoped to exactly one project + environment; scopes are a subset of ingest, read, admin. Keys are stored hashed and shown once at creation.
  • consoleSession (http (bearer)) — Console session (003 §2): Identity Platform JWT. Data-plane calls add X-Tessera-Project + X-Tessera-Environment headers, validated against org membership.

Parameters​

NameInRequiredTypeDescription
cursorquerynostringOpaque pagination cursor from a previous response's next_cursor.
limitquerynointegerPage size (cursor pagination, 003 §1). Maximum 200.
statusquerynostringOnly the rows in this lifecycle status.

Pagination​

Cursor pagination (003 §1): pass limit (max 200) and cursor; the response is the pagination envelope — data plus next_cursor (null on the last page).

Responses​

StatusSchemaDescription
200DatasetFileStatePagePage of dataset build file lifecycles.
defaultErrorEnvelopeError response (003 §1). Every non-2xx response uses this envelope; the HTTP status matches error.code. Rate-limited requests get 429 resource_exhausted with retry_after_seconds and a Retry-After header; the X-RateLimit-* headers ride on EVERY response, success or error (003 §1; plan-driven per API key + endpoint class — WP-B2). A 503 unavailable carries retry_after_seconds too for a lock-wait timeout (SQLSTATE 55P03), a rule preview at capacity (preview_capacity_exhausted), or a label preview during which the erasure of every entity holding its censor clock began (label_preview_clock_entity_erased): how long to wait before retrying.

GET /dataset-files/{dataset_id}​

Get where a dataset build's files stand

The file lifecycle of one dataset build (WP-O7 reqs 3, 6, 9, 10 and 12): its status, the typed holds keeping a due purge waiting, the schedule (idle_since, due_at, not_before), the purge's instants and, once its files are deleted, what was deleted in total and by kind (deleted, deleted_by_kind) and how many soft-deleted generations the storage provider still holds, until provider_retained_until. snapshot is the build's event-store snapshot and what the purge did to its pin (pinned, released, shared with another build or preparation, or none), and blocking_erasures counts the project's entity deletions waiting for that snapshot's release. A build of the resolved project and environment without a lifecycle row reads as retained, its updated_at the build's own last update. Another project's or environment's build, or a missing one, is 404 not_found (003 §3: no existence leak). Requires the read scope.

Operation ID: getDatasetFiles · Service: gateway-api

Authentication​

  • apiKey (http (bearer)) — Project API key (003 §2): tsk_live_… or tsk_test_… in Authorization: Bearer. Scoped to exactly one project + environment; scopes are a subset of ingest, read, admin. Keys are stored hashed and shown once at creation.
  • consoleSession (http (bearer)) — Console session (003 §2): Identity Platform JWT. Data-plane calls add X-Tessera-Project + X-Tessera-Environment headers, validated against org membership.

Parameters​

NameInRequiredTypeDescription
dataset_idpathyesstringDataset build id (ds_ or eds_ ULID, 002 §1).

Responses​

StatusSchemaDescription
200DatasetFileStateThe dataset build's file lifecycle.
defaultErrorEnvelopeError response (003 §1). Every non-2xx response uses this envelope; the HTTP status matches error.code. Rate-limited requests get 429 resource_exhausted with retry_after_seconds and a Retry-After header; the X-RateLimit-* headers ride on EVERY response, success or error (003 §1; plan-driven per API key + endpoint class — WP-B2). A 503 unavailable carries retry_after_seconds too for a lock-wait timeout (SQLSTATE 55P03), a rule preview at capacity (preview_capacity_exhausted), or a label preview during which the erasure of every entity holding its censor clock began (label_preview_clock_entity_erased): how long to wait before retrying.

POST /dataset-files/{dataset_id}:cancel-purge​

Cancel the explicit purge request of a dataset build's files

Withdraws an explicit purge request (purgeDatasetFiles) before the purge starts (WP-O7 req 5, amended in specs/ERRATA.md): a request is due 24 hours after it is made at the earliest, its cancellation window, and a hold may keep it waiting longer. Model file purges the request made with include_archived_models are not withdrawn (cancelModelFilePurge withdraws each). When the project's dataset retention is on, the kind is registered and the build is idle, the row returns to the retention plan (trigger retention, due retention_days after it became idle): scheduled, or, when that is already due, evaluated once as a request is (held with the holds that apply, a failed integrity check the purge worker recorded kept with its since). Either way a cancel never shortens the window the request left, so it never makes a deletion start sooner: not_before stays at least the cancelled request's own not_before, with a drain_window hold until it when the plan is already due. Otherwise it returns to retained. Its requester is cleared, and dataset_files.purge_cancel (dataset_id and the optional reason) is recorded with the caller, with dataset_files.purge_hold when the set of hold reasons changes. A build without an explicit request (retained, or planned by retention) answers its current state with no write and no audit event, so a retried cancel is safe; Idempotency-Key replays the first response. Once the purge has started (deleting or later) its files are being or have been deleted and nothing can be undone: 400 failed_precondition, details[0] a DatasetFileRefusalDetail with reason purge_already_started and the current status. The cancel takes THE per-build lock exclusively, so it and the purge worker's transition into deleting never interleave. Another project's or environment's build, or a missing one, is 404 not_found. Requires an org role of admin or above, or the admin scope (the purgeDatasetFiles floor).

Operation ID: cancelDatasetFilePurge · Service: gateway-api

Authentication​

  • apiKey (http (bearer)) — Project API key (003 §2): tsk_live_… or tsk_test_… in Authorization: Bearer. Scoped to exactly one project + environment; scopes are a subset of ingest, read, admin. Keys are stored hashed and shown once at creation.
  • consoleSession (http (bearer)) — Console session (003 §2): Identity Platform JWT. Data-plane calls add X-Tessera-Project + X-Tessera-Environment headers, validated against org membership.

Parameters​

NameInRequiredTypeDescription
Idempotency-KeyheadernostringIdempotency key accepted by all POSTs (003 §1): 24 h dedup window, keyed per project. POSTs that are reads (batchSlotScores, scoreSlotEvent, deriveSlotBands, validateSourceRawPreset, sampleSource, watchEventFeedHeads and the preview operations: previewSourceMapping, previewCatalogMapping, previewLabel, previewPatternLabel, previewMarker, previewDatasetEligibility, previewSlotAudience, previewSharedCheckpointImport, previewSlotAtlasInquiry) accept the key but never replay a stored response: a retry with the same key is executed again (and metered again) against current data. A stored response that names an entity is removed when that entity is erased. A key that already answered answers only a retry of the request that stored its response: the same caller, environment, operation, path, query, scoping headers and body (a JSON body by its content, a gzip body by its inflated content). The same key with any other request is 409 already_exists with details[0].reason idempotency_key_reused, and that request is not run. The window holds no in-flight marker: two first requests sent at once can both run. A response stored by a release before this rule cannot be matched to its request and is refused the same way until it expires; send the request again with a new key. A body over 8 MiB (ingest: over its 5 MiB request limit) is run and its response is not stored. The ingest batch (POST /events) is the exception, except for an application's credentials: under a key that answered another batch it runs again, uncached, and its events' ids deduplicate them; so it does when its stored response cannot be checked. The operations that return a one-time secret store their response sealed: the stored copy opens only for the same key and credential, and one that cannot be opened is 409 idempotency_key_reused. While the server cannot check a stored response (it cannot read the key that checks it, for one), the request is 503 unavailable with Retry-After: retry with the same key. An operation whose handler claims the key itself answers the same 503, before it runs, while the key-hash pepper cannot be read; any other request sent with a key runs meanwhile, and its response is not stored.
dataset_idpathyesstringDataset build id (ds_ or eds_ ULID, 002 §1).

Idempotency​

This operation accepts an Idempotency-Key header (003 §1): within 24 hours, keyed per project and caller, a retry of the same request gets the first response, and the same key with another request is 409 already_exists (idempotency_key_reused). See the Idempotency-Key parameter for the exceptions.

Request body​

DatasetFilePurgeCancelRequest (optional)

Responses​

StatusSchemaDescription
200DatasetFileStateThe dataset build's file lifecycle after the cancel.
defaultErrorEnvelopeError response (003 §1). Every non-2xx response uses this envelope; the HTTP status matches error.code. Rate-limited requests get 429 resource_exhausted with retry_after_seconds and a Retry-After header; the X-RateLimit-* headers ride on EVERY response, success or error (003 §1; plan-driven per API key + endpoint class — WP-B2). A 503 unavailable carries retry_after_seconds too for a lock-wait timeout (SQLSTATE 55P03), a rule preview at capacity (preview_capacity_exhausted), or a label preview during which the erasure of every entity holding its censor clock began (label_preview_clock_entity_erased): how long to wait before retrying.

POST /dataset-files/{dataset_id}:purge​

Request the purge of a dataset build's files

Requests the deletion of a finished dataset build's own files (WP-O7 req 5, amended in specs/ERRATA.md): its shards, manifest, encoders, audit and derived-label files and build state, and its model-less baseline evaluations. Records, label stores, event-store files and other builds' files are never touched. The row gets trigger explicit and is due 24 hours after the request at the earliest, and never inside the 24-hour drain after the build became idle: due_at is the later of the request time plus 24 hours and the drain's end. Until the purge starts, cancelDatasetFilePurge withdraws the request. With include_archived_models: true, the same transaction requests the model file purge (purgeModelFiles) of every archived model version trained on the build that has none yet, each recorded as model_files.purge_request with the reason with dataset \{dataset_id\}; without it, the build waits held (referenced_by_model) until those models' purges start. The request is evaluated against the holds once: the 202 shows held with its holds, drain_window until due_at and any other that applies (a model not fenced yet, work in flight, a pending project purge, the organization's legal hold, and a failed integrity check the purge worker recorded, kept with its since). An explicit purge never bypasses a hold or the drain: the worker evaluates every hold again before it starts. Once it starts, the files cannot be restored; the build's released reads serve its frozen manifest and every launch that names it is refused. A build deleting or later answers its current state with no write and no audit event. A build whose purge was already requested answers its current state without requesting it again, so a retried request is safe; with include_archived_models: true it still requests the model file purge of every archived model trained on it that has none yet. Idempotency-Key replays the first response. A build that has not finished is 400 failed_precondition, details[0] a DatasetFileRefusalDetail with reason dataset_not_terminal; one with a model version trained on it that is not archived, reason dataset_in_use with up to 64 model_ids; a kind not registered for retention, reason kind_not_supported. Another project's or environment's build, or a missing one, is 404 not_found. Records dataset_files.purge_request (dataset_id, kind, include_archived_models, the number of models requested and the optional reason) with the requester, and dataset_files.purge_hold when holds apply. Requires an org role of admin or above, or the admin scope.

Operation ID: purgeDatasetFiles · Service: gateway-api

Authentication​

  • apiKey (http (bearer)) — Project API key (003 §2): tsk_live_… or tsk_test_… in Authorization: Bearer. Scoped to exactly one project + environment; scopes are a subset of ingest, read, admin. Keys are stored hashed and shown once at creation.
  • consoleSession (http (bearer)) — Console session (003 §2): Identity Platform JWT. Data-plane calls add X-Tessera-Project + X-Tessera-Environment headers, validated against org membership.

Parameters​

NameInRequiredTypeDescription
Idempotency-KeyheadernostringIdempotency key accepted by all POSTs (003 §1): 24 h dedup window, keyed per project. POSTs that are reads (batchSlotScores, scoreSlotEvent, deriveSlotBands, validateSourceRawPreset, sampleSource, watchEventFeedHeads and the preview operations: previewSourceMapping, previewCatalogMapping, previewLabel, previewPatternLabel, previewMarker, previewDatasetEligibility, previewSlotAudience, previewSharedCheckpointImport, previewSlotAtlasInquiry) accept the key but never replay a stored response: a retry with the same key is executed again (and metered again) against current data. A stored response that names an entity is removed when that entity is erased. A key that already answered answers only a retry of the request that stored its response: the same caller, environment, operation, path, query, scoping headers and body (a JSON body by its content, a gzip body by its inflated content). The same key with any other request is 409 already_exists with details[0].reason idempotency_key_reused, and that request is not run. The window holds no in-flight marker: two first requests sent at once can both run. A response stored by a release before this rule cannot be matched to its request and is refused the same way until it expires; send the request again with a new key. A body over 8 MiB (ingest: over its 5 MiB request limit) is run and its response is not stored. The ingest batch (POST /events) is the exception, except for an application's credentials: under a key that answered another batch it runs again, uncached, and its events' ids deduplicate them; so it does when its stored response cannot be checked. The operations that return a one-time secret store their response sealed: the stored copy opens only for the same key and credential, and one that cannot be opened is 409 idempotency_key_reused. While the server cannot check a stored response (it cannot read the key that checks it, for one), the request is 503 unavailable with Retry-After: retry with the same key. An operation whose handler claims the key itself answers the same 503, before it runs, while the key-hash pepper cannot be read; any other request sent with a key runs meanwhile, and its response is not stored.
dataset_idpathyesstringDataset build id (ds_ or eds_ ULID, 002 §1).

Idempotency​

This operation accepts an Idempotency-Key header (003 §1): within 24 hours, keyed per project and caller, a retry of the same request gets the first response, and the same key with another request is 409 already_exists (idempotency_key_reused). See the Idempotency-Key parameter for the exceptions.

Request body​

DatasetFilePurgeRequest (optional)

Responses​

StatusSchemaDescription
202DatasetFileStateThe dataset build's file lifecycle after the request.
defaultErrorEnvelopeError response (003 §1). Every non-2xx response uses this envelope; the HTTP status matches error.code. Rate-limited requests get 429 resource_exhausted with retry_after_seconds and a Retry-After header; the X-RateLimit-* headers ride on EVERY response, success or error (003 §1; plan-driven per API key + endpoint class — WP-B2). A 503 unavailable carries retry_after_seconds too for a lock-wait timeout (SQLSTATE 55P03), a rule preview at capacity (preview_capacity_exhausted), or a label preview during which the erasure of every entity holding its censor clock began (label_preview_clock_entity_erased): how long to wait before retrying.

Errors​

Every non-2xx response uses the 003 §1 error envelope; error.request_id matches the X-Request-Id response header, and the HTTP status always matches error.code:

{
  "error": {
    "code": "invalid_argument",
    "details": [
      {
        "type": "field_violation"
      }
    ],
    "message": "human-readable",
    "request_id": "req_01hxample"
  }
}

Closed error-code set (003 §1): invalid_argument, unauthenticated, permission_denied, not_found, already_exists, failed_precondition, resource_exhausted, internal, unavailable.

resource_exhausted responses (HTTP 429) additionally carry retry_after_seconds. See ErrorEnvelope.