Batch catalogs: map nested records without a per-column mapping

A batch catalog maps the rows of a batch source through a declared, versioned catalog instead of a hand-written column mapping. It is the same catalog format the webhook adapters use (adapter_catalog.json), in its kind: batch arm, plus one bounded extension the data shapes need: JSON-pointer nested-record flattening.

Use a batch catalog when your rows are nested — Parquet struct and list columns, such as security-event lakes whose records nest actor, device and endpoint objects, or analytics exports that nest event parameters as a key/value array. Column mappings address top-level columns only; a catalog addresses any nested field, routes each row by a discriminator, and promotes only the key/value parameters you declare.

How rows are addressed

Every extracted row is presented as the document {"row": <record>} after Parquet decoding (structs become objects, lists become arrays). Every pointer in a batch catalog is an RFC 6901 JSON Pointer rooted at /row:

Pointer	Resolves to
`/row/class_uid`	a top-level column
`/row/actor/user/uid`	a field nested two structs deep
`/row/event_params`	a list column of key/value records

flatten.max_depth (1–8) bounds every pointer: its depth is the number of reference tokens, row included (/row/actor/user/uid has depth 4). A deeper pointer is rejected when you store the mapping — with a field_violation naming the pointer — never discovered at sync time.

The catalog document

{
  "catalog": {
    "kind": "batch",
    "name": "fixture.security-authentication",
    "version": 1,
    "source_event_pointer": "/row/class_uid",
    "flatten": {
      "max_depth": 4,
      "kv_promotions": [
        {
          "pointer": "/row/enrichments",
          "key_pointer": "/name",
          "value_pointers": ["/value"],
          "keys": ["geo_country", "risk_level"],
          "prefix": "enrich_",
          "max_items": 8
        }
      ]
    },
    "entries": {
      "3002": {
        "event_type": "auth.attempt",
        "entity": { "type": "user", "id_pointer": "/row/actor/user/uid" },
        "secondary_entities": [{ "type": "device", "id_pointer": "/row/device/uid" }],
        "timestamp_pointer": "/row/time_dt",
        "event_id_pointer": "/row/metadata/uid",
        "attributes": {
          "activity": "/row/activity_id",
          "source_ip": "/row/src_endpoint/ip",
          "status": "/row/status_id"
        }
      }
    }
  },
  "overrides": { "entries": { "3002": { "enabled": true } } }
}

source_event_pointer picks the entry per row. Integer discriminators select the entry keyed by their decimal string (3002).
Each entry declares the event type, the primary entity, optional secondary entities (an unresolvable secondary id drops only that participant), the event id, attributes, and skip_if rules.
timestamp_pointer is required on batch entries: extracted rows carry their own event time. Timezone-aware timestamp columns and RFC 3339 strings with an offset are read directly; epoch-number columns declare their unit with timestamp_format (unix_s, unix_ms, unix_us — integer microsecond exports included). A unit is never guessed: naive timestamps reject (naive_timestamp) unless the mapping opts in to an assume_timezone, and out-of-range values reject (unparseable_timestamp).
overrides is optional: partial entries merged per source event (override fields win, enabled: false disables an entry). The merged catalog is validated against the same bounds.

Local timestamps with no offset

Some exports write the publisher's local wall-clock time with no UTC offset (2010-12-01 08:26:00). By default a catalog rejects these rows with naive_timestamp, because the instant cannot be known without the zone. When you know the zone, set it on the mapping with the opt-in assume_timezone, an IANA name beside catalog and overrides:

{
  "catalog": { "kind": "batch", "name": "retail-lines", "version": 1 },
  "assume_timezone": "Europe/London"
}

With it, a timestamp that has no offset is read in that zone and stored as the corresponding UTC instant (2010-12-01 08:26:00 London in December is 2010-12-01T08:26:00Z; in July the same wall time is one hour earlier in UTC). It applies to the entry's timestamp_pointer value and to timestamp attributes. The same setting works in the pack-catalog form ({pack_catalog, overrides?, assume_timezone?}), where it is carried onto the stored mapping. Things it does not do:

Values that carry an offset or a Z are never reinterpreted, and epoch numbers still need their declared timestamp_format.
A bare date such as 2010-12-01 has no time of day to place in a zone, so it still rejects as naive_timestamp.
One zone covers the whole mapping. A source whose columns are in different zones needs those columns converted before they reach Tessera.

Clock changes

Twice a year a local wall-clock reading is not a single instant, and Tessera does not guess. The rule is the same recipe the automatic onboarding path uses (tessera.local-time.v1), pinned to reject repeated times (onboarding lets you choose how a repeated time is handled; a catalog does not): a reading that occurs once converts exactly, and one that does not occur exactly once is rejected with the existing naive_timestamp reason and a message that says which case it is.

Zone	Wall time	What happened	Result
`Europe/London`	`2026-03-29T00:59:59`	GMT, last second before the change	`2026-03-29T00:59:59Z`
`Europe/London`	`2026-03-29T01:30:00`	clocks went forward at 01:00; 01:00 to 01:59 never happened	rejected: "does not exist in Europe/London (clocks go forward)"
`Europe/London`	`2026-03-29T02:00:00`	BST, first reading after the change	`2026-03-29T01:00:00Z`
`Europe/London`	`2026-10-25T01:30:00`	clocks went back at 02:00; 01:00 to 01:59 happened twice	rejected: "is ambiguous in Europe/London (clocks go back)"
`Europe/London`	`2026-10-25T02:00:00`	GMT, first reading after the repeated hour	`2026-10-25T02:00:00Z`

This is stricter than the column mapping's timezone setting. A column mapping reads a repeated local time as its first occurrence (the earlier instant) and converts a skipped local time with the offset in force before the change, so it never rejects either case. That behaviour is unchanged, because changing it would alter ingestion for existing sources; it applies only to column mappings. assume_timezone on a catalog follows the table above.

Rejected rows count toward the source's reject gate like any other naive_timestamp row. If your data has many readings in the repeated hour, convert them to UTC before they reach Tessera or use a zone without clock changes.

An unknown zone name is 400 invalid_argument at /assume_timezone; so are host-local, posix/ and right/ names and anything missing from the IANA tz database, so storing the setting never depends on the machine it runs on. The setting is part of the mapping document, so it is recorded on the mapping version and in its fingerprint, and changing it creates a new version. Preview shows the effect before you store it.

Sources that already have data

The zone applies only to rows mapped after the new mapping version is stored. Events already ingested are not reinterpreted.

Incremental syncs skip files that are already in the ledger, so rows that were rejected earlier as naive_timestamp are not read again by the next incremental sync. Run a full sync to re-read them.
Changing a zone that was already set gives the same rows different instants.

Changing an existing zone can duplicate events

If an entry has no event_id_pointer, an event's identity is derived from its content including its timestamp. Re-reading rows after changing assume_timezone therefore lands them as new events at shifted instants next to the old ones, not as corrections. Declare an event_id_pointer on the entries before you change a zone.

Key/value promotion

kv_promotions lift parameters stored as an array of key/value records into attributes named prefix + key:

only the declared keys are promoted — undeclared keys never become attributes;
value_pointers are tried in order: absent and null values fall through, while 0 and false are values;
an array longer than max_items (1–256) rejects the whole row with flatten_bounds_exceeded — it is never silently truncated;
a declared key that occurs twice in one array, or a promoted name that collides with an entry attribute, is refused rather than resolved arbitrarily.

Store, preview and sync

Store the catalog as the source's mapping. The whole document is fingerprinted (fingerprint(canonical_json(…))); re-sending an identical document is idempotent and every sync records the fingerprint it ran with.

curl -X PUT "$TESSERA_API/v1/sources/$SOURCE_ID/mapping" \
  -H "Authorization: Bearer $TESSERA_KEY" -H "Content-Type: application/json" \
  --data @catalog-mapping.json

A source stores either a column mapping or a catalog in each mapping version — never both. POST /v1/sources/{id}/mapping:preview runs a candidate (or the stored) catalog side-effect free over inline rows (sample_payload) or the latest landed snapshot (from_snapshot) and answers in the same shape it gives a column mapping. To see why each row was skipped or rejected, call POST /v1/sources/{id}/catalog-mapping:preview with the same body: its response adds row_outcomes, one per skipped or rejected row (the first 1,000), with its typed reason and the offending pointer. It accepts batch-catalog mappings on batch sources only and has no side effects.

During a sync:

Row	Outcome
discriminator has no enabled entry	skipped, counted (`unmapped_event_type` / `entry_disabled`)
a `skip_if` rule matches	skipped, counted (`skip_if`)
mapped entry fails	rejected with a typed reason in `rejected/part-*.parquet`
key/value array over `max_items`	rejected (`flatten_bounds_exceeded`)
a secondary entity id is not an identifier (a fractional number, a boolean, …) or is over 256 characters	kept without that participant, counted (`secondary_ids_dropped`)

Rejected rows count toward the source's max_reject_fraction; skip counts appear on the sync's details as {type: "catalog_skips", reason, count}, and dropped secondary participants as {type: "secondary_ids_dropped", column, reason, count} (column holds the entry's pointer; see how identifiers become entity ids). Catalog-produced events are profiled and stay excluded from models until their contract is approved, exactly like column-mapped events.

Limits

Batch catalogs map sources whose rows land whole: file uploads, GCS and S3 (Azure Blob is planned). Warehouse kinds keep column mappings (their extraction projects the mapped columns); storing a catalog on one fails with failed_precondition (catalog_mapping_unsupported).
Catalogs declare no cursor: mode=incremental relies on the per-source object ledger of file kinds.

How rows are addressed​

The catalog document​

Local timestamps with no offset​

Clock changes​

Sources that already have data​

Key/value promotion​

Store, preview and sync​

Limits​