Bridge Kafka to Tessera

Planned (Phase 2)

Planned for Phase 2. This guide describes the planned setup; the configuration shown below is validated against the current API schema so it stays accurate.

tessera/bus-bridge is a lightweight container you run next to your Kafka cluster: it consumes a consumer group, maps each JSON message to a canonical event with a declarative mapping config, and delivers batches to POST /v1/events — at-least-once from Kafka, exactly-once into Tessera via idempotent event ids derived from (bridge_id, topic, partition, offset). Your brokers stay private; Tessera never needs Kafka credentials.

Prerequisites

• An API key with the ingest scope.
• Somewhere to run a container (Docker, Kubernetes, ECS, …) with network access to your brokers and to ingest.staging.tessera.zone.
• Topics carrying JSON messages.

Write the mapping config

One YAML file drives the bridge. The mapping block is the declarative message→event mapping (JSON Pointers into the message body); it is validated up front — unknown fields and wrong versions are rejected:

bus-bridge.yaml

bridge_id: warehouse-bridge # keep stable: part of derived event ids
kafka:
  topics: [warehouse-scans]
  group_id: tessera-bus-bridge
  rejects_topic: tessera-rejects # optional: terminally-rejected messages
mapping:
  version: 1
  encoding: json
  entity_type: shipment
  entity_id_pointer: /shipment/id
  event_type_source: { static: shipment.scanned }
  timestamp_pointer: /scanned_at # RFC 3339 or epoch seconds/millis
  event_id_pointer: /scan_id # omit/null → derive from bus coordinates
  attributes: passthrough_flat # or { attr_key: /json/pointer, … }
  attribute_drops: [shipment_id]
  secondary_entities:
    - { type: hub, id_pointer: /hub/code }

Validate it before deploying:

docker run --rm -v $PWD/bus-bridge.yaml:/etc/tessera/bus-bridge.yaml:ro \
  tessera/bus-bridge:0.1.0 --check-config

Run the bridge

Secrets are environment-only — the config file refuses TLS/SASL keys. Any KAFKA_SOME_SETTING env var maps to the librdkafka setting some.setting:

bridge.env

TESSERA_API_KEY=…                # ingest-scope key; never in the config file
KAFKA_BOOTSTRAP_SERVERS=broker-1.internal:9096
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=SCRAM-SHA-512
KAFKA_SASL_USERNAME=bridge
KAFKA_SASL_PASSWORD=…

docker run -d --name tessera-bus-bridge --env-file bridge.env -p 8080:8080 \
  -v $PWD/bus-bridge.yaml:/etc/tessera/bus-bridge.yaml:ro \
  tessera/bus-bridge:0.1.0 # pin a semver tag — never latest

Give the consumer principal READ on the topics and the consumer group (plus WRITE on rejects_topic if configured) — nothing else.

Delivery semantics

• Ordering — partitions map to ordered per-partition delivery queues, so per-entity order is preserved when you partition by entity id.
• Offsets — committed only after Tessera accepted the events (or they were terminally handled); a crashed bridge replays, and replayed events dedup on their derived event_id.
• Backpressure — rate limits (429) and outages pause consumption (bounded in-flight budget, default 5,000 events); the bridge rides out multi-minute ingest outages without loss or rebalance storms.
• Rejects — unmappable messages (missing entity id, an entity id that is not an identifier such as true, 12.5, NaN or a whole number above 2^53 − 1 sent as a JSON number, an entity id with #, bad timestamp, an event_id with # in its first 16 characters, non-JSON) and per-item ingest rejections are counted, logged with their coordinates, and republished to rejects_topic with tessera-* failure headers when rejects_topic is set (otherwise they are only counted and logged, and the platform dead-letter queue never sees them). They never block the stream.
• Secondary ids — a participant id (secondary_entities) that is not an identifier (the rule), or is longer than 256 characters, never rejects the message: the event is delivered with its primary entity, without that participant, and the drop is counted in bridge_secondary_dropped_total{topic,reason} and logged without the value (at most one line a minute per pointer and reason).

Monitor it

/healthz is the liveness probe; /metrics exposes Prometheus counters: bridge_delivered_total, bridge_rejected_total{reason}, bridge_secondary_dropped_total{topic,reason}, bridge_retry_total, bridge_rekey_total, bridge_paused, bridge_delivery_latency_seconds.

Troubleshooting

Symptom	Cause	Fix
bridge_paused stuck at 1	ingest 429s or an outage	check GET /usage/quotas; the bridge catches up on its own
messages on rejects_topic with tessera-reason: invalid_event and a tessera-detail naming entity.id or secondary_entities.{i}.id	an entity id containing # (Tessera's storage key separator)	fix the id upstream (or the pointer), then resend; these messages never reach the dead-letter queue
messages on rejects_topic with tessera-reason: bad_timestamp	unparseable timestamp values	fix the producer or point timestamp_pointer at an RFC 3339/epoch field
container exits non-zero at startup	invalid config / missing TESSERA_API_KEY	run --check-config; set the key env var
duplicate-looking deliveries upstream	Kafka redelivery after a crash	none needed — derived event_ids dedup in Tessera