GCS bucket connector

Syncs CSV/Parquet objects from a bucket you own into canonical events. Object-incremental: each sync lands only objects that are new or changed since the last one (by generation), so a daily export prefix "just works". Read-only against your bucket — the connector never writes, moves or deletes your objects.

Prerequisites

An API key with the admin scope.
A GCS bucket with event-shaped CSV or Parquet objects under a prefix.

Create the credential (least privilege)

No credential is stored. Tessera reads your bucket as its per-project service account: the connection test and syncs impersonate that account, so your bucket grant is the only access Tessera has. The connector lists objects under the prefix (storage.objects.list) and reads them (storage.objects.get) — exactly what roles/storage.objectViewer on the one bucket grants. Not project-wide, not writable:

gcloud storage buckets add-iam-policy-binding gs://acme-events-lake \
  --member="serviceAccount:SERVICE_ACCOUNT" \
  --role="roles/storage.objectViewer"

The source form does not show SERVICE_ACCOUNT. A connection test that is refused access names the exact account and repeats this command with it filled in. Per-project service accounts are provisioned on the Tessera side: if a test reports an authentication failure, Tessera could not use that account at all — contact your operator, because no bucket grant can fix it.

Create the source

source-config.json
{
  "name": "media-events-lake",
  "kind": "gcs",
  "config": {
    "bucket": "acme-events-lake",
    "prefix": "events/",
    "format": "csv"
  }
}

Download connect/gcs/source-config.json — this exact file runs in CI on every change (hosts shown are for this environment).

The same body, schema-checked on every docs build:

{
  "name": "media-events-lake",
  "kind": "gcs",
  "config": {
    "bucket": "acme-events-lake",
    "prefix": "events/",
    "format": "csv"
  }
}

Optional config: glob to narrow object names, manifest_object to sync exactly the objects a manifest lists, csv to pin a dialect, and on_object_change to control re-landing behavior when an object is rewritten in place.

Map columns, test, and run the first sync

POST /sources/{id}/connection-diagnostics runs two checks: list (stage object) lists up to five objects under the prefix, and read (stage authorization) reads one byte of the first one. With an empty prefix there is nothing to read, so read reports warn — object read access is not verified until objects arrive. A failed check carries a typed failure cause (authentication, authorization, not_found, network, …) and the fix; the response's checked_at is the server time of the test, which is also recorded in the audit log as source.test_connection. POST /sources/{id}/test-connection runs the same checks and returns only their names, statuses and details.

01-create-and-sync.sh
#!/usr/bin/env bash
# GCS bucket connector: create the source, test the connection, map, sync.
set -euo pipefail

: "${TESSERA_API_BASE:?e.g. https://api.staging.tessera.zone/v1}"
: "${TESSERA_API_KEY:?your tsk_test_… key}"
AUTH=(-H "Authorization: Bearer $TESSERA_API_KEY")

# 1. Create the source (no credential: extraction runs as your project's
#    dedicated service account, which you grant objectViewer on the bucket).
SOURCE="$(curl -sS -X POST "$TESSERA_API_BASE/sources" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-gcs-source" \
  --data-binary @source-config.json)"
SOURCE_ID="$(printf '%s' "$SOURCE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["source_id"])')"
echo "source created: $(printf '%s' "$SOURCE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["kind"])')"

# 2. Test the connection: names each check and, on failure, the exact grant.
curl -sS -X POST "$TESSERA_API_BASE/sources/$SOURCE_ID/test-connection" "${AUTH[@]}" \
  | python3 -c '
import json, sys
result = json.load(sys.stdin)
print("test-connection:", result["status"])
for check in result["checks"]:
    print("  %s: %s" % (check["name"], check["status"]))
'

# 3. Map columns, then run the first sync.
curl -sS -X PUT "$TESSERA_API_BASE/sources/$SOURCE_ID/mapping" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  --data-binary @../file-upload/mapping-media.json > /dev/null

SYNC="$(curl -sS -X POST "$TESSERA_API_BASE/sources/$SOURCE_ID/syncs" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-gcs-first-sync" \
  -d '{"mode": "full"}')"
SYNC_ID="$(printf '%s' "$SYNC" | python3 -c 'import json,sys; print(json.load(sys.stdin)["sync_id"])')"
for _ in $(seq 1 300); do
  SYNC="$(curl -sS "$TESSERA_API_BASE/syncs/$SYNC_ID" "${AUTH[@]}")"
  STATUS="$(printf '%s' "$SYNC" | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')"
  if [ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ]; then break; fi
  sleep 1
done
printf '%s' "$SYNC" | python3 -c '
import json, sys
sync = json.load(sys.stdin)
print("first sync: %s rows_landed=%s rejected=%s"
      % (sync["status"], sync["rows_landed"], sync["rows_rejected"]))
'

Download connect/gcs/01-create-and-sync.sh — this exact file runs in CI on every change (hosts shown are for this environment).

In CI this script runs against the local stack's fake customer bucket (seeded with the sample dataset's media-streaming CSV); against your real bucket it is byte-for-byte the same calls.

Rotate the credential

Tessera stores no credential for this connector, so there is nothing to rotate on the source. Access is the bucket grant you gave the per-project service account; revoke that grant to end access.

Troubleshooting

Symptom	Cause	Fix
`list` fails with `failure: authorization`	missing `objectViewer`	run the exact `gcloud` command from the check detail
`list` fails with `failure: not_found`	the bucket does not exist	check the bucket name
`list` fails with `failure: authentication`	Tessera could not use its per-project service account	contact your operator; a bucket grant cannot fix this
`read: warn`	no objects under the prefix yet	re-run the test once objects arrive
sync lands 0 objects	`prefix`/`glob` matches nothing	list the bucket with the same prefix; check trailing `/`
objects re-land every sync	upstream rewrites objects in place	write immutable, dated object names; or set `on_object_change`
`mapping_rejection_rate_exceeded`	mixed schemas under one prefix	one source per schema; split prefixes

Prerequisites​

Create the credential (least privilege)​

Create the source​

Map columns, test, and run the first sync​

Rotate the credential​

Troubleshooting​