BigQuery connector

Extracts a BigQuery table into canonical events with the BigQuery Storage Read API (Arrow format): column projection is limited to the columns your mapping references, incremental syncs push the cursor predicate down as a row_restriction, and read sessions are capped at 4 streams. The connector never writes to your project — no temp tables, no DDL.

Prerequisites

An API key with the admin scope.
A BigQuery table (or view materialized to one) holding event-shaped rows: an entity id column, a timestamp column, attribute columns.
Permission to grant IAM roles on the dataset's GCP project.

Create the credential (least privilege)

The connector makes exactly these calls, so these are the only permissions it needs — no job-creation rights, no write access, no project-wide viewer:

Call	Needs	Least-privilege grant
Table metadata (`tables.get`: connection test, mapping validation, every sync)	`bigquery.tables.get` on the table	`roles/bigquery.dataViewer` on the table
Storage Read API session and stream reads	`bigquery.tables.getData` on the table; `bigquery.readsessions.create` and `bigquery.readsessions.getData` on the table's project	`roles/bigquery.dataViewer` on the table + `roles/bigquery.readSessionUser` on the project

Grant dataViewer on the single table, the narrowest scope. Grant it on the dataset instead only when new tables should be covered without another grant. readSessionUser can only be granted on the project.

-- In BigQuery (table-level read access):
GRANT `roles/bigquery.dataViewer`
ON TABLE `CUSTOMER_PROJECT.DATASET.TABLE`
TO "serviceAccount:SERVICE_ACCOUNT";

Then project-level Storage Read API sessions:

gcloud projects add-iam-policy-binding CUSTOMER_PROJECT \
  --member="serviceAccount:SERVICE_ACCOUNT" \
  --role="roles/bigquery.readSessionUser"

The console's BigQuery form renders the same two grants with your project, dataset and table filled in. Which identity SERVICE_ACCOUNT is depends on the auth_mode you create the source with:

`cross_project_grant`

Tessera stores no customer key for this source. Both the connection test and the sync use Application Default Credentials from the runtime that executes them. Connection tests run in the gateway; syncs run in connector-runner jobs. Their identities can differ.

The source form does not currently expose a verified service-account address, and this connector does not select or impersonate an account for your Tessera project. Ask your operator to confirm both runtime identities before granting access. A successful gateway connection test alone does not verify the sync runtime's credentials.

For each confirmed service-account identity, use its exact email as SERVICE_ACCOUNT in the grants above. The grants are templates; they do not provision an identity.

`service_account_key`

Provide your own service account's key file as credentials.service_account_key_json; SERVICE_ACCOUNT above is the client_email in that key. The key is envelope-encrypted at rest (per-credential AES-256-GCM data key wrapped by your project's KMS key) and is never readable through the API. The account needs the same two roles as above — grant nothing more.

Use the key file exactly as Google Cloud issues it. Its token_uri must be exactly https://oauth2.googleapis.com/token, or exactly https://accounts.google.com/o/oauth2/token (older keys). Its universe_domain, when present, must be googleapis.com. Tessera refuses any other key before it makes a request: authenticate fails with failure: configuration, PUT /mapping returns 400 invalid_argument with a field_violation on credentials/service_account_key_json, and a sync fails with extraction_failed. Keys downloaded from Google Cloud already meet this rule.

Create the source

{
  "name": "match-events",
  "kind": "bigquery",
  "config": {
    "gcp_project": "customer-proj",
    "dataset": "analytics",
    "table": "match_events",
    "auth_mode": "cross_project_grant"
  }
}

The config object validates against the connector's registered schema:

{
  "gcp_project": "customer-proj",
  "dataset": "analytics",
  "table": "match_events",
  "auth_mode": "cross_project_grant"
}

Map columns to canonical events

The mapping declares entity / event type / timestamp / attribute columns — see the quickstart's worked mapping. Two BigQuery-specific rules:

Incremental syncs. Declare a mapping cursor on a TIMESTAMP, DATETIME, DATE or INT64 column (other types are rejected at PUT /mapping). Incremental syncs extract only rows with cursor > watermark, pushed down as a literal restriction, e.g. ts > TIMESTAMP "2026-06-01T00:00:00+00:00"; full syncs never set a restriction. The watermark only advances when a sync commits.

Type map. Every mapped column is converted per this normative table (generated from the connector's type map — code and docs cannot drift). Mapping an unsupported column fails PUT /sources/{id}/mapping with the column and its type named.

BigQuery type	Canonical type	Notes
`INT64`	`integer`	Exact 64-bit integers.
`FLOAT64`	`number`	IEEE-754 float64, as-is.
`NUMERIC`	`number`	Converted to float64; values with > 15 significant digits convert lossily and the profile flags `precision_loss`. An entity or event id column keeps its exact decimal value.
`BIGNUMERIC`	`number`	Converted to float64; values with > 15 significant digits convert lossily and the profile flags `precision_loss`. An entity or event id column keeps its exact decimal value.
`BOOL`	`boolean`	Booleans, as-is.
`STRING`	`string`	UTF-8 strings, as-is (≤ 4096 chars per 002 §2).
`TIMESTAMP`	`timestamp`	Converted to RFC 3339 UTC.
`DATETIME`	`timestamp`	Naive wall-clock time: the mapping's IANA `timezone` is required — without it, rows carrying a value are rejected.
`DATE`	`timestamp`	Midnight UTC of the date (`YYYY-MM-DDT00:00:00Z`).
`ARRAY<STRING>`	`array`	Array of strings (≤ 64 items per 002 §2).
`ARRAY<INT64>`	`array`	Array of integers (≤ 64 items).
`ARRAY<FLOAT64>`	`array`	Array of numbers (≤ 64 items).
`ARRAY<NUMERIC>`	`array`	Array of numbers; lossy elements flag `precision_loss`.
`BYTES`	`unsupported`	Cannot be mapped; `PUT /mapping` fails naming the column.
`GEOGRAPHY`	`unsupported`	Cannot be mapped; `PUT /mapping` fails naming the column.
`STRUCT`	`unsupported`	Cannot be mapped (flatten upstream); `PUT /mapping` fails naming the column.
`JSON`	`unsupported`	Cannot be mapped (flatten upstream); `PUT /mapping` fails naming the column.
`INTERVAL`	`unsupported`	Cannot be mapped; `PUT /mapping` fails naming the column.
`RANGE`	`unsupported`	Cannot be mapped; `PUT /mapping` fails naming the column.

Date and naive-datetime semantics

DATE values become midnight UTC of that date (2026-02-01 → 2026-02-01T00:00:00Z).
DATETIME values are naive wall-clock times: the mapping's IANA timezone is required, and each value is converted from that zone to UTC. Without a timezone, rows carrying a value are rejected.
If your dates carry local-date semantics, map them through a naive DATETIME column and set the mapping timezone instead of relying on the midnight-UTC convention.
NUMERIC/BIGNUMERIC values with more than 15 significant digits convert lossily to float64; the sync's profiling report flags the attribute with precision_loss.

Test the connection and run the first sync

POST /sources/{id}/connection-diagnostics runs three checks in order, each tagged with the diagnostic stage it verifies:

Check	Stage	Verifies
`authenticate`	`authentication`	the key (or the runtime's credentials) can be selected or materialized
`table_exists`	`object`	the table metadata is readable — the first call that uses the credentials
`storage_read_session`	`authorization`	a Storage Read session opens; it is closed again without reading rows

A failed check carries a typed failure cause (network, authentication, authorization, not_found, configuration or provider_error) and a detail that names the least-privilege grant or setting that fixes it; checks after it report not_attempted. Details never repeat Google's error text or any key material. The response's checked_at is the server time of the test, and each test is recorded in the audit log as source.test_connection. POST /sources/{id}/test-connection runs the same checks and returns only their names, statuses and details. The test does not validate a mapping, show a sample, start a sync or confirm ingestion. The full flow, as one runnable script (it needs a real BigQuery table, so CI runs it against our hosted deployment rather than the local stack):

01-create-and-sync.sh
#!/usr/bin/env bash
# BigQuery connector: create the source, test the connection, first sync.
# Requires a real BigQuery table (hosted runs only, not the local stack): set
# TESSERA_BQ_PROJECT / TESSERA_BQ_DATASET / TESSERA_BQ_TABLE.
set -euo pipefail

: "${TESSERA_API_BASE:?e.g. https://api.staging.tessera.zone/v1}"
: "${TESSERA_API_KEY:?your tsk_test_… key}"
: "${TESSERA_BQ_PROJECT:?customer GCP project}"
: "${TESSERA_BQ_DATASET:?BigQuery dataset id}"
: "${TESSERA_BQ_TABLE:?table id}"
AUTH=(-H "Authorization: Bearer $TESSERA_API_KEY")

# cross_project_grant: no stored credential — you grant Tessera's per-project
# service account dataViewer on the dataset + readSessionUser on the project.
SOURCE="$(curl -sS -X POST "$TESSERA_API_BASE/sources" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-bigquery-source" \
  -d '{
    "name": "warehouse-events",
    "kind": "bigquery",
    "config": {
      "gcp_project": "'"$TESSERA_BQ_PROJECT"'",
      "dataset": "'"$TESSERA_BQ_DATASET"'",
      "table": "'"$TESSERA_BQ_TABLE"'",
      "auth_mode": "cross_project_grant"
    }
  }')"
SOURCE_ID="$(printf '%s' "$SOURCE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["source_id"])')"

# The checks name the exact missing IAM grant on failure.
curl -sS -X POST "$TESSERA_API_BASE/sources/$SOURCE_ID/test-connection" "${AUTH[@]}" \
  | python3 -c '
import json, sys
result = json.load(sys.stdin)
print("test-connection:", result["status"])
for check in result["checks"]:
    print("  %s: %s" % (check["name"], check["status"]))
'

SYNC="$(curl -sS -X POST "$TESSERA_API_BASE/sources/$SOURCE_ID/syncs" "${AUTH[@]}" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: docs-bigquery-first-sync" \
  -d '{"mode": "full"}')"
SYNC_ID="$(printf '%s' "$SYNC" | python3 -c 'import json,sys; print(json.load(sys.stdin)["sync_id"])')"
for _ in $(seq 1 600); do
  STATUS="$(curl -sS "$TESSERA_API_BASE/syncs/$SYNC_ID" "${AUTH[@]}" \
    | python3 -c 'import json,sys; print(json.load(sys.stdin)["status"])')"
  if [ "$STATUS" = "succeeded" ] || [ "$STATUS" = "failed" ]; then break; fi
  sleep 1
done
echo "first sync: $STATUS"

Download connect/bigquery/01-create-and-sync.sh — this exact file runs in CI on every change (hosts shown are for this environment).

Rotate the credential

With service_account_key, the stored service account key stays valid until you delete or disable it in Google Cloud. To rotate: create a new key for the same service account, PATCH /sources/{id} with credentials.service_account_key_json, confirm POST /sources/{id}/connection-diagnostics passes, then delete the old key. If you change to a different service account, grant it the two roles above first. With cross_project_grant there is nothing stored to rotate; access follows the grants you gave the runtime identities.

Troubleshooting

Symptom	Cause	Fix
`authenticate: failed`	no usable key is stored	replace the service-account key
`authenticate` fails with `failure: configuration`	the key's `token_uri` or `universe_domain` is not Google's	download a new key from Google Cloud and replace the stored one
`table_exists` fails with `failure: authentication`	Google rejected the key, or the Tessera runtime has no credentials (cross-project grant)	replace the key; for a cross-project grant, ask your operator
`table_exists` fails with `failure: authorization`	missing `dataViewer` — BigQuery also reports a table that does not exist this way	grant it on the table (or dataset) and check the names
`table_exists` fails with `failure: not_found` or `configuration`	wrong or invalid project, dataset or table name	enter the table name only, without the dataset
`storage_read_session` fails with `failure: authorization`	missing `readSessionUser`, or the BigQuery Storage API is disabled	grant it at the project level, not the dataset; enable the API
`failure: network`	the runtime could not reach the BigQuery API	retry; if it persists, contact your operator
`PUT /mapping` fails naming a column	unsupported type (`STRUCT`, `JSON`, …)	flatten upstream into scalar columns
incremental sync extracts nothing	watermark ahead of new rows	check the cursor column monotonically increases; run one `mode: full` sync
rows rejected as `invalid_timestamp`	naive `DATETIME` without `timezone`	set the mapping's IANA `timezone`

Prerequisites​

Create the credential (least privilege)​

cross_project_grant​

service_account_key​

Create the source​

Map columns to canonical events​

Date and naive-datetime semantics​

Test the connection and run the first sync​

Rotate the credential​

Troubleshooting​