Databricks connector

Extracts a Unity Catalog table through a SQL warehouse using the Statement Execution REST API with disposition=EXTERNAL_LINKS, format=ARROW_STREAM: result chunks stream straight from presigned links, only mapped columns are selected, and incremental syncs push the cursor predicate down. Results estimated over 4 GiB switch to a volume unload: COPY INTO '/Volumes/{unload_volume}/tessera/{sync_id}/' … FILEFORMAT = PARQUET, downloaded via the Files API and deleted afterwards — the per-sync volume directory is removed even when the sync fails. The connector never creates tables (CREATE TABLE … AS is never used).

Prerequisites

• An API key with the admin scope.

• A SQL warehouse the token's user may run statements on.

• A Unity Catalog table holding event-shaped rows.

• For very large tables: a Unity Catalog volume the token can write (unload_volume).

• A workspace URL of the form https://<workspace-host>, with no path, port or query. The Azure ?o=<workspace-id> suffix is accepted and ignored. The host must be in Databricks' own domains:

  • cloud.databricks.com
  • gcp.databricks.com
  • azuredatabricks.net
  • databricks.azure.us
  • databricks.azure.cn
  • cloud.databricks.us

  Every request goes to a public address that Tessera checked, with TLS verified against the workspace name. Result download links must also be https, use port 443 and resolve to public addresses. A refused workspace URL fails authenticate with failure: configuration. PUT /mapping returns 400 invalid_argument with a field_violation on config/workspace_url, and a sync fails with extraction_failed. A refused result link fails the sync with no field violation, because Databricks issued it.

Create the credential (least privilege)

Create a service principal (or dedicated user) with a personal access token and grant it exactly:

GRANT USE CATALOG ON CATALOG main TO `tessera-svc`;
GRANT USE SCHEMA ON SCHEMA main.analytics TO `tessera-svc`;
GRANT SELECT ON TABLE main.analytics.events TO `tessera-svc`;
-- Only when configuring unload_volume:
GRANT READ VOLUME, WRITE VOLUME ON VOLUME main.analytics.tessera_unload TO `tessera-svc`;

The warehouse additionally needs CAN USE for the same principal (granted in the warehouse's permissions). These match what the connector calls: statements on the SQL warehouse (CAN USE), information_schema.columns and the mapped columns of the one table (USE CATALOG, USE SCHEMA, SELECT), and — only for results over 4 GiB with unload_volume set — COPY INTO the volume plus Files API reads and deletes (READ VOLUME, WRITE VOLUME). The connection test does not exercise the volume path. A personal access token is the Phase-2 contract; service-principal OAuth M2M is planned.

Test the connection

POST /sources/{id}/connection-diagnostics runs three checks in order, each tagged with the diagnostic stage it verifies:

Check	Stage	Verifies
authenticate	authentication	the workspace accepts the personal access token (current-user lookup; warn when that endpoint gives no verdict, and the warehouse probe still exercises the token)
warehouse_usable	authorization	the SQL warehouse at http_path runs a probe statement
table_readable	schema	the table's columns are visible to the token's principal

A failed check carries a typed failure cause (authentication, authorization, not_found, network, …) and a detail naming the grant or setting that fixes it; later checks report not_attempted. Unity Catalog hides tables a principal may not see, so a missing table grant often reports not_found; that detail lists the grants to check. Details never repeat the workspace's message or the token. The response's checked_at is the server time of the test, which is also recorded in the audit log as source.test_connection. POST /sources/{id}/test-connection runs the same checks and returns only their names, statuses and details.

Create the source

{
  "name": "product-events",
  "kind": "databricks",
  "config": {
    "workspace_url": "https://adb-1234567890123456.7.azuredatabricks.net",
    "http_path": "/sql/1.0/warehouses/abcdef1234567890",
    "catalog": "main",
    "schema": "analytics",
    "table": "events",
    "unload_volume": "main/analytics/tessera_unload"
  },
  "credentials": {
    "personal_access_token": "dapi..."
  }
}

The config object validates against the connector's registered schema:

{
  "workspace_url": "https://adb-1234567890123456.7.azuredatabricks.net",
  "http_path": "/sql/1.0/warehouses/abcdef1234567890",
  "catalog": "main",
  "schema": "analytics",
  "table": "events"
}

Rotate the credential

The source holds one personal access token. Tessera does not renew it: if the token has an expiry, replace it before that date. To rotate, create a new token for the same principal, PATCH /sources/{id} with credentials.personal_access_token, confirm POST /sources/{id}/connection-diagnostics passes, then revoke the old token. Service-principal OAuth is planned and not available.

How extraction chooses its path

1. The connector estimates the result size: SELECT COUNT(*) (with the same cursor predicate) × the mean row width of the projected columns.
2. ≤ 4 GiB → Statement Execution with EXTERNAL_LINKS/ARROW_STREAM; chunks stream from presigned links.
3. > 4 GiB (or when the API reports a truncated result) → volume unload: COPY INTO '/Volumes/{unload_volume}/tessera/{sync_id}/' FROM (SELECT …) FILEFORMAT = PARQUET, Files-API downloads, then the directory is deleted (always — a finally path).
4. Large result with no unload_volume configured → the sync fails with unload_volume_required; set config.unload_volume and re-sync.

Incremental syncs require a mapping cursor on a TIMESTAMP/TIMESTAMP_NTZ/DATE or integer column; the watermark advances only when a sync commits.

Type map

Column types map onto canonical attribute types exactly as follows; mapping a column whose type is unsupported fails PUT /sources/{id}/mapping naming the column and its type.

Databricks type	Canonical type	Notes
int	integer	Exact 32-bit integers.
bigint	integer	Exact 64-bit integers (LONG is a synonym).
smallint	integer	Exact 16-bit integers (SHORT is a synonym).
tinyint	integer	Exact 8-bit integers (BYTE is a synonym).
float	number	IEEE-754 floats (REAL is a synonym).
double	number	IEEE-754 float64, as-is.
decimal	number	Converted to float64 (values beyond float64 precision convert lossily). An entity or event id column keeps its exact decimal value.
boolean	boolean	Booleans, as-is.
string	string	UTF-8 strings, as-is (≤ 4096 chars per 002 §2).
timestamp	timestamp	Session-zoned instants, converted to RFC 3339 UTC.
timestamp_ntz	timestamp	Naive wall-clock time: the mapping's IANA timezone is required — without it, rows carrying a value are rejected.
date	timestamp	Midnight UTC of the date (YYYY-MM-DDT00:00:00Z).
array<string>	array	Array of strings (≤ 64 items per 002 §2).
array<numeric>	array	Array of INT/BIGINT/SMALLINT/TINYINT/FLOAT/DOUBLE/DECIMAL elements (≤ 64 items); elements convert like their scalar type.
map	unsupported	Cannot be mapped (flatten upstream); PUT /mapping fails naming the column.
struct	unsupported	Cannot be mapped (flatten upstream); PUT /mapping fails naming the column.
binary	unsupported	Cannot be mapped; PUT /mapping fails naming the column.
variant	unsupported	Cannot be mapped (flatten upstream); PUT /mapping fails naming the column.
interval	unsupported	Cannot be mapped; PUT /mapping fails naming the column.

DATE maps to midnight UTC by convention; if your dates carry local-day semantics, expose a naive TIMESTAMP_NTZ column instead and set the mapping timezone.