Skip to content

Databricks AI Functions usage

Databricks contributes system-table usage rows for model-serving and foundation model workloads, plus cluster and Lakeflow context for attribution.

Release state

This guide documents the read-only databricks_ai_functions connector implemented in the platform connector surface.

Required access

Grant read access to these Databricks system tables:

Table Purpose
system.billing.usage Read AI billing usage rows.
system.compute.clusters Correlate usage to cluster metadata.
system.lakeflow.job_run_timeline Correlate usage to Lakeflow job runs.

Do not grant workspace mutation, job mutation, or cluster administration permissions.

Filter system.billing.usage.billing_origin_product to documented values. Model serving uses MODEL_SERVING; foundation-model fine-tuning uses FOUNDATION_MODEL_TRAINING. Unknown values remain visible as unknown rather than being silently dropped.

Databricks documents the billing table and product origins in model-serving cost monitoring. “AI Functions” in this guide is the connector package name; the usage rows are model-serving and foundation-model-training charges from system.billing.usage, not a claim that every row came from an SQL AI Function.

Setup

  1. Create or select a service principal for Venturi.
  2. Grant read access to the system tables above.
  3. Store the credential reference in Venturi.
  4. The default lookback is 24 hours. Set VENTURI_DATABRICKS_AI_LOOKBACK_HOURS and VENTURI_DATABRICKS_AI_HIGH_VOLUME_THRESHOLD only when you need to override the default poll window or high-volume flag.
  5. In Venturi, open Integrations and select Databricks AI Functions and run Test connection.

Verification

  • Only MODEL_SERVING and FOUNDATION_MODEL_TRAINING usage rows survive the connector filter.
  • Cluster and Lakeflow pipeline context appears when available.
  • High-volume AI usage is flagged without blocking ingestion.
  • The connector remains read-only and continues operating, skipping upstream read errors without interrupting ingestion.

Rotation and offboarding

Rotate the service-principal credential on your Databricks schedule. Removing the connector stops new Databricks AI usage correlation.