Skip to content

Databricks AI Functions usage

Databricks contributes system-table usage rows for AI Functions, model serving, and foundation-model fine-tuning, plus cluster, job, and pipeline ownership context for attribution.

Onboarding scope

This guide covers the read-only databricks_ai_functions connector in the Venturi platform connector package. It reads Unity Catalog system tables, which Databricks makes read-only, and never changes workspaces, jobs, or compute.

Required access

The connector reads these Databricks system tables. They are vendor object names:

System table Purpose
system.billing.usage Read AI billing usage rows.
system.compute.clusters Correlate usage to cluster owners and tags.
system.lakeflow.jobs Correlate usage to job creators and run-as identities through usage_metadata.job_id.
system.lakeflow.pipelines Correlate usage to pipeline creators and run-as identities through usage_metadata.dlt_pipeline_id.

System tables require a Unity Catalog-enabled workspace. Grant a dedicated service principal USE CATALOG on the system catalog and USE SCHEMA and SELECT on each schema it reads:

SQL
GRANT USE CATALOG ON CATALOG system TO `<service-principal-application-id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.billing TO `<service-principal-application-id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.compute TO `<service-principal-application-id>`;
GRANT USE SCHEMA, SELECT ON SCHEMA system.lakeflow TO `<service-principal-application-id>`;

The service principal also needs compute to run these reads. Add it to the workspace with the Databricks SQL access entitlement, and grant it CAN USE on a small SQL warehouse dedicated to monitoring. CAN USE lets it start the warehouse and run queries but not edit, stop, or delete the warehouse. See Databricks’ entitlements and SQL warehouse ACLs.

Do not grant workspace, job, or cluster administration permissions. See Databricks’ system tables access requirements, billable usage table reference, and Lakeflow jobs system tables.

How AI Functions usage appears

The connector keeps system.billing.usage rows whose billing_origin_product is one of these documented values:

billing_origin_product What it covers
AI_FUNCTIONS AI Functions that Databricks bills as their own product, such as ai_parse_document, ai_extract, and ai_classify. product_features.ai_functions.ai_function names the function.
MODEL_SERVING Model serving endpoints, including AI Functions that run as batch inference through Model Serving (product_features.model_serving.offering_type = 'BATCH_INFERENCE').
FOUNDATION_MODEL_TRAINING Foundation-model fine-tuning.

AI Functions that run entirely on the calling compute, such as ai_forecast, have no separate inference charge; their cost stays in that compute’s usage. Rows with other billing_origin_product values are excluded from this connector’s output. Databricks documents these billing paths in AI Functions and model serving cost monitoring.

Setup

  1. Create or select a service principal for Venturi, and add it to the workspace with the Databricks SQL access entitlement.
  2. Run the grants above, and grant the service principal CAN USE on the SQL warehouse dedicated to monitoring.
  3. Store the service principal’s OAuth secret as an encrypted credential reference in Settings / Admin / Credentials.
  4. The default lookback is 24 hours. Set VENTURI_DATABRICKS_AI_LOOKBACK_HOURS and VENTURI_DATABRICKS_AI_HIGH_VOLUME_THRESHOLD only when you need to override the lookback or the high-volume flag. See sync windows and failure behavior.
  5. In Venturi, go to Integrations / Databricks AI Functions and run Test connection.

Verification

  • Test connection passes the read-only boundary check: system-table reads succeed on the monitoring SQL warehouse, the service principal holds only CAN USE on that warehouse, and it holds no workspace, job, or cluster administration permission.
  • Only AI_FUNCTIONS, MODEL_SERVING, and FOUNDATION_MODEL_TRAINING usage rows survive the connector filter.
  • Cluster, job, and pipeline ownership context appears when available.
  • High-volume AI usage, such as one SQL query that triggers millions of AI Function calls, is flagged without blocking ingestion.
  • The connector inventory marks Databricks AI Functions as read-only.

Rotation and offboarding

Rotate the service principal’s OAuth secret on your Databricks schedule, then update the credential reference in Settings / Admin / Credentials. Removing the connector stops new Databricks AI usage correlation.

Give feedback on this page

Sign in to send feedback. Opens in a new tab.

Copy page details 
Docs feedback: Databricks AI Functions usage
Page: https://docs.venturi.systems/connectors/databricks-ai-functions/