Skip to content

BigQuery ML and Vertex AI usage

BigQuery ML contributes query-job metadata, and Vertex AI contributes request-log context, for AI usage that runs through Google data-platform workloads such as BigQuery ML remote models that call Gemini.

Onboarding scope

This guide covers the read-only bigquery_ml connector in the Venturi platform connector package. It correlates BigQuery jobs with Vertex AI request logs by BigQuery job ID and grants no dataset, job, or model mutation.

Required access

Grant these Google Cloud IAM roles to a dedicated service account. They are vendor permissions:

The Scope column identifies where to grant each role; Permission lists the access it provides.

IAM role Scope Permission Purpose
roles/bigquery.resourceViewer Each project that runs BigQuery AI workloads bigquery.jobs.listAll Read job metadata for every user’s jobs in the project through INFORMATION_SCHEMA.JOBS.
roles/bigquery.jobUser The project where the connector runs its read queries bigquery.jobs.create Run the read-only metadata queries.
roles/bigquery.dataViewer Only the dataset that holds the Vertex AI request-response log table bigquery.tables.getData Read Vertex AI request-log metadata for correlation.

Google requires both bigquery.jobs.listAll and bigquery.jobs.create to query INFORMATION_SCHEMA.JOBS. With bigquery.jobs.list alone, Google redacts the details of jobs that other users created, so cross-user attribution needs bigquery.jobs.listAll. Do not grant dataset mutation, job cancellation, or model administration permissions. See Google’s INFORMATION_SCHEMA.JOBS reference, jobs.list permissions, and BigQuery IAM roles.

Vertex AI logging prerequisite

Vertex AI request-response logging writes requests and responses for Gemini models to a BigQuery table that you choose. Google documents it, as a Preview feature, under the Gemini Enterprise Agent Platform name. Enable it per publisher model with setPublisherModelConfig, setting loggingConfig.enabled to true and loggingConfig.bigqueryDestination to your log table. A samplingRate below 1.0 logs only a sample, so correlation covers only the sampled requests. See Google’s request-response logging guide.

BigQuery labels each Vertex AI request that a BigQuery job makes with bigquery_job_id_prefix: the job ID truncated to its first 63 characters, with uppercase letters converted to lowercase. The log table keeps the provider’s label in the full_request column. Preserve that received label and compare it with LOWER(SUBSTR(job_id, 1, 63)) when joining BigQuery jobs to logged requests. For example, BQ_JOB_ABC joins on bq_job_abc. See Google’s BigQuery generative AI overview.

Content-bearing logs. Read access to a log source is not storage of its content. When you enable request-response logging, the log table stores the full request and response for each sampled call, including prompts and completions. Grant roles/bigquery.dataViewer on that log dataset only. The connector’s queries select only metadata columns (such as logging_time, request_id, model, and api_method), the labels.bigquery_job_id_prefix value from each request, and the token-usage metadata from each response, never the request or response content, and the connector never persists, logs, retries, or exports prompt or completion text. INFORMATION_SCHEMA.JOBS also carries each job’s SQL text, which can contain prompt literals; the connector uses it in memory only to classify AI jobs by their ML function and never stores it.

Setup

  1. Create a service account for the projects that host BigQuery AI workloads.
  2. Grant the three roles above at the scopes shown.
  3. Enable Vertex AI request-response logging to a dedicated dataset, as described above.
  4. Store the service account key as an encrypted credential reference in Settings / Admin / Credentials, or bind a workload identity.
  5. The default lookback is 24 hours; set VENTURI_BIGQUERY_ML_LOOKBACK_HOURS to change it. See sync windows and failure behavior.
  6. In Venturi, go to Integrations / BigQuery ML and run Test connection.

Verification

  • Test connection passes the read-only boundary check: job metadata and log metadata reads succeed, and the service account holds no dataset, job cancellation, or model administration permission.
  • BigQuery remote-model jobs and Vertex AI request logs appear in the latest sync.
  • Non-AI BigQuery jobs are excluded from connector output.
  • Repeated Vertex AI request IDs are deduplicated, so BigQuery processing and Vertex AI inference are not counted twice.
  • Remote-model usage is visible in attribution evidence where matching workload context exists.
  • The connector inventory marks BigQuery ML as read-only.

Rotation and offboarding

Rotate the service account key or workload identity binding on your standard schedule, then update the credential reference in Settings / Admin / Credentials. Removing the connector stops new data-platform AI usage correlation but leaves historical evidence auditable.

Give feedback on this page

Sign in to send feedback. Opens in a new tab.

Copy page details 
Docs feedback: BigQuery ML and Vertex AI usage
Page: https://docs.venturi.systems/connectors/bigquery-ml/