Skip to content

File import

Some estates cannot hand Venturi a live credential on day one. The provider API is behind a procurement gate, the billing export lands in someone else’s bucket, or the only record of a gateway’s traffic is a log file an engineer can export. File import is the ingestion path for exactly that situation: you upload a provider export or a log file, and Venturi treats it as a first-class source rather than a workaround.

The contract in one sentence

A file-derived event is processed through the same pipeline as a connector-derived event once it is normalized: same normalization, same pathway classification, same cross-layer deduplication, same attribution stages, same dashboards, same alerts, same exports. It keeps its own provenance markers and import labels throughout, and rows from generic or custom mappings are capped at the Provisional band until they are reconciled against an authoritative source (see what file-derived evidence is worth).

The consequence matters more than the mechanism. No attribution, scoring, dashboard, alarm, or export capability is conditional on having an active programmatic connector. A tenant with zero connectors that imports a supported billing or usage export reaches materialized attribution records and, where the evidence supports it, at least one Chargeback-ready edge.

Formats

The importer accepts two tiers.

Tier Formats
First tier .csv, .tsv, .json, .jsonl / .ndjson, .parquet, .xlsx, each optionally .gz or .zip compressed
Extended tier .avro, .orc, plaintext syslog, CEF, LEEF, and OTLP file exports in JSON or protobuf encoding

An unsupported format fails with a typed error that names the formats that are supported, rather than a partial or best-effort parse. A compressed file and its uncompressed twin produce identical deduplication keys, so how you packaged the export cannot change the result.

Source templates

Rather than making every customer describe their file, the importer ships maintained mapping templates for the exports most estates already produce:

  • AWS Cost and Usage Report, as a file
  • Azure Cost Management export
  • Google Cloud BigQuery billing export, extracted
  • FOCUS v1.2
  • Anthropic and OpenAI usage and cost exports
  • LiteLLM, Helicone, and Portkey gateway logs, and Langfuse log exports
  • Bedrock invocation logs
  • The Power Platform consumption report, the ingestion mechanism behind the Power Platform connector

Each template carries a pin to the upstream schema version it was written against, plus drift tests. When an upstream provider changes its export shape, the contract test fails before the change reaches production rather than after it has quietly corrupted a month of attribution. Each template ingests its pinned fixture to at least the field coverage the equivalent connector achieves.

Files that match no template

For an unrecognized file, the importer infers column types and headers and presents a mapping preview against the invocation-event field dictionary, with per-field confidence. Mappings are saved, versioned, and scoped to your tenant, so the second file of the same shape does not require re-entry. Required-field completeness is surfaced before you commit, and a mapping that leaves required fields unfilled cannot commit those rows.

Nothing is written before you have seen the report

Every import runs a dry run first, producing a downloadable validation report that names the row, the column, the error code, and a suggested fix, with zero writes. You then explicitly commit the validated subset. Per-row outcomes are reported as created, deduplicated, quarantined, or error, and quarantined rows are exportable so they can be corrected.

The correction loop is idempotent: you fix the flagged rows or re-upload a corrected file, and the importer reconciles against the prior attempt by stable key, so corrections do not duplicate rows that already committed.

Re-uploads and overlapping periods

Double-counting is the failure mode that destroys trust in a cost number, so the importer addresses it in two places.

  • File level. Each import persists a manifest: SHA-256 digest, byte size, row count, covered time window, and the mapping or template version used. A byte-identical re-upload is an audited no-op. Re-uploading the same Cost and Usage Report twice changes no total.
  • Row level. Rows that overlap a window you have already imported collapse on a deterministic source event ID minted during normalization. Two exports that overlap by five days produce each invocation exactly once, and post-import reconciliation shows zero double-counted cost.

Timestamps and time zones

Every row timestamp resolves to a time-zone-aware UTC value by one of three routes, in order: a time zone declared in the mapping or template; otherwise an inference that is presented for your explicit confirmation; otherwise rejection. A silent assumption of local time or ingest time is prohibited. Epoch values have their unit detected (seconds, milliseconds, or microseconds) and confirmed.

Period-granular rows, such as daily or monthly billing lines, carry an explicit period label and are priced by the event-time pricing rule. A daily-granularity row is never rendered as though it were a point-in-time invocation, and a file whose timestamps are naive cannot commit until a time zone is confirmed. Re-importing under a corrected time zone supersedes the prior rows instead of duplicating them.

What file-derived evidence is worth

File-derived events carry their own ingestion-layer and source-of-observation markers, with the file manifest ID recorded in the provenance chain, so any materialized figure traces back to the file it came from.

Confidence is not uniform across file sources:

  • Rows ingested through a recognized billing or usage authority template rank as equivalent to cloud-billing-derived evidence. The Chargeback-ready band (coper >= 0.80) is reachable for them, under the same 0.95 policy ceiling that applies everywhere else in the product. See Confidence & evidence.
  • Rows from generic or custom mappings are capped at the Provisional band (0.50 <= coper < 0.80) until they are reconciled against an authoritative source. A generic-mapping row never reaches Chargeback-ready without that reconciliation.

Totals include template-derived rows and never silently exclude any of them.

Freshness for a source that arrives as files

A file source declares a cadence: one_shot, ad_hoc, or scheduled(interval). Staleness watermarks, freshness rendering, degradation transitions, and alert or digest bindings evaluate against the declared cadence rather than against a live-connector assumption.

That distinction keeps a one-off import from being reported as a failing live feed. A one_shot source never enters a degraded state and is labeled with the window it covers. A source declared as scheduled monthly alerts only after its interval plus grace has elapsed, not the moment it stops looking fresh.

Large files and import jobs

Uploads are multipart and resumable up to the published per-file size and row ceilings shown on the import surface, and imports run asynchronously, off the decision-time interceptor’s request path, behind an import-job status surface that reports progress and supports cancel and retry. A canceled job leaves zero committed rows. Import throughput is bounded so that concurrent imports cannot breach the platform’s latency and availability objectives, and a bulk import causes no P99 latency regression on the decision-time interceptor, which keeps its 50 ms P99 budget.

What happens to the contents of your file

The no-content-capture posture applies to uploads, and the importer adds parser hardening on top of it.

  • Prompt, completion, and message-body columns are dropped in memory at the ingest edge before any record is materialized and are never persisted, logged, retried, or exported. Content capture is structural, not configurable: the normalized InvocationEvent schema has no content field and no setting turns capture on.
  • A PII scan runs on any retained free-text field.
  • No cell content is ever evaluated as code, a formula, or a path. A formula-injection spreadsheet is rejected with a typed error and zero partial writes.
  • Parsers enforce decompression-ratio and entity-expansion limits, MIME and type sniffing, and archive-member allowlists. A decompression bomb is rejected, not processed.
  • The upload endpoint is in the attack-surface inventory with an owning control family and security-gate fixtures, and format fuzzing extends to every supported format.

The worked acceptance case is a gateway log that contains prompt bodies: it imports successfully, and the content columns are absent from every store.

Recurring drops from your own storage

Instead of uploading by hand every month, a tenant admin can register a customer-owned object-storage prefix as a recurring file source. The registration is read-only and is fetched only through the mediated access path that governs every other Venturi integration, so it does not widen the trust boundary.

Discovered files flow through the identical pipeline described on this page, including manifest deduplication, and obey the same cadence contract. Dropping a new monthly Cost and Usage Report into the registered prefix ingests it within the declared interval. Revoking the registration stops ingestion and is audited.

How imported data is labeled in the product

Import-derived data is visibly distinct from live-captured data, so nobody reads a retrospective number as a live one. Every surface that renders detections, attributions, coverage, or anomalies derived from imported files displays:

  • the acquisition mode,
  • the time window the imported data covers, with the as-of date of that data, and
  • a per-capability availability state: available, degraded with reduced granularity, or requires live capture.

Alerts computed from imports are labeled retrospective. A capability that genuinely needs live capture shows requires live capture, never a score of zero, because a zero would read as a measured finding rather than an absence of data. Import-derived tiles carry no live-freshness affordances, so the live toggle on the Invocation feed does not appear on them.

Audit trail

Every import job is recorded in the audit trail with the actor, the file manifest, the mapping or template version, a reference to the dry-run report, the per-outcome counts, and the commit decision. Import lineage is queryable, the audit-log explorer filters by import job, and a materialized figure traces back to the manifest of the file that produced it.

Onboarding with files alone

File import appears in the first-run wizard and on the integrations surface as a peer of connecting a source, not as a fallback buried in a settings page. Analytic empty states deep-link into the importer. Where API access has not been granted, representative export files are accepted in place of the connector prerequisites, and the first-attribution checkpoint has a file-based variant.

Import-only onboarding is a first-class supported path, not an error state, and the end-to-end onboarding objective explicitly counts a tenant that has uploaded one provider export and has no connectors at all. The quickstart covers both the connector and the provider-export path.

What this page does not cover

This page describes the import contract. It names no API path, request or response shape, CLI command or flag, configuration key, provider export filename, or numeric size, row, or throughput limit; the import surface publishes those.

Give feedback on this page

Sign in to send feedback. Opens in a new tab.

Copy page details 
Docs feedback: File import
Page: https://docs.venturi.systems/ingestion/file-import/