Push Warehouse Data to Monte Carlo

This skill generates Python scripts that collect metadata, lineage, and query logs from a data warehouse and push them to Monte Carlo via its push-ingestion API, filling gaps the pull-collector model can't always cover (integrations without query-history access, custom lineage between non-warehouse assets, or customers who already have this data and want to send it directly). The push model works with any data source: templates exist for common warehouses (Snowflake, BigQuery, BigQuery Iceberg, Databricks, Redshift, Hive), and for anything else the skill derives equivalent collection queries from that warehouse's own system catalog or metadata APIs. Pushed data flows through Monte Carlo's integration gateway, dedicated Kinesis streams, and a normalizer into the same downstream systems that power the pull model - only the ingress layer differs.

Generation is template-first and mandatory: before writing any code, the skill reads the matching template under scripts/templates/<warehouse>/ (globbing for it rather than assuming a working-directory-relative path), falling back to the Snowflake template as the canonical reference if no warehouse-specific template exists, adapting only the warehouse-specific collection queries. Templates follow a collect_<flow>.py / push_<flow>.py / collect_and_push_<flow>.py naming pattern for collection-only, push-only, or combined scripts.

The skill treats one specific pycarlo API surface as authoritative and overrides any conflicting training-data assumption about it:

from pycarlo.core import Client, Session
from pycarlo.features.ingestion import IngestionService
from pycarlo.features.ingestion.models import RelationalAsset, AssetMetadata, AssetField, LineageEvent, QueryLogEntry

client = Client(session=Session(mcd_id=key_id, mcd_token=key_token, scope="Ingestion"))
service = IngestionService(mc_client=client)
service.send_metadata(resource_uuid=..., resource_type=..., events=[RelationalAsset(...)])

RelationalAsset is nested (not flat), with type normalized to uppercase "TABLE"/"VIEW" regardless of the warehouse's native casing, and optional volume/freshness sub-objects. Required environment variables are fixed names - MCD_INGEST_ID/MCD_INGEST_TOKEN for push scripts, MCD_ID/MCD_TOKEN for verification, MCD_RESOURCE_UUID for all scripts - never invented alternatives. After any push, the skill must surface the returned invocation_id(s), since that ID is the only way to trace pushed data downstream and is required for the /mc-validate-metadata and /mc-validate-lineage validation commands.

Use it when a customer needs to collect metadata, lineage, freshness, volume, or query-log data from a warehouse or adjacent system and push it into Monte Carlo, especially for sources the pull collector can't reach. Always start from the matching template rather than writing pycarlo imports or SDK calls from memory - the skill's own guidance is explicit that model training data may suggest incorrect names for this API and the documented signatures are the only correct ones.

Input is the target warehouse/platform and its Monte Carlo resource UUID. Output is a ready-to-run Python script that connects to the warehouse, discovers databases/schemas/tables, extracts column and volume/freshness metadata, builds the correct pycarlo objects, pushes them to Monte Carlo, and writes an output manifest containing the invocation ID for later tracing and validation.

Built on the pycarlo SDK's IngestionService (metadata, lineage, and query-log push methods) against Monte Carlo's push-ingestion API, with warehouse-specific templates for Snowflake, BigQuery (including Iceberg/BigLake), Databricks, Redshift, and Hive, plus published reference examples in the mcd-public-resources repo.

Data engineers setting up Monte Carlo observability for warehouses or data sources the pull collector doesn't fully cover, who need a correctly-parameterized pycarlo push script rather than one written from possibly-incorrect memorized API assumptions.