Ingest Custom Logs into Azure Monitor

Azure Monitor Ingestion SDK for Python explains how to send custom logs into an Azure Monitor Log Analytics workspace through the Logs Ingestion API. It connects a LogsIngestionClient to an ingestion endpoint using DefaultAzureCredential, then uploads records against a Data Collection Rule and named stream. Both synchronous and asynchronous clients are covered.

The workflow requires a target workspace, ingestion endpoint, DCR, and custom table. The DCR defines the schema, transformations, and destination, while its stream identifies the data flow. Endpoint selection can depend on whether the setup uses a DCE or DCR ingestion endpoint. A configured client alone does not establish that the endpoint, rule, or stream is authorized for the intended data.

Use this skill for a Python application that needs to upload custom operational records, including records loaded from a JSON file. The async client is an option for high-throughput scenarios. Confirm scoped identity permissions, retention, and ingestion charges before sending data, and redact personal information and secrets.

Do not upload logs merely to validate the guide. Its historical timestamps are sample data rather than a current live-ingestion probe; approved validation uses synthetic records with current timestamps. Missing destination details, permissions, or schema requirements need clarification before execution.

Inputs include the endpoint, DCR immutable ID, stream name, and a collection of log objects. Environment variables hold the endpoint, rule identifier, and stream configuration. Records must match the DCR column definitions and include TimeGenerated. Custom-table streams use the Custom prefix and table name ending in _CL; built-in-table streams use the Microsoft prefix.

The upload call can report partial failures through an on_error callback. Because that callback can suppress the default exception, collect a bounded, protected failure summary and report incomplete uploads as partial or failed. Retry classified transient failures with bounded backoff, reconciling ambiguous outcomes and possible duplicate ingestion rather than assuming every retry is harmless.

The described SDK batching splits data into chunks of 1 MB or less, compresses them with gzip, and uploads chunks in parallel. Verify the installed version's batching and concurrency. Application input bytes and row counts still need limits because SDK chunking does not bound total memory consumption or ingestion cost. Check Log Analytics for ingestion status and close both client and credential, including failure paths.

Install the ingestion and identity packages using the supplied commands:

pip install azure-monitor-ingestion
pip install azure-identity

Azure Government configuration combines the appropriate authority, ingestion endpoint, and credential scope. Context managers support client cleanup, while asynchronous code uses the corresponding aio ingestion and identity clients.