Configure AWS Glue Crawlers for Data Cataloging

This skill covers configuring AWS Glue Crawlers for automated schema discovery across S3, RDS, DynamoDB, and JDBC data sources. It addresses crawler targeting strategy (using specific S3 prefixes instead of broad bucket crawls, exclusion patterns for temporary files and logs, grouping targets that share schemas, and setting appropriate crawl depth) and schema evolution and versioning (schema change detection policies - LOG, UPDATE_IN_DATABASE, DELETE_FROM_DATABASE - plus column addition/deletion policies and backward/forward compatibility settings).

It provides concrete configuration patterns: an optimized crawler setup via boto3 with schema-change and recrawl policies, advanced partitioning configuration (AddOrUpdateBehavior for partitions and tables, table grouping with TableLevelConfiguration), a multi-format S3 crawler using custom classifiers, a JDBC/RDS crawler with path-based exclusions, custom CSV and JSON classifiers, CloudWatch monitoring integration, and retry-wrapped logic for starting a crawler that may already be running.

Use this skill when setting up or tuning AWS Glue Crawlers for data catalog population - designing S3 targeting and exclusion strategy, choosing schema-change policies, building custom classifiers for non-standard CSV or JSON layouts, or applying cost and performance controls (recrawl policy, sample sizes, DPU-hour monitoring) across S3, RDS, DynamoDB, or JDBC sources.

It is not a fit for general AWS data pipeline or ETL job design beyond the crawler stage - the guidance is scoped to crawler configuration, classifiers, and the schema-change, security, and cost settings around them, not to authoring Glue ETL jobs themselves.

Inputs are your data source type and location (S3 paths, RDS/JDBC connection names, existing IAM roles), the schema-change behavior you want, and any non-standard file formats that need custom classifiers. Outputs are boto3-based crawler creation code, classifier definitions, crawler configuration JSON (grouping, partition and table update behavior, logging level), CloudWatch alarm setup, and retry-wrapped crawler-start logic that handles an already-running crawler gracefully.

import boto3

glue_client = boto3.client('glue')

Targets S3, RDS, DynamoDB, and JDBC connections, and integrates with AWS IAM for least-privilege crawler roles, Secrets Manager for JDBC database credentials, CloudWatch for metrics and DPU-hour monitoring, CloudTrail for logging crawler configuration changes, VPC endpoints for private-subnet crawling, and EventBridge for scheduling crawlers during off-peak hours.

Data engineers and platform teams building or maintaining AWS Glue-based data catalogs who need concrete, ready-to-adapt configuration patterns - crawler targeting, schema-evolution policy, custom classifiers for non-standard formats, cost controls, and security hardening with least-privilege IAM and encryption - rather than working out crawler configuration from the AWS documentation alone. It also covers finer cost-control levers: setting sample sizes (100 files by default) based on how uniform your data is, and applying table-level configuration limits to prevent a crawler from creating an excessive number of tables in the catalog.