Accelerate DataFrame Workflows with Polars

This skill provides comprehensive guidance for Polars, the lightning-fast DataFrame library for Python and Rust built on Apache Arrow. It covers expression-based tabular operations (pl.col(), chained transformations), lazy vs eager evaluation (LazyFrame/scan_csv with automatic query optimization, predicate/projection pushdown, and parallel execution vs immediate DataFrame/read_csv execution), common operations (select, filter, with_columns, group_by/aggregations), window functions via .over(), data I/O across CSV/Parquet/JSON/Excel/databases/cloud storage/BigQuery, transformations (joins, vertical/horizontal/diagonal concatenation, pivot/unpivot), and a direct pandas-to-Polars migration guide with an operation mapping table and key syntax pattern comparisons.

It also carries the conceptual differences from pandas that trip up migrators: Polars has no index (integer positions only), enforces strict typing with no silent type conversions, and parallelizes operations by default rather than requiring explicit configuration. Performance guidance includes avoiding Python functions in hot paths (using .map_elements() only when necessary and preferring native expressions), using lf.collect(streaming=True) for very large data, and selecting only needed columns early in a query chain rather than filtering across all columns first.

Use it when you need a faster in-memory DataFrame workflow than pandas for data that still fits in RAM, when building ETL, analytics, or transformation pipelines that benefit from lazy evaluation and parallel execution, or when you want expression-based tabular operations on top of Apache Arrow semantics.

Treat its output as a starting implementation to validate in your own environment, not a substitute for testing or expert review - use it only when the task clearly matches Polars-based data work, and stop to ask for clarification if required inputs or success criteria are missing.

Input: a description of the DataFrame operation, pipeline, or pandas-to-Polars migration needed. Output: runnable Polars Python code using the expression API, including aggregation functions such as pl.len(), pl.col("x").sum(), pl.col("x").mean(), pl.col("x").min()/pl.col("x").max(), and pl.first()/pl.last(), plus conditional expressions built with pl.when(condition).then(value).otherwise(other_value) and null handling via .fill_null(), .is_null(), and .drop_nulls().

lf = pl.scan_csv("file.csv")  # Doesn't read yet
result = lf.filter(pl.col("age") > 25).select("name", "age")
df = result.collect()  # Now executes optimized query

# Add group statistics to each row
df.with_columns(
    avg_age_by_city=pl.col("age").mean().over("city"),
    rank_in_city=pl.col("salary").rank().over("city")
)

Reads and writes CSV, Parquet, JSON, Excel, databases via connectors, cloud storage (S3, Azure, GCS), and Google BigQuery, with pl.write_parquet() called out as the recommended format for performance. Handles joins (inner, left, and joins on differently-named columns), concatenation (vertical, horizontal, and diagonal for differing schemas), and reshaping via pivot()/unpivot(). Bundles detailed reference files covering core concepts, operations, pandas migration, I/O, transformations, and performance best practices, loaded as needed for specific topics.

Data engineers and analysts building ETL or analytics pipelines who need fast, parallel DataFrame operations, and pandas users migrating to Polars who need a direct operation-mapping reference rather than relearning the API from scratch.