Manage Azure Data Lake Storage Gen2 with Python SDK

Azure Data Lake Storage Gen2 SDK for Python is a skill for working with ADLS Gen2's hierarchical file system, aimed at big data analytics workloads rather than flat object storage.

pip install azure-storage-file-datalake azure-identity

Authentication uses DefaultAzureCredential against an AZURE_STORAGE_ACCOUNT_URL, building a DataLakeServiceClient. The client hierarchy runs from account-level (DataLakeServiceClient) down through container/file-system (FileSystemClient), directory (DataLakeDirectoryClient), and file (DataLakeFileClient) operations.

Use it when data needs true directory-and-file semantics - nested directories, renames, per-directory access control - rather than the flat key-value model of plain Blob Storage; the skill's own best practices note to consider the Blob API instead for simple object storage use cases. File systems (containers) are created, fetched, deleted, and listed via create_file_system/get_file_system_client/delete_file_system/list_file_systems, and directories support creation (including nested paths in one call), retrieval, deletion, and renaming via rename_directory.

File upload supports both a full local-file or in-memory bytes upload via upload_data, and a chunked pattern for large files using repeated append_data(data, offset, length) calls followed by flush_data(total_length) to commit. Downloads support full-content (download_file().readall()), streaming to a local file handle (readinto), or a partial byte-range download by passing offset/length to download_file, useful for fetching just a slice of a large file rather than the whole object. Listing uses get_paths(), optionally scoped to a subdirectory and optionally recursive=True for a full tree walk, distinguishing files from directories via is_directory. File and directory properties (get_file_properties, returning size and last-modified timestamp) and custom metadata (set_metadata) are also exposed, and directories can be moved or renamed in place via rename_directory(new_name=...) without a separate copy-then-delete step.

Access control is managed per directory: get_access_control/set_access_control read and write owner and POSIX-style permission strings (e.g. rwxr-x---), and update_access_control_recursive applies ACL entries down a directory tree. An async client (azure.storage.filedatalake.aio, paired with azure.identity.aio) mirrors the sync API with await-based calls inside an async with DataLakeServiceClient(...) context - uploading and downloading a file both become awaitable calls, wired up under asyncio.run() - suited to high-throughput scenarios per the skill's best practices, which also recommend setting ACLs at the directory level and letting them inherit to children rather than setting them per-file, and setting custom metadata on files for application-specific attributes.

Python developers building big-data or analytics pipelines on Azure that need real hierarchical file system semantics - nested directories, large chunked uploads, and directory-level access control - rather than flat blob storage.