Skill v1.0.0
currentAutomated scan100/100name: azure-storage-file-datalake-py description: Azure Data Lake Storage Gen2 SDK for Python. Use for hierarchical file systems, big data analytics, and file/directory operations. risk: critical source: community date_added: '2026-02-27' metadata: version: "1.0.0"
Azure Data Lake Storage Gen2 SDK for Python
Hierarchical file system for big data analytics workloads.
Installation
pip install azure-storage-file-datalake azure-identity
Environment Variables
AZURE_STORAGE_ACCOUNT_URL=https://<account>.dfs.core.windows.net
Authentication
from azure.identity import DefaultAzureCredentialfrom azure.storage.filedatalake import DataLakeServiceClientcredential = DefaultAzureCredential()account_url = "https://<account>.dfs.core.windows.net"service_client = DataLakeServiceClient(account_url=account_url, credential=credential)
Client Hierarchy
| Client | Purpose | |
|---|---|---|
DataLakeServiceClient | Account-level operations | |
FileSystemClient | Container (file system) operations | |
DataLakeDirectoryClient | Directory operations | |
DataLakeFileClient | File operations |
These are independent fragments, not a run-all script. Confirm account/filesystem/path and protect existing data before writes, renames, recursive ACL changes or deletes. Never use production placeholders as cleanup targets. Bound recursive listings/download bytes and use non-overwriting local outputs. Recursive ACL updates can partially fail: inspect counters/failure details and resume only with exact scope. Close async credentials as well as clients in failure paths; upload success does not establish business correctness.
File System Operations
# Create file system (container)file_system_client = service_client.create_file_system("myfilesystem")# Get existingfile_system_client = service_client.get_file_system_client("myfilesystem")# Deleteservice_client.delete_file_system("myfilesystem")# List file systemsfor fs in service_client.list_file_systems():print(fs.name)
Directory Operations
file_system_client = service_client.get_file_system_client("myfilesystem")# Create directorydirectory_client = file_system_client.create_directory("mydir")# Create nested directoriesdirectory_client = file_system_client.create_directory("path/to/nested/dir")# Get directory clientdirectory_client = file_system_client.get_directory_client("mydir")# Delete directorydirectory_client.delete_directory()# Rename/move directorydirectory_client.rename_directory(new_name="myfilesystem/newname")
File Operations
Upload File
# Get file clientfile_client = file_system_client.get_file_client("path/to/file.txt")# Upload from local filewith open("local-file.txt", "rb") as data:file_client.upload_data(data, overwrite=False)# Upload bytesfile_client.upload_data(b"Hello, Data Lake!", overwrite=False)# Separate append example: create an explicitly new empty file first.file_client = file_system_client.create_file("disposable-append-example.txt")file_client.append_data(data=b"chunk1", offset=0, length=6)file_client.append_data(data=b"chunk2", offset=6, length=6)file_client.flush_data(12) # Commit the data
Download File
file_client = file_system_client.get_file_client("path/to/file.txt")# Download all contentdownload = file_client.download_file()content = download.readall()# Download to filewith open("downloaded.txt", "wb") as f:download = file_client.download_file()download.readinto(f)# Download rangedownload = file_client.download_file(offset=0, length=100)
Delete File
file_client.delete_file()
List Contents
# List paths (files and directories)for path in file_system_client.get_paths():print(f"{'DIR' if path.is_directory else 'FILE'}: {path.name}")# List paths in directoryfor path in file_system_client.get_paths(path="mydir"):print(path.name)# Recursive listingfor path in file_system_client.get_paths(path="mydir", recursive=True):print(path.name)
File/Directory Properties
# Get propertiesproperties = file_client.get_file_properties()print(f"Size: {properties.size}")print(f"Last modified: {properties.last_modified}")# Set metadatafile_client.set_metadata(metadata={"processed": "true"})
Access Control (ACL)
# Get ACLacl = directory_client.get_access_control()print(f"Owner: {acl['owner']}")print(f"Permissions: {acl['permissions']}")# Set ACLdirectory_client.set_access_control(owner="user-id",permissions="rwxr-x---")# Update ACL entriesfrom azure.storage.filedatalake import AccessControlChangeResultdirectory_client.update_access_control_recursive(acl="user:user-id:rwx")
Async Client
from azure.storage.filedatalake.aio import DataLakeServiceClientfrom azure.identity.aio import DefaultAzureCredentialasync def datalake_operations():credential = DefaultAzureCredential()async with DataLakeServiceClient(account_url="https://<account>.dfs.core.windows.net",credential=credential) as service_client:file_system_client = service_client.get_file_system_client("myfilesystem")file_client = file_system_client.get_file_client("test.txt")await file_client.upload_data(b"async content", overwrite=False)download = await file_client.download_file()content = await download.readall()import asyncioasyncio.run(datalake_operations())
Best Practices
- Use hierarchical namespace for file system semantics
- Use `append_data` + `flush_data` for large file uploads
- Review ACL scope: default ACLs affect new descendants; existing descendants require a separately authorized recursive operation
- Use async client for high-throughput scenarios
- Use `get_paths` with `recursive=True` for full directory listing
- Set metadata for custom file attributes
- Consider Blob API for simple object storage use cases
When to Use
This skill is applicable to execute the workflow or actions described in the overview.
Limitations
- Use this skill only when the task clearly matches the scope described above.
- Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
- Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.