Skill v1.0.0
Trusted Publisher100/100version: "1.0.0" name: dataflows-authoring-cli description: > Create, update, delete, and manage Fabric Dataflows Gen2 artifacts with Power Query M mashup definitions via CLI (az rest / curl). Uses az rest and curl against the Fabric REST API to author definitions containing base64-encoded mashup.pq, queryMetadata.json, and .platform parts. Supports creating dataflows with inline definitions, modifying mashup queries, binding connections, triggering Execute refresh jobs with typed parameter overrides, and exporting definitions for CI/CD. Use when the user wants to: (1) create a new Dataflow Gen2 with Power Query M queries, (2) update a dataflow mashup definition, (3) trigger a dataflow refresh job, (4) bind or manage dataflow connections, (5) set up CI/CD via definition export and import, (6) delete a dataflow, (7) configure staging destinations. Triggers: "create dataflow", "author dataflow", "Power Query M", "mashup document", "update dataflow definition", "refresh dataflow", "dataflow connection", "ETL dataflow", "dataflow CI/CD".
Update Check — ONCE PER SESSION (mandatory)The first time this skill is used in a session, run the check-updates skill before proceeding.- GitHub Copilot CLI / VS Code: invoke thecheck-updatesskill.- Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.- Skip if the check was already performed earlier in this session.
CRITICAL NOTES1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
Dataflows Gen2 — Authoring via CLI
Table of Contents
| Task | Reference | Notes | |
|---|---|---|---|
| Finding Workspaces and Items in Fabric | COMMON-CLI.md § Finding Workspaces and Items in Fabric | Mandatory — READ link first [needed for finding workspace id by its name or item id by its name, item type, and workspace id] | |
| Fabric Topology & Key Concepts | COMMON-CORE.md § Fabric Topology & Key Concepts | ||
| Environment URLs | COMMON-CORE.md § Environment URLs | ||
| Authentication & Token Acquisition | COMMON-CORE.md § Authentication & Token Acquisition | Wrong audience = 401; read before any auth issue | |
| Core Control-Plane REST APIs | COMMON-CORE.md § Core Control-Plane REST APIs | Includes pagination, LRO polling, and rate-limiting patterns | |
| Definition Envelope | ITEM-DEFINITIONS-CORE.md § Definition Envelope | Definition payload structure | |
| Per-Item-Type Definitions | ITEM-DEFINITIONS-CORE.md § Per-Item-Type Definitions | Support matrix, decoded content, part paths — REST specs, CLI recipes | |
| Job Execution | COMMON-CORE.md § Job Execution | ||
| Tool Selection Rationale | COMMON-CLI.md § Tool Selection Rationale | ||
| Authentication Recipes | COMMON-CLI.md § Authentication Recipes | az login flows and token acquisition | |
Fabric Control-Plane API via az rest | COMMON-CLI.md § Fabric Control-Plane API via az rest | Always pass `--resource`; includes pagination and LRO helpers | |
| Item CRUD Operations | COMMON-CLI.md § Item CRUD Operations | Create, get/update definition, delete patterns | |
| Job Execution (CLI) | COMMON-CLI.md § Job Execution | ||
| Gotchas & Troubleshooting (CLI-Specific) | COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific) | az rest audience, shell escaping, token expiry | |
| Quick Reference | COMMON-CLI.md § Quick Reference | az rest template + token audience/tool matrix | |
| Authoring Capability Matrix | DATAFLOWS-AUTHORING-CORE.md § Authoring Capability Matrix | Full CRUD lifecycle, parameterized refresh, Git CI/CD | |
| Dataflow Definition Structure | DATAFLOWS-AUTHORING-CORE.md § Dataflow Definition Structure | Read first — 3-part definition: queryMetadata.json, mashup.pq, .platform | |
| REST API Surface (Authoring) | DATAFLOWS-AUTHORING-CORE.md § REST API Surface | Create, getDefinition, updateDefinition, delete, run job | |
| Power Query M Code Structure | DATAFLOWS-AUTHORING-CORE.md § Power Query M Code Structure | Section documents, multiple queries, parameters, fast copy | |
| Connection Model | DATAFLOWS-AUTHORING-CORE.md § Connection Model | Connection kinds, path patterns, connectionId requirements | |
| Job Execution Patterns | DATAFLOWS-AUTHORING-CORE.md § Job Execution Patterns | Trigger refresh, LRO polling, parameterized refresh | |
| ALM / Git Integration | DATAFLOWS-AUTHORING-CORE.md § ALM / Git Integration | Export/import, CI/CD pipeline, cross-environment overrides | |
| Gotchas and Troubleshooting | DATAFLOWS-AUTHORING-CORE.md § Gotchas and Troubleshooting | 15-row issue/cause/resolution table | |
| Quick Reference: Authoring Decision Guide | DATAFLOWS-AUTHORING-CORE.md § Quick Reference | Scenario → recommended approach lookup | |
| Core Authoring via CLI | authoring-cli-quickref.md § Core Authoring via CLI | Create, get/update definition, delete, refresh az rest one-liners | |
| Base64 Encoding Helpers | authoring-cli-quickref.md § Base64 Encoding Helpers | Encode/decode definition parts in bash and PowerShell | |
| Definition Manipulation | authoring-cli-quickref.md § Definition Manipulation Patterns | Read-modify-write workflow for definitions | |
| Bash Templates | authoring-script-templates.md § Bash Templates | Create dataflow, read-modify-write, refresh with LRO, CI/CD export/import | |
| PowerShell Templates | authoring-script-templates.md § PowerShell Templates | Create dataflow, trigger refresh with polling | |
| Tool Stack | SKILL.md § Tool Stack | az CLI + jq + base64; verify before first op | |
| Connection | SKILL.md § Connection | Workspace/dataflow ID discovery via REST | |
| Agentic Workflows | SKILL.md § Agentic Workflows | Start here — discover→formulate→execute→verify | |
| Gotchas, Rules, Troubleshooting | SKILL.md § Gotchas, Rules, Troubleshooting | MUST DO / AVOID / PREFER checklists | |
| Agent Integration Notes | authoring-cli-quickref.md § Agent Integration Notes | Platform-specific tips (Copilot CLI, Claude Code) |
Tool Stack
| Tool | Role | Install | |
|---|---|---|---|
az CLI | Primary: Auth (az login), REST API calls (az rest), token acquisition. | Pre-installed in most dev environments | |
jq | Parse and manipulate JSON responses and definition payloads. | Pre-installed or trivial | |
base64 | Encode/decode definition parts for the REST API. | Built into bash / [Convert]::ToBase64String() in PowerShell | |
curl | Alternative to az rest when raw HTTP control is needed. | Pre-installed |
Agent check — verify before first operation:```bashaz --version 2>/dev/null || echo "INSTALL: https://aka.ms/install-azure-cli"jq --version 2>/dev/null || echo "INSTALL: apt-get install jq OR brew install jq"```
Connection
Discover Workspace and Dataflow IDs
Per COMMON-CLI.md Finding Workspaces and Items in Fabric:
# List workspaces — find workspace ID by nameaz rest --method get \--resource "https://api.fabric.microsoft.com" \--url "https://api.fabric.microsoft.com/v1/workspaces" \--query "value[?displayName=='MyWorkspace'].id" --output tsv# List dataflows in workspace — find dataflow ID by nameWS_ID="<workspaceId>"az rest --method get \--resource "https://api.fabric.microsoft.com" \--url "https://api.fabric.microsoft.com/v1/workspaces/$WS_ID/dataflows" \--query "value[?displayName=='MyDataflow'].id" --output tsv
Reusable Connection Variables
WS_ID="<workspaceId>"DF_ID="<dataflowId>"API="https://api.fabric.microsoft.com/v1"RESOURCE="https://api.fabric.microsoft.com"
Agentic Workflows
Discover → Formulate → Execute → Verify
- Discover → List workspaces, list dataflows, get definition (decode to inspect M code and metadata).
- Formulate → Create or modify M code (
mashup.pq), updatequeriesMetadata, prepare all 3 definition parts. - Execute → Create or update dataflow via REST API, optionally trigger refresh.
- Verify → Poll LRO for completion, get definition again to confirm changes, check refresh status.
# 1. Discover — get current definitionRESULT=$(az rest --method post \--resource "$RESOURCE" \--url "$API/workspaces/$WS_ID/dataflows/$DF_ID/getDefinition")# Decode mashup.pq to inspect current queriesecho "$RESULT" | jq -r '.definition.parts[] | select(.path=="mashup.pq") | .payload' | base64 -d# 2. Formulate — edit M code, prepare new definition parts# (modify the decoded mashup, re-encode, build JSON payload)# 3. Execute — update definitionaz rest --method post \--resource "$RESOURCE" \--url "$API/workspaces/$WS_ID/dataflows/$DF_ID/updateDefinition?updateMetadata=true" \--body @definition.json# 4. Verify — trigger refresh and pollLOCATION=$(az rest --method post \--resource "$RESOURCE" \--url "$API/workspaces/$WS_ID/items/$DF_ID/jobs/Execute/instances" \--headers "Content-Length=0" \--output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r')
Gotchas, Rules, Troubleshooting
For full authoring gotchas: DATAFLOWS-AUTHORING-CORE.md Gotchas and Troubleshooting. For CLI-specific issues: COMMON-CLI.md Gotchas & Troubleshooting (CLI-Specific).
MUST DO
- `az login` first — all
az restcalls use the active session. No session → 401. - Always `--resource "https://api.fabric.microsoft.com"` — wrong audience = 401.
- Base64-encode all definition parts — REST API expects
payloadType: "InlineBase64". - Send all 3 parts in updateDefinition — it's a full replacement, not incremental.
- Handle LRO polling —
getDefinition,updateDefinition(with def), and job execution return 202; poll viaLocationheader. - Verify workspace has capacity —
GET /v1/workspaces/{id}and checkcapacityId. - Set `formatVersion` to `"202502"` in
queryMetadata.json. - Set `loadEnabled: true` for queries that should write output on refresh.
AVOID
- Hardcoded GUIDs — discover workspace/dataflow IDs via REST API (Connection section).
- Skipping base64 encoding — payloads will be rejected or corrupted.
- Partial definition updates — sending 1 or 2 parts silently drops queries.
- Creating dataflows with unbound connections — queries referencing nonexistent
connectionIdwill fail at refresh. - Using GET for `getDefinition` — it's a POST endpoint; GET returns 405.
- Constructing operation URLs manually — always use the
Locationheader from 202 responses. - Duplicate `displayName` values — not enforced but causes confusion.
PREFER
- `az rest` over raw `curl` — handles token acquisition and refresh automatically.
- `getDefinition` before `updateDefinition` — read-modify-write prevents accidental data loss.
- Validate M code before creating dataflow — syntax errors won't surface until refresh.
- `jq` for JSON manipulation — build definition payloads programmatically.
- `"Automatic"` for parameter type in job execution — lets engine infer from definition.
- `?updateMetadata=true` on
updateDefinition— ensures.platformchanges (display name) are applied. - Env vars (
WS_ID,DF_ID,API,RESOURCE) for script reuse.
TROUBLESHOOTING
| Symptom | Fix | |
|---|---|---|
| 401 Unauthorized | Verify az login is active; check --resource "https://api.fabric.microsoft.com" | |
| 405 Method Not Allowed on getDefinition | Use POST, not GET — getDefinition follows LRO pattern | |
| updateDefinition silently drops queries | Send all 3 parts (queryMetadata.json, mashup.pq, .platform) | |
| Refresh fails on new dataflow | Ensure connectionId values exist in user's Fabric connection store | |
| formatVersion mismatch error | Set formatVersion to "202502" in queryMetadata.json | |
| Query doesn't produce output | Set loadEnabled: true in queriesMetadata | |
| Fast copy not engaged | Add [StagingDefinition = [Kind = "FastCopy"]] before section in mashup.pq | |
| LRO polling returns 404 | Use Location header URL — don't construct operation URLs manually | |
| 429 Too Many Requests | Respect Retry-After header; implement exponential backoff | |
| base64 decode produces garbage | Ensure no trailing newlines; use base64 -w0 (Linux) for encoding |
Examples
Example 1: Create a Dataflow Gen2 with Inline Definition
# 1. Prepare mashup.pqMASHUP='section Section1;shared Output = letSource = #table(type table [Name = text, Value = number],{{"Alpha", 100}, {"Beta", 200}, {"Gamma", 300}})in Source;'# 2. Prepare queryMetadata.jsonQUERY_META='{"formatVersion": "202502","queriesMetadata": {"Output": { "queryId": "00000000-0000-0000-0000-000000000001","queryName": "Output", "loadEnabled": true }}}'# 3. Prepare .platformPLATFORM='{"$schema":"https://developer.microsoft.com/json-schemas/fabric/gitIntegration/platformProperties/2.0.0/schema.json","metadata":{"type":"Dataflow","displayName":"MyDataflow"},"config":{"version":"2.0","logicalId":"00000000-0000-0000-0000-000000000001"}}'# 4. Base64-encode all partsMASHUP_B64=$(echo -n "$MASHUP" | base64 -w0)META_B64=$(echo -n "$QUERY_META" | base64 -w0)PLAT_B64=$(echo -n "$PLATFORM" | base64 -w0)# 5. Create the dataflowaz rest --method post \--url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items" \--resource "https://api.fabric.microsoft.com" \--headers "Content-Type=application/json" \--body "{\"displayName\": \"MyDataflow\",\"type\": \"Dataflow\",\"definition\": {\"parts\": [{\"path\": \"mashup.pq\", \"payload\": \"${MASHUP_B64}\", \"payloadType\": \"InlineBase64\"},{\"path\": \"queryMetadata.json\", \"payload\": \"${META_B64}\", \"payloadType\": \"InlineBase64\"},{\"path\": \".platform\", \"payload\": \"${PLAT_B64}\", \"payloadType\": \"InlineBase64\"}]}}"
Example 2: Trigger a Refresh Job
# Trigger refresh (returns 202 + Location header for polling)LOCATION=$(az rest --method post \--url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items/${DF_ID}/jobs/instances?jobType=Pipeline" \--resource "https://api.fabric.microsoft.com" \--headers "Content-Length=0" \--output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r')# Poll until completewhile true; doSTATUS=$(az rest --method get --url "$LOCATION" \--resource "https://api.fabric.microsoft.com" --query "status" -o tsv)echo "Status: $STATUS"[[ "$STATUS" == "Completed" || "$STATUS" == "Failed" ]] && breaksleep 10done
Example 3: Update a Dataflow Definition
# Get current definition (POST, not GET — returns 202)LOCATION=$(az rest --method post \--url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items/${DF_ID}/getDefinition" \--resource "https://api.fabric.microsoft.com" \--headers "Content-Length=0" \--output none --include-response-headers 2>&1 | grep -i "^location:" | awk '{print $2}' | tr -d '\r')# Poll for the definitionDEF=$(az rest --method get --url "${LOCATION}" \--resource "https://api.fabric.microsoft.com")# Decode, modify, re-encode, then updateDefinition with all 3 partsaz rest --method post \--url "https://api.fabric.microsoft.com/v1/workspaces/${WS_ID}/items/${DF_ID}/updateDefinition" \--resource "https://api.fabric.microsoft.com" \--headers "Content-Type=application/json" \--body "{\"definition\": {\"parts\": [ ... ]}}"