astronomer/agents/configuring-airflow-language-sdks

Configuring Airflow for Language SDKs

To run language SDK tasks, Airflow needs to know two things: which coordinator launches the native subprocess, and which queue routes to that coordinator. The mechanism is identical across every language SDK — only each coordinator's classpath and kwargs differ. This skill documents the shared wiring once, then the per-coordinator options. It is platform-neutral: the same settings apply on open-source Airflow and on managed platforms like Astro.

Experimental. The language SDKs are in preview; configuration keys may change.

For the task code, see authoring-language-sdk-tasks (and the per-language authoring skill, e.g. authoring-java-sdk-tasks, authoring-go-sdk-tasks). For building and shipping the artifact, see the per-language deploy skill (e.g. deploying-java-sdk-bundles, deploying-go-sdk-bundles).

Prerequisites on the worker

• The runtime or artifact the SDK needs must be present on the worker nodes, because the coordinator spawns a native subprocess per task instance. The exact requirement is per-SDK — see Per-coordinator options (the Java SDK needs a JRE 17+; the Go SDK needs no language runtime — the bundle is a self-contained native executable, but it must be built for the worker's OS/arch).
• The compiled/native artifact(s) must be reachable on the worker. See the per-language deploy skill.
• The coordinators ship with the Airflow Task SDK (apache-airflow-task-sdk, installed with Airflow). No extra Python package is required.

The two settings

Both live in the [sdk] configuration section and apply to every language SDK:

1. `coordinators` — a JSON object mapping a coordinator name you choose to its implementation (classpath) and constructor kwargs.
2. `queue_to_coordinator` — a JSON object mapping a task queue to a coordinator name.

A task whose stub sets queue="..." is handed to the named coordinator, which launches the native subprocess. The coordinator name is arbitrary — it just has to be the same string in both settings. The queue name must match the queue= set on the Python @task.stub.

Option A: airflow.cfg

[sdk]
coordinators = {
  "java-jdk17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {"jars_root": ["/opt/airflow/jars"]}
  },
  "go": {
    "classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator",
    "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}
  }
}
queue_to_coordinator = {"java": "java-jdk17", "golang": "go"}

Option B: environment variables

Each value must be valid one-line JSON. This form is convenient for containers, .env files, Docker Compose, and Helm.

export AIRFLOW__SDK__COORDINATORS='{"java-jdk17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/opt/airflow/jars"]}}, "go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}'
export AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "java-jdk17", "golang": "go"}'

The examples above register multiple coordinators at once (one per language) and map a different queue to each — register only the ones you use.

Per-coordinator options

The classpath and kwargs are specific to each coordinator. Add a subsection here as new language SDKs land.

JavaCoordinator

• `classpath`: airflow.sdk.coordinators.java.JavaCoordinator
• Worker runtime: JRE 17+ (java on PATH, or set java_executable).

Java logging via `java.util.logging`. Of the SDK logging integrations, only JPL and SLF4J are zero-config build dependencies; Log4j 2 and JUL need extra setup — see the logging integration section in deploying-java-sdk-bundles. JUL's documented alternative to calling AirflowJulHandler.setup() in main() is a logging.properties file, wired through jvm_args:

[sdk]
coordinators = {
  "java-jdk17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {
      "jars_root": ["/opt/airflow/jars"],
      "jvm_args": ["-Djava.util.logging.config.file=/opt/airflow/logging.properties"]
    }
  }
}

ExecutableCoordinator (Go and other self-contained-executable SDKs)

• `classpath`: airflow.sdk.coordinators.executable.ExecutableCoordinator
• Worker runtime: none beyond the bundle itself. The bundle is a self-contained native executable (AFBNDL01), so it needs no language runtime, but it must be built for the worker's OS/arch (a mismatch fails with exec format error).

(Future coordinators — for other languages — will list their own `classpath`, runtime, and `kwargs` here.)

Verifying the configuration

1. Confirm the runtime/artifact is usable where workers run — for the Java SDK, java -version via astro dev bash or docker compose exec ...; for the Go SDK, the packed bundle exists and matches the worker's OS/arch.
2. Confirm the artifact directory referenced in kwargs (e.g. jars_root, executables_root) actually contains your artifact on the worker filesystem.
3. Trigger the DAG and open the native task's logs — you should see the subprocess start and your task output.

Troubleshooting

Related Skills

• authoring-language-sdk-tasks: The shared Python-stub pattern and conceptual model.
• authoring-java-sdk-tasks: Java task code and matching Python stubs.
• deploying-java-sdk-bundles: Build the bundle and put the artifact where the coordinator scans.
• authoring-go-sdk-tasks: Go task code and matching Python stubs.
• deploying-go-sdk-bundles: Build/pack the Go bundle and place it where the coordinator scans.
• deploying-airflow: General deployment of Airflow on Astro, Docker Compose, or Kubernetes.