datasqrl/sqrl/configure-sqrl

Update the configuration of a DataSQRL project by editing its *-package.json files or creating new ones. Read Configuration Files first (how the files are split and layered), then look up the key you need under Configuration Content.

The configuration supports environment placeholders in the configuration fields that process them. Read Environment Variables before adding credentials or tokens: the normal and secret forms have different compile-time behavior, and support is not universal across every configuration field.

Configuration Files

Base Config and Environment Overlays

For projects that need multiple deployment profiles, split configuration into a base (shared) and thin per-environment overlays to reduce duplication. This is a recommended project convention: SQRL accepts any ordered list of package JSON files and has no special meaning for the filenames or environment names.

• `<project>-shared-package.json` is the base config file that includes common settings to every environment which could be version, enabled-engines, script.main, script.api, and the engines/compiler settings.
• `<project>-<env>-package.json` (one per environment -test, -prod, -local, -dev) — add or overwrite ONLY what differs from the base configuration file (e.g. script.config.environment, test-runner, an engine override). Note that -dev is a local development environment, like -local.

Example for the base config and environment overlays: The lending360 project with a test and a production environment:

`lending360-shared-package.json` — the base: everything both environments share.

{
  "version": "1",
  "enabled-engines": ["flink", "postgres", "kafka", "vertx"],
  "script": {
    "main": "lending360.sqrl",
    "api": {
      "v1": {
        "schema": "lending360-api/schema.v1.graphqls",
        "operations": ["lending360-api/operations.v1.graphql"]
      }
    }
  },
  "engines": {
    "flink": {
      "config": {
        "execution.runtime-mode": "STREAMING",
        "table.exec.source.idle-timeout": "30 s"
      }
    }
  },
  "compiler": {
    "api": { "protocols": ["GRAPHQL", "REST", "MCP"], "endpoints": "FULL" }
  }
}

`lending360-test-package.json` the test overlay: ONLY what test changes.

{
  "script": {
    "config": { "environment": "test" }
  },
  "engines": {
    "flink": {
      "config": { "table.exec.source.idle-timeout": "0 s" } // Disabling source idleness is advised for tests, see the `/test-sqrl` skill
    }
  },
  "test-runner": {
    "delay-sec": -1,
    "required-checkpoints": 1,
    "snapshot-folder": "snapshots/lending360/",
    "test-folder": "lending360-api/tests"
  }
}

`lending360-prod-package.json` the production overlay.

{
  "script": {
    "config": { "environment": "prod" }
  },
  "engines": {
    "flink": {
      "deployment": {
        "jobmanager-size": "small",
        "taskmanager-size": "large.mem",
        "taskmanager-count": 3
      }
    }
  }
}

Rules for writing the configuration files:

• Merging is per field, last file wins. In the example above, the test overlay's table.exec.source.idle-timeout: "0 s" replaces the base's "30 s", while execution.runtime-mode: "STREAMING" from the base is kept. An overlay does not replace the whole engines.flink.config object.
• `test-runner` belongs only in the `-test` overlay.
• Every independently compiled or tested project and sub-project MUST have a named package file for each environment it uses: <project-or-sub-project>-<env>-package.json (for example, fraud_store-test-package.json). Do not use an unqualified standalone package file for a deployable sub-project. The name lets the agent's package-discovery fallback identify the package that belongs to each build.
• If settings are common across environments or sibling sub-projects, share them through a *-shared-package.json base; keep only environment- or sub-project-specific settings in the named package files.
• Fix duplicated configuration: move common settings to <project>-shared-package.json and retain only differing fields in <project>-<env>-package.json.

Sub-projects (multiple deployments in one project)

One project folder may hold several sub-projects, meaning separate deployments over the same source files, each with its own configuration.

Every sub-project has its own per-environment overlays and layers them under a base. The base is either its own or shared with sibling sub-projects. Never combine sub-project specific configurations from different sub-projects.

An available examples directory may include a useful multi-deployment reference. For example, healthcare-study/ in datasqrl-examples illustrates separate stream, API, and analytics deployments alongside a shared data catalog; use it only when it is present and relevant.

fraud_store-shared-package.json      fraud_training-shared-package.json
fraud_store-test-package.json        fraud_training-test-package.json
fraud_store-prod-package.json

Example with one shared base for two sub-projects that differ only in their main script and snapshot folder:

fraud-shared-package.json            # version, enabled-engines, common engines/compiler settings — no script.main
fraud_store-test-package.json        # "script": { "main": "fraud_store.sqrl" }, test-runner with snapshots/fraud_store/
fraud_store-prod-package.json        # "script": { "main": "fraud_store.sqrl" }, prod settings
fraud_training-test-package.json     # "script": { "main": "fraud_training.sqrl" }, test-runner with snapshots/fraud_training/
fraud_training-prod-package.json     # "script": { "main": "fraud_training.sqrl" }, prod settings

/opt/agent/cmd.sh test -r . fraud-shared-package.json fraud_store-test-package.json -b fraud_store
/opt/agent/cmd.sh test -r . fraud-shared-package.json fraud_training-test-package.json -b fraud_training

Each sub-project's -test-package.json is its own test run. To setup a test and run, invoke the /test-sqrl skill.

How to Compile/Test

Compile

Compile by layering base then overlay (merged last-wins):

/opt/agent/cmd.sh compile -r . lending360-shared-package.json lending360-test-package.json -b lending360

The -b <sub-project> flag gives each sub-project its own build folder inside build/, so a compile keeps sub-project's build artifacts intact.

Test Invoke the /test-sqrl skill for which test packages to run and how.

Configuration Content

Engines (enabled-engines, engines)

Enable the minimal set of engines the project actually uses — nothing more. Enabled engines determine what pipeline and deployment artifacts SQRL produces; some engines also provision or run services. Start from flink and add an engine only when a concrete capability below requires it. At the end of the implementation, always check your engine set whether if is minimal or not, and prune engines that are no longer used.

{
  "enabled-engines": ["flink", "postgres", "vertx"]
}

Enable each engine only if its trigger is present:

Common minimal recipes (pick the smallest that fits, then adjust):

• API over a database, no streaming pub/sub → ["flink", "postgres", "vertx"]
• API with subscriptions or mutations → ["flink", "postgres", "kafka", "vertx"]
• Data lake queried from Snowflake, no API → base ["flink", "iceberg", "duckdb"] for tests and local runs, prod overlay ["flink", "iceberg", "snowflake"]
• API over a data lake → ["flink", "iceberg", "duckdb", "vertx"]
• API over a data lake that Redshift also queries → ["flink", "iceberg", "duckdb", "redshift", "vertx"]
• Catalog / schema test → ["flink", "postgres", "vertx"]

Only enable iceberg plus query engines and postgres if both high-volume & high latency and low latency querying is needed in the same project.

Iceberg query engines come in two kinds. duckdb is the full engine: it is connected to the server and runs the API queries and the tests. snowflake, sparksql, redshift and trino are shallow engines: the compiler generates table definitions and query SQL to run separately in those external engines, and they are not integrated with the DataSQRL server. Read iceberg-query.md before choosing between them.

Configuring the selected engines

Enabled engines are configured under the engines field. Engine settings that hold for every environment go in the base <project>-shared-package.json; only the ones that differ go in an environment overlay, checkout Base Config and Environment Overlays.