mitodl/ol-data-platform/ol-dbt-local-dev

Local dbt development (DuckDB + Iceberg) with ol-dbt

ol-dbt local gives you a zero-copy local warehouse: production Glue/Iceberg tables are mounted as DuckDB views, and dbt's ref()/source() fall back to those views for models you haven't built. So you can build old + new models side-by-side against real prod data — without copying it or writing to the warehouse. The default dbt target for all of this is `dev_local` (DuckDB).

Prerequisite: run everything through uv run --frozen

Every command below — and every dbt command in ol-dbt-migration-validation — must resolve dbt from this project's venv:

uv sync                          # once, to create/refresh .venv
source .venv/bin/activate        # then every bare command on this page works
dbt --version                    # expect: dbt 1.12.4, plugin duckdb 1.11.x

The examples below are written bare (`ol-dbt run`, `ol-dbt diff`) and assume you activated the venv. Activation is what puts .venv/bin first on PATH, which is the whole fix — verified: after source .venv/bin/activate, which dbt resolves to this repo's 1.12.4. If you would rather not activate, prefix every command with uv run --frozen (uv run --frozen ol-dbt run ...); what does not work is doing neither, because ol-dbt run and ol-dbt diff both shell out to dbt and will pick up whatever is on your PATH.

Why this is not optional. ol-dbt shells out to bare "dbt" (commands/run.py: cmd = ["dbt", subcommand, ...]), resolved through PATH rather than from the venv ol-dbt itself lives in. Any other dbt earlier on PATH wins. A pipx/pip --user dbt is the usual culprit — one at ~/Library/Python/3.9/bin/dbt (dbt 1.8.1, no duckdb adapter) shadowed the project's 1.12.4 here, and every dev_local command died with:

Error importing adapter: No module named 'dbt.adapters.duckdb'
Runtime Error  Credentials in profile "open_learning", target "dev_local" invalid:
  Could not find adapter type duckdb!

That error names the adapter, not the PATH, so it reads like a broken install. Check with which -a dbt — if the first hit is not this repo's .venv/bin/dbt, prefix with uv run --frozen. Activating the venv (source .venv/bin/activate) works too. --frozen keeps uv from re-resolving and rewriting uv.lock as a side effect of running a command.

Workflow

1. One-time setup

ol-dbt local setup          # bootstrap the local DuckDB + Iceberg env (installs deps, dbt debug)

2. Register production tables as DuckDB views

ol-dbt local register --all-layers      # register all standard dbt layer databases
ol-dbt local register --database ol_warehouse_production_raw   # one layer
ol-dbt local register --force           # re-register everything (default: only new/changed)
ol-dbt local list-sources               # show what's currently registered

register reads AWS Glue + S3, so it needs AWS credentials (unlike the validation commands, which are fully offline). It is incremental by default and parallelized; use --dry-run to preview.

Re-register the layers you are about to read, immediately before every build — not when the staleness warning tells you to. list-sources warns at ">1 day old", and that threshold is tuned for the wrong failure. register stores the Iceberg metadata_location that Glue reports at that instant, and for a dbt-built table Glue routinely points the canonical name at a __dbt_tmp directory. The next production materialization of that model swaps and deletes the directory, so a registration that was correct an hour ago now resolves to nothing:

HTTP Error: HTTP GET error reading
  's3://ol-data-lake-intermediate-production/<model>__dbt_tmp-<hash>/metadata/<n>.metadata.json'
  (HTTP 404 Not Found)

Measured 2026-09-14: a registry refreshed 60 minutes earlier failed exactly this way on int__micromasters__dedp_proctored_exam_grades, killing two mart models. Re-registering only the intermediate layer (41 pointers moved, 118 unchanged) fixed it. Sixty minutes, not the day the warning implies. Re-registering the same three layers 26 hours later moved another 336 pointers (184 staging, 108 intermediate, 44 dimensional) — that is the churn rate you are racing.

The 404 is the lucky outcome. When the __dbt_tmp directory still exists but holds a mid-build snapshot, the view returns duplicated or partial rows and nothing fails — a clean run and wrong numbers, which is far worse when you are about to quote those numbers in a PR. That is why the rule is unconditional re-registration rather than a reaction to an error you can see. #2660 stopped __dbt_tmp tables being registered as sources in their own right, but deliberately did not fix canonical names pointing at __dbt_tmp locations — 621 of 636 dbt-built tables at last count. Until that is fixed, freshness is your responsibility:

# re-register only the layers the models you are about to build actually read
ol-dbt local register --database ol_warehouse_production_staging
ol-dbt local register --database ol_warehouse_production_intermediate
ol-dbt local register --database ol_warehouse_production_dimensional

Read the `✗ Errors:` line — and do not stop there. register exits 0 whatever failed. A per-table failure is counted (✗ Errors: N) and leaves that table on its previous view. A database-level failure is not counted at all: the Glue call raises, register prints ✗ Error accessing database and skips the layer (local_dev.py:824-828), and the summary still reads ✗ Errors: 0. A layer that comes back empty is skipped the same way (⚠ No Iceberg tables found, :832-834). So a run that looks successful can leave you on exactly the stale pointer you were trying to replace, or on a layer that was never touched. Success is all three lines — ✗ Errors: 0, no ✗ Error accessing database, no ⚠ No Iceberg tables found — not the exit status. ol-dbt-migration-validation step 1 has this as a gate script.

One exception, and it is not really an exception. When you are building two relations to compare against each other, both sides must come from the same registration — re-register between them and the difference you measure is drift, not code. That is compatible with the rule above because you build both sides in one invocation: register immediately before it. If you need to rebuild, re-register and rebuild both sides, never one. ol-dbt-migration-validation step 1 states this as the acceptance rule.

The snapshot workflow below is the one shape where the two sides cannot share an invocation — the baseline is already frozen when you rebuild. There the rule becomes one registration for the whole sequence, with no refresh in the middle.

3. Iterate on models

ol-dbt run                  # incremental: rebuild only changed/errored models (state:modified+ result:error+/fail+ --defer)
ol-dbt run --select my_model+   # build a model and its downstream
ol-dbt run --full-refresh   # full rebuild; also re-initialises state for the next incremental run

ol-dbt run saves dbt state under <dbt_project>/.dbt-state/ (manifest.json, run_results.json) so subsequent runs only touch what changed — this is the same slim-CI mechanism used in Phase 2.

4. Teardown

ol-dbt local cleanup-local          # drop locally-registered DuckDB views/tables
ol-dbt local cleanup --dry-run      # preview cleanup of remote dev schemas (namespaced by schema_suffix)

Cleanup honors a PROTECTED_SCHEMAS guard — it will not drop production or shared schemas. Always --dry-run first when cleaning remote schemas.

Typical use: materialize both sides for a diff

ol-dbt local register --all-layers          # mount prod data
ol-dbt run --select "dim_user_old dim_user" --full-refresh   # build both relations on dev_local
ol-dbt diff --old dim_user_old --new dim_user --primary-key user_pk

Quote a multi-model selector. ol-dbt run declares --select as a single str, so an unquoted second model binds to the SUBCOMMAND positional and the command dies with Invalid value "dim_user" for SUBCOMMAND. Choose from: "build", "run", "test" — it never reaches dbt. Note the asymmetry with ol-dbt diff, whose -k/--exclude-columns are list[str] and do take repeated or comma-separated values; that inconsistency is what makes this easy to get wrong.

--full-refresh applies to the selection, so it re-derives both sides in full. Drop it only if neither model is materialized='incremental'; otherwise the diff can compare rows the incremental predicate never reselected. For a model whose grain is more than one column, pass the whole key — comma-separated (-k a,b,c) or by repeating the flag (-k a -k b -k c), which are equivalent. -k a b c does not work. A non-unique key pairs rows many-to-many and reports join artifacts as mismatches.

Typical use: freeze a baseline before an in-place edit

When you change a model's SQL rather than adding a _new copy, snapshot the pre-change build so the diff reflects only your change, not upstream data drift:

ol-dbt local register --database ol_warehouse_production_<layer>  # once, for the whole sequence
ol-dbt run --select my_model --full-refresh             # baseline build, on that registration
ol-dbt local snapshot my_model --as my_model_baseline   # materialize a frozen copy
# ...edit the SQL...
ol-dbt run --select my_model --full-refresh             # do NOT re-register first
ol-dbt diff --old my_model_baseline --old-raw --new my_model --primary-key my_model_pk

Do not re-register before the rebuild — the one place the "re-register before every build" rule is deliberately suspended. The snapshot froze the baseline against one set of Glue pointers; moving them before the rebuild lets upstream drift show up in the diff as if your edit caused it. If the rebuild fails or the sequence drags, redo it from the register rather than refreshing in the middle.

The snapshot is frozen, but the rebuild is not: without --full-refresh an incremental my_model may leave the edited rows untouched, and the diff then reports "no change" for a change that simply never ran. --old-raw is required because the snapshot is a literal table, not a dbt ref()-able model.

Rules