hinna0818/ukbanalytica_v2/ukbsci-preprocess

ukbsci-preprocess — UKB variable cleaning & composite-feature builders

0. RAP guardrails

Strict local-agent boundary: this skill is for script generation, workflow planning, package guidance, and interpretation of aggregate outputs. The agent must not read, inspect, summarize, or process real UK Biobank RAP participant-level data, including de-identified row-level tables, raw RAP fields, exact dates, per-row predictions, row-level SHAP matrices, screenshots, tracebacks, or logs containing row-level values. Generate scripts for the user to run inside RAP; only aggregate results or rendered figures may be shared back with the agent. See ../references/agent-privacy-boundary.md.

The cleaned cohort is participant-level. Stay inside /mnt/project/.... Summary statistics, missingness counts, and pre-vs-post sanity tables are aggregate and may be shared with the local agent.

1. When to load

• Convert UKB negative sentinel codes (-1, -2, -3, -7, -9, -10, -11, -13,

-15) to NA.

• Build composite variables: averaged blood pressure, averaged air pollution,

diet score.

• Apply a predefined variable set (e.g. clinical_core) to a freshly

extracted RAP table.

• Wrap up preprocessing + optional imputation in one call

(prepare_analysis_dataset).

2. When NOT to load

• Pulling fields off RAP → ukbsci-rap-extract.
• Multiple imputation (more than the optional flag here) →

ukbsci-imputation.

• Defining disease cases / survival outcome → ukbsci-cohort.

3. Prerequisites

library(UKBAnalytica); library(data.table)
stopifnot(data.table::is.data.table(dt))

4. Pipeline

Phase 1 — Browse predefined variables

get_variable_info(category = "all")         # 9 categories supported
get_variable_info(category = "biomarkers")
get_variable_sets(category = "lifestyle")    # all sets, with field IDs
get_variable_set("clinical_core", output = "field_id")  # int vector

The catalogues cover: demographics, anthropometrics, lifestyle, socioeconomic, blood_pressure, medications, biomarkers, pollution, diet.

Phase 2 — Sanitize negative codes

dt <- ukb_clean_missing(
  dt,
  cols          = NULL,              # NULL = scan all numeric cols
  invalid_codes = c(-1, -3, -7, -9, -11, -13, -15)
)

This is the first transformation. Run it before computing composite variables — otherwise the means / scores are silently corrupted by sentinels.

Phase 3 — Build composite variables

# Mean SBP / DBP across two readings; falls back to manual if automated missing
dt <- calculate_blood_pressure(dt, type = "sbp", prefer = "auto")
dt <- calculate_blood_pressure(dt, type = "dbp", prefer = "auto")
 
# Average air pollution across time points
dt <- calculate_air_pollution(dt, pollutants = c("NO2","PM10","PM2.5","NOx"))
 
# Healthy diet score 0–7
dt <- calculate_diet_score(
  dt,
  components  = c("fruit","vegetable","fish","meat","cereal","milk"),
  na_handling = "strict"           # "strict" = NA in any component → NA score
)

Phase 4 — Unified call (preprocess_baseline)

For routine workflows:

dt <- preprocess_baseline(
  dt,
  variables       = c("age","sex","bmi","sbp","dbp","ldl_cholesterol",
                       "hba1c","crp","smoking_status","alcohol_intake"),
  custom_mapping  = NULL,
  missing_action  = "keep",        # "drop" filters rows with any NA
  invalid_codes   = c(-1, -3)
)

preprocess_baseline() understands the same variable names as get_variable_info(); passing a name not in the catalogue raises an error.

Phase 5 — Optional imputation

imp <- run_imputation(
  dt,
  variables = c("age","sex","bmi","sbp","ldl_cholesterol","hba1c"),
  m = 5,
  method = "pmm",
  seed = 1234
)

For full multiple-imputation model fitting and pooling, use ukbsci-imputation.

5. Common pitfalls

1. Order matters. Always ukb_clean_missing() before any composite

variable; otherwise sentinel codes contaminate the means.

1. `calculate_diet_score()` rounds to one decimal place in some

versions. If you need higher precision for downstream Cox, scale the raw components yourself.

1. `prefer = "manual"` in calculate_blood_pressure() swaps the

precedence — handy if your cohort has more manual readings than automated.

1. `na_handling = "partial"` in calculate_diet_score() averages

available components and scales — useful but introduces a different missingness mechanism. Document it explicitly.

1. `missing_action = "drop"` in preprocess_baseline() removes any

row with any NA in the listed variables. Confirm this is what the user wants, especially for high-dimensional feature sets.

1. Custom mappings. custom_mapping = list(my_alias = "p123_i0")

lets you rename without touching the catalogue. Useful for non-standard fields (e.g. derived imaging metrics).

1. Predefined variables can drift. When UKB releases new fields, the

catalogue is updated. Run get_variable_info() interactively before freezing scripts.

6. Key functions

See `references/variable-catalog.md` for the 9 categories and example variables.

7. Related skills

8. References

• `references/functions.md`
• `references/variable-catalog.md` — the

9 categories and what's in each

• `references/examples.md`