billy-lyc/neuropilot/neuroimage-pre-skill

Brain MRI Data Preprocessing Pipelines

Preprocesses brain MRI data on any SLURM HPC cluster with Apptainer to produce functional connectivity (FC), structural connectivity (SC), and cortical surface metrics.

All pipeline scripts are bundled as assets under scripts/. Each script takes all paths as arguments — no manual path editing is needed. Cluster-specific settings (bind paths, module names, SLURM resources) are centralized in a single config file.

Step 0 — ALWAYS ask first: adult or infant?

When this skill is invoked, the first thing you must do is ask the user whether the subjects are adult or infant / neonate. Do not pick a pipeline or ask anything else until they answer — the population changes which pipelines are valid (infant anatomy needs QSIPrep --infant / infant atlases, not adult sMRIPrep+FreeSurfer).

If the user answers adult → pick by available modalities

Ask which of these matches their data, then read the matching reference.

If the user answers infant / neonate → offer the SC pipeline

The infant route in this skill produces structural connectivity (SC). Ask the user whether they want to run SC; if yes, use:

(For infant functional connectivity, that is a separate skill — fc-infant — not this one.)

Required user inputs

Only `BIDS_DIR` is truly required — ask for it. OUTPUT_DIR, WORK_DIR, LOG_DIR and CONFIG_FILE are fully customizable — the user may give any path they want — but each has a sensible default under the current working directory so they don't have to specify one:

Offer the defaults, let the user override any of them (e.g. put WORK_DIR on fast scratch, OUTPUT_DIR on a project share), then mkdir -p whatever paths are chosen and proceed — don't block on it. CONFIG_FILE still needs the cluster-specific values (containers, FS license) filled in.

Setup

The scripts contain no hardcoded paths — every cluster-specific value lives in your own copy of scripts/pipeline.env.template. After pulling the repo:

cp <skill_path>/scripts/pipeline.env.template ~/pipeline.env
# Edit ~/pipeline.env — fill in every path for your cluster
chmod +x <skill_path>/scripts/*/*.sh <skill_path>/scripts/check_status.sh

The config declares: APPTAINER_BIND (filesystem roots the container must see), module names (MOD_APPTAINER/MOD_FREESURFER/MOD_MRTRIX/MOD_ANTS, empty if on PATH), the container .sif paths, FS_LICENSE, TEMPLATEFLOW_HOME, atlas dirs, and SLURM defaults.

Download the containers

Containers, the FreeSurfer license, and atlas files are not bundled (multi-GB / license-gated). Build each .sif once from its official Docker image, then point the matching variable in pipeline.env at it. Only build the ones your pipeline needs.

Pin these versions. The run scripts pass version-specific CLI flags. The tags

below are the tested, known-working combinations — using older/newer tags can

reintroduce the exact unrecognized arguments / missing-flag crashes these were

chosen to avoid. If you must change a version, re-check the flags in the matching

scripts/*/run_*.sh against that container's --help.

module load apptainer          # or singularity; skip if already on PATH
mkdir -p ~/containers && cd ~/containers
apptainer build fmriprep-24.1.1.sif  docker://nipreps/fmriprep:24.1.1   # fmri_dwi
apptainer build xcp_d-0.10.1.sif     docker://pennlinc/xcp_d:0.10.1     # fmri_dwi
apptainer build smriprep-0.17.0.sif  docker://nipreps/smriprep:0.17.0   # adult struct-only SC
apptainer build qsirecon-1.0.1.sif   docker://pennlinc/qsirecon:1.0.1   # adult SC (recon)
apptainer build qsiprep-0.19.0.sif   docker://pennbbl/qsiprep:0.19.0    # adult SC (DWI preproc)
apptainer build qsiprep-1.1.1.sif    docker://pennbbl/qsiprep:1.1.1     # infant SC (needs --infant, 1.x only)

Version notes (why the pins matter):

• sMRIPrep ≥ 0.17: run_smriprep.sh does not pass --skip-bids-validation

(sMRIPrep 0.17 has no such flag; passing it aborts the job). Fixed in the script.

• Infant QSIPrep must be 1.x (e.g. 1.1.1): --infant and

--subject-anatomical-reference sessionwise are 1.x-only. run_qsiprep.sh (struct_dti) passes both — a 0.x qsiprep will reject them.

• Adult QSIPrep 0.19.x: run_qsiprep.sh (struct_dti_recon) uses --freesurfer-input;

present in 0.16–0.19. The adult and infant SC routes may therefore use different qsiprep .sif files — set QSIPREP_IMG accordingly per route.

FreeSurfer license (free): register at https://surfer.nmr.mgh.harvard.edu/registration.html and set FS_LICENSE.

TemplateFlow (do this right — it is the #1 cause of "0%" batches)

fMRIPrep / sMRIPrep / QSIPrep read MNI (and infant) templates from TEMPLATEFLOW_HOME. Two failure modes bite people:

1. Read-only shared cache — pointing TEMPLATEFLOW_HOME at someone else's dir

(e.g. a group /proj/.../templateflow) fails with PermissionError: ... denied the moment a needed file is missing, because TemplateFlow can't write to repair it.

1. 0-byte stub files — a bare TemplateFlow install seeds empty placeholder files;

on an offline compute node the pipeline can't download the real file and crashes.

Do this instead — pre-populate a cache you own on a login node (which has internet), fetching the real files through the container so versions match:

export TEMPLATEFLOW_HOME=/your/writable/path/templateflow   # a dir YOU own
module load apptainer
# adult SC / fMRI templates (swap the .sif for the one you'll run):
APPTAINERENV_TEMPLATEFLOW_HOME=$TEMPLATEFLOW_HOME \
apptainer exec -B /work,/proj ~/containers/smriprep-0.17.0.sif python -c "
from templateflow import api
for t in ['MNI152NLin2009cAsym','MNI152NLin6Asym','MNI152NLin2009cSym',
          'OASIS30ANTs','fsaverage','fsLR']:
    api.get(t)"
# infant SC additionally needs the infant templates:
APPTAINERENV_TEMPLATEFLOW_HOME=$TEMPLATEFLOW_HOME \
apptainer exec -B /work,/proj ~/containers/qsiprep-1.1.1.sif python -c "
from templateflow import api
for t in ['MNIInfant','UNCInfant','fsLR']: api.get(t)"

Then set TEMPLATEFLOW_HOME in pipeline.env to that path. Verify the key file is a real (non-zero) file, e.g. ls -l $TEMPLATEFLOW_HOME/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_T1w.nii.gz. (Leftover empty stubs for unused atlases/resolutions are harmless — only the templates your pipeline requests must be real.)

Infant --infant: age metadata is REQUIRED

QSIPrep --infant picks the MNIInfant cohort from each participant's age and aborts (RuntimeError: Could not find age for sub-XXX) if it is missing. Provide a per-subject sub-XXX/sub-XXX_sessions.tsv (or a root participants.tsv with an age column):

session_id	age_months
ses-1	1

For term neonates age_months=1 selects the neonate cohort. If your dataset lacks these files, add them (or a symlinked BIDS overlay carrying them) before submitting — otherwise every infant job fails at QSIPrep. Infant runs also need the MNIInfant/UNCInfant templates (see above) and the UNC_ATLAS_DIR / DHCP_ATLAS_DIR atlases for ACT-Atropos.

Checkpoints

Every pipeline step has two checkpoints built into the submission scripts:

1. Pre-run checkpoint: Before submitting jobs, verifies that required input files/directories exist. Prints a summary of found/missing inputs and aborts if critical inputs are missing.
2. Post-run checkpoint: After jobs complete, scans output directories to verify expected files were produced. Reports which subjects succeeded and which need rerun.

Run checkpoints standalone:

# Check inputs before a step
bash <skill_path>/scripts/check_status.sh pre <directory> <expected_file_pattern>
 
# Check outputs after a step
bash <skill_path>/scripts/check_status.sh post <directory> <expected_file_pattern>

DWI input QC (run before any SC/QSIPrep step)

dwi-qc catches the malformed DWI that makes QSIPrep crash deep in the run (missing .bval/.bvec/.json, missing PhaseEncodingDirection, or a bvec/bval direction count that does not match the number of image volumes — i.e. truncated DWI). Run it first and exclude/fix the flagged subjects so you don't submit jobs that are guaranteed to fail. The SC submit_qsiprep.sh scripts also call it automatically as a pre-run warning.

bash <skill_path>/scripts/check_status.sh dwi-qc <bids_dir>

• Portable & dataset-agnostic: globs BIDS-standard */dwi/*_dwi.nii*, checks

the standard sidecars, and reads the volume count straight from the NIfTI header with Python stdlib (gzip+struct) — no FSL/nibabel needed. Verified across single- and multi-shell data and heterogeneous subject-ID schemes.

• Scope: checks structural completeness only. It will not catch image

content problems (motion, low SNR) or rare in-tool runtime issues where the files are structurally valid — use the post-run checkpoint for those.

HPC environment

# Use interactive node for computation (adjust partition name for your cluster)
srun -t 8:00:00 -p interact -N 1 --pty /bin/bash
 
# Job management
squeue -u $USER       # check job status
scancel -u $USER      # cancel all your jobs

Processing strategy for large datasets

• Process in batches of ~500 subjects
• Run cleanup daemons concurrently to delete work directories as jobs finish
• Multiple users can run simultaneously on disjoint subject ranges
• Check completion after each batch before starting the next