<< All versions

Skill v1.0.0

currentAutomated scan100/100
lzwei196/kiss-knowledge-infrastructure-for-scientific-simulation/pysteps
──Details
PublishedNovember 1, 2026 at 12:34 AM
Content Hashsha256:8ec1f8214672f617...
Git SHAad02922484de
──Files
Files (1 file, 21.6 KB)
SKILL.md21.6 KBactive
SKILL.md · 392 lines · 21.6 KB

metadata: version: "1.0.0"


MANDATORY EXECUTION POLICY — READ BEFORE PROCEEDING
You MUST run the actual model binary or package described in this document.
If the model fails to import, compile, or execute, you MUST:
1. Check diagnostics/triplets.yaml for a matching error pattern
2. Try the fix suggested in the triplet's remedy section
3. If still failing, report the error to the user with full details
You MUST NOT substitute a simplified Python formula, regression equation,
or hand-coded approximation in place of the real model. Doing so produces
scientifically invalid results and defeats the purpose of the KI.
Before starting, run: python preflight_check.py (in this KI directory)
to verify that the model binary/package and required data are available.
DEBUGGING PROTOCOL — When something goes wrong (model crashes, wrong output,
unexpected values), follow this order. Do NOT skip steps or write debug scripts:
1. Check triplets — diagnostics/triplets.yaml may already cover this error
2. Read official docs — Check the model's own documentation (PDF manual, README,
official examples) for expected input formats, variable names, and units
3. Find working examples — Look in outputs/ for previous successful runs of
this model, or check if the model ships with test/example data
4. Fix the tool — Now that you know what "correct" looks like, make targeted fixes
Resist the urge to write diagnostic/debug Python scripts. The answers are almost
always in the official docs and working examples, not in reverse-engineering the binary.

<!-- KI-MAP:BEGIN (projected by generate_skill_map.py — edit the KI, not this table) -->

KI map — what to read, and when

when you needreadwhy
FIRST, alwayspreflight_check.pyrun it (python preflight_check.py): proves env/binary/data are usable and emits a machine-readable PREFLIGHT_REPORT= line. Do not debug a run that never had a healthy environment.
to run the pipeline stagestools/ (5 tools)the executable pipeline. Read each tool's argparse (--help) before composing a command; SKILL.md's stage table says which tool serves which stage.
before running a stagedocs/s*_*.md (5 stage docs)per-stage procedure, verification and traps — the how-to that SKILL.md's overview compresses.
on ANY error, before debuggingdiagnostics/triplets.yaml (20 entries)symptom → diagnosis → remedy for this model's known failure modes. Check here FIRST; the answer usually exists. Never renumber or rewrite entries.
to know what an output ISdag.yamlthe model's identity: every output's medium, units, validation_rank (1 = the headline variable) and observability. Scoring and obs-binding read THIS — when asked 'what does this model predict', the dag is the answer, not a guess.
when building inputs / parsing outputsdocs/format_spec.yamlexact I/O shapes + known_issues, projected from dag + triplets. Regenerate with ki_tools_common/generate_format_spec.py after changing either — never hand-edit.
to judge a run's skilldocs/validation_convention.yamlhow this model's field judges it validated: per-dag_variable metrics, directions and CITED pass-bands. A run is graded against these, not against intuition.
for claims and thresholdsdocs/gathered_papers.json (16 papers) + docs/papers_index.mdthe literature this KI is judged by; each entry's text_path is fetched full text in the central paper cache. role: benchmark marks the model's own skill paper.
for a machine-readable summaryknowledge_infrastructure.yamlthe manifest (package, pipeline, validation tier, counts) — projected by ki_tools_common/generate_ki_manifest.py; regenerate after structural changes, never hand-edit.

Projected 2026-08-17 from the KI's actual contents — 9 components present. Refresh: `python3 ki_tools_common/generate_skill_map.py --ki_dir <this KI>`. <!-- KI-MAP:END -->

<!-- KI-TOOL-INDEX:BEGIN (projected by generate_skill_map.py — the discoverability contract: every public tool, exact path; PURPOSE stays human-authored elsewhere) -->

Executable tool index (projected — complete by construction)

Every public tool in this KI, by exact path. What each is FOR lives in the human-written Tool Inventory above; --help on any of these prints its arguments.

tool (exact path)invocation
tools/s1_data_import/import_radar_data.pyKISSPATH_PYTHON_ENV/bin/python {KI}/tools/s1_data_import/import_radar_data.py --help
tools/s2_motion_estimation/estimate_motion.pyKISSPATH_PYTHON_ENV/bin/python {KI}/tools/s2_motion_estimation/estimate_motion.py --help
tools/s3_nowcast/run_nowcast.pyKISSPATH_PYTHON_ENV/bin/python {KI}/tools/s3_nowcast/run_nowcast.py --help
tools/s4_verification/verify_nowcast.pyKISSPATH_PYTHON_ENV/bin/python {KI}/tools/s4_verification/verify_nowcast.py --help
tools/s5_postprocess/export_nowcast.pyKISSPATH_PYTHON_ENV/bin/python {KI}/tools/s5_postprocess/export_nowcast.py --help

5 public tools; `_`-prefixed helpers and packaging files excluded. <!-- KI-TOOL-INDEX:END -->


model: pySTEPS version: 1.20.0 domain: precipitation_nowcasting not_a_hydrological_model: true


pySTEPS -- Knowledge Infrastructure

Version: 1.20.0
Domain: precipitation_nowcasting
Last updated: 2026-08-18
Validation status: binary_only

1. Model Identity

PropertyValue
Full namepySTEPS
Version1.20.0
LanguagePython
LicenseSee the installed pySTEPS package metadata
RepositorySee the installed pySTEPS package metadata
CitationSee docs/gathered_papers.json and docs/papers_index.md
Primary domainprecipitation_nowcasting
Spatial modeDistributed gridded radar precipitation fields
Install pathKISSPATH_HOME/.local/lib/python3.12/site-packages/pysteps/
Config filepystepsrc in the install directory

pySTEPS is a Python library, not a standalone binary. The package version is 1.20.0 via importlib.metadata; the module itself has no __version__ attribute.

It is NOT a hydrological model and cannot be validated directly against hydrocraft.db station discharge or water-quality observations.


2. What This Model Does

pySTEPS performs deterministic and probabilistic precipitation nowcasting from radar QPE sequences. It estimates atmospheric precipitation advection with optical-flow methods and advects observed precipitation intensity fields forward on the same radar grid.

This KI treats pySTEPS as a physics/signal-processing library. The validated path exercises the real pySTEPS code paths for Proesmans motion estimation, extrapolation nowcast, and det_cat_fct/fss verification.


3. Input Requirements

Exact shapes live in docs/format_spec.yaml, projected from dag.yaml and diagnostics/triplets.yaml. Regenerate that file after changing either source; do not hand-edit it.

3.1 Radar Precipitation Sequence

VariableUnit model expectsSource datasetSource unitConversion
Precipitation intensitymm/hRadar QPE sequencemm/h or radar-native unitKeep mm/h when already converted; convert radar reflectivity before nowcasting
Radar reflectivitydBZRadar productdBZConvert to mm/h with the Marshall-Palmer relation before nowcasting
Log rainfall ratedBRDerived from precipitation intensitymm/hApply pySTEPS log transform when a method requires log-domain rainfall

Input is a sequence of 2-D precipitation intensity fields on a uniform radar grid, typically 3-4 past frames at 5-15 min cadence. Supported importers include BoM RF3, DWD HDF5, FMI PGM, KNMI HDF5, MCH GIF, OPERA HDF5, and SAF NetCDF.

3.2 Static Inputs

InputSourceTool that prepares it
Radar grid metadataRadar importer metadatatools/ import and postprocess stages
Domain mask / NaN handlingRadar product metadata and stage docsdocs/s1_data_import.md

3.3 Configuration Files

FileFormatNotes
pystepsrcpySTEPS configuration fileLives in the pySTEPS install directory
docs/format_spec.yamlYAMLExact I/O contract projected from the KI

4. Build Instructions

pySTEPS is used as the installed Python package in this environment. Before any run, verify the package, data expectations, and KI structure:

bash
cd KISSPATH_KI_ROOT/pySTEPS/knowledge_infrastructure
python3 preflight_check.py

Known build issues must be handled through diagnostics/triplets.yaml first, then the official pySTEPS docs and working examples.


5. Execution

bash
cd KISSPATH_KI_ROOT/pySTEPS/knowledge_infrastructure
python3 preflight_check.py
python3 diagnostics/run_synthetic_advection.py

The synthetic runner writes diagnostics/last_run.json with per-lead CSI/POD/FAR/FSS scores plus timing. A tester_result.json is also written to the KI root for the orchestrator.

Expected runtime is the local runtime of the synthetic analytical test on a 200x200 grid with 4 past frames and 5 lead times.


6. Output Description

This section restates dag.yaml. If this section and dag.yaml disagree, dag.yaml wins.

Headline output (the dag's validation_rank: 1 variable -- the one this model is judged by):

motion_field -- Estimated 2-D atmospheric radar precipitation advection velocity field (2, ny, nx) from the optical-flow stage on the radar precipitation frame sequence; an intermediate product also returnable on its own. (pixels/timestep)
Output variable (dag var)RankUnitDescription
motion_field1pixels/timestepEstimated 2-D atmospheric radar precipitation advection velocity field (2, ny, nx) from the optical-flow stage on the radar precipitation frame sequence; an intermediate product also returnable on its own.
precipitation_intensitydag outputSee dag.yamlListed by the dag as an output variable.
exceedance_probabilitydag outputSee dag.yamlListed by the dag as an output variable.

7. Tool Inventory

ToolPurposeInputsOutputs
preflight_check.pyVerify environment, package, and required KI dataKI directoryMachine-readable PREFLIGHT_REPORT= line
diagnostics/run_synthetic_advection.pyRun the synthetic analytical validation pathSynthetic advected precipitation sequencediagnostics/last_run.json, tester_result.json, stdout JSON
tools/Executable pipeline stagesRadar frames, metadata, stage argumentsModel-ready inputs, motion fields, nowcasts, verification outputs

Stage Docs

  • docs/s1_data_import.md - import radar frames, metadata, units, and NaN handling.
  • docs/s2_motion_estimation.md - estimate pySTEPS optical-flow motion fields.
  • docs/s3_nowcast.md - run extrapolation, STEPS, or ANVIL nowcasts.
  • docs/s4_verification.md - score forecasts against held-out radar frames.
  • docs/s5_postprocess.md - export nowcasts to NetCDF, GeoTIFF, or CSV.

Before composing a command, read each tool's argparse help and the corresponding stage doc.


8. Unit Conversion Table

pySTEPS operates on 2-D precipitation fields. This table documents the units and conversions encountered in this KI pipeline.

VariableSource unit (verified)Model unitFactor / conversionType
motion_fieldpixels/timesteppixels/timestepx1none
precipitation_intensitymm/hmm/hx1none
exceedance_probabilitydimensionless probabilitydimensionless probabilityx1none
Radar reflectivitydBZmm/hR = (10**(dBZ/10) / 200)**(1/1.6)nonlinear
Log rainfall ratemm/hdBRdBR = 10 * log10(R) with R > 0logarithmic
Accumulated precipitation depthmm/hmmdepth_mm = rate_mm_h * timestep_minutes / 60.0timestep integration

8c. Sign Conventions and Output Units

VariableConvention in this modelCommon alternativeImpact if wrong
motion_field2-D advection velocity in pixels/timestep, shaped (2, ny, nx)Physical velocity such as m/sAdvection displacement is scaled incorrectly
precipitation_intensityInstantaneous or timestep-rate rainfall intensity in mm/hAccumulated depth in mmAccumulation and categorical thresholds are wrong
exceedance_probabilityDimensionless probabilityDeterministic rainfall intensityProbabilistic verification is invalid

Primary units encountered:

UnitDescriptionTypical RangeContext
mm/hRainfall intensity (linear)0 - 100Standard input/output unit for pySTEPS nowcast routines. Motion estimation and extrapolation operate in this space or its log transform.
dBZRadar reflectivity (logarithmic)-10 to 60Raw radar measurement unit. Must be converted to mm/h before nowcasting via the Marshall-Palmer Z-R relation.
dBRLog-transformed rainfall rate-10 to 20Used internally by STEPS cascade decomposition for spectral analysis.
mmAccumulated precipitation depth0 - 200+Obtained by integrating mm/h over time. Common accumulation periods: 5 min, 10 min, 15 min, 1 h, 6 h, 24 h.

Critical unit conversion rules:

  • dBZ to mm/h: R = (10**(dBZ/10) / 200)**(1/1.6) using the default Z=200*R^1.6.
  • mm/h to dBR: dBR = 10 * log10(R); set a threshold first because R > 0 is required.
  • mm/h to mm: depth_mm = rate_mm_h * timestep_minutes / 60.0.
  • Radar frame timestep (metadata['accutime']) is in minutes. Common values are 5 for OPERA and BoM, 10 for FMI, and 15 for DWD.

Accumulation period pitfalls:

  • A "5-minute rainfall rate" in mm/h means the average rate during a 5-min window, not 5 mm/h.
  • To get 1-hour accumulation from 5-min frames, sum 12 consecutive frames in mm/h and multiply each by 5/60.
  • STEPS ensemble output is instantaneous rate at each lead time, not accumulated between lead times.

9. Diagnostic Triplets (Top 5)

Check diagnostics/triplets.yaml before debugging any model crash, wrong output, or unexpected values. The full corpus is there (current count: the KI map at the top of this file); cite the real triplet ids from that file in run notes instead of duplicating or renumbering them here.

#ErrorDiagnosisRemedy
1dt_pysteps_001: unrealistic precipitation values or chaotic motion vectorsInput fields are dBZ but pySTEPS nowcast routines expect mm/hConvert dBZ to mm/h before nowcasting
2dt_pysteps_002: nowcast output values are near zero everywhereAlready-linear mm/h data was passed through an extra dB transformCheck metadata['transform'] and remove the redundant transform
3dt_pysteps_003: accumulated totals are 2-12x too high or too lowRadar frame timestep and assumed accumulation period differMultiply each rate frame by the timestep in hours from metadata['accutime']
4dt_pysteps_004: importer raises file or parse errors for existing radar filesWrong importer was selected for the radar file formatIdentify source and format, then use the matching pySTEPS importer
5dt_pysteps_005: HDF5 radar field is rotated, flipped, or transposedAgency-specific HDF5 axis conventions differVerify orientation and flip or transpose data and metadata when needed

10. Coupling Interfaces

Upstream modelVariable exchangedUnitTemporal resolution
Radar QPE sourceprecipitation_intensity input framesmm/h5-15 min cadence
Downstream modelVariable exchangedUnitTemporal resolution
Hydrologic or impact modelprecipitation_intensity nowcast framesmm/h or accumulated mm after conversionNowcast lead timestep
Warning/verification workflowexceedance_probabilitydimensionless probabilityNowcast lead timestep

For real-data extension, see examples/README.md.


11. Validated Results

Test Case: synthetic_advection

PropertyValue
Comparison typeanalytical
Grid200x200
Synthetic precipitationGaussian blob
Sigma18
Peak intensity8 mm/h
Translationu=2, v=1 px/frame
Past frames4
Lead times5
Motion methodProesmans
Nowcast methodextrapolation
VerificationHeld-out truth frames

Because no real radar archive is mounted, this KI validates pySTEPS against a synthetic advected precipitation sequence where the true future frames are known exactly.

Performance Metrics -- judged against the field's bar, not intuition

The convention bars below restate docs/validation_convention.yaml. A null convention band is written as "no cited threshold"; no substitute threshold is implied.

Dag variableMetricDirectionSatisfactory bandGood bandVery good bandCitation keys
motion_fieldcsimaximizeno cited thresholdno cited thresholdno cited thresholdno cited threshold
motion_fieldendpoint_errorminimizeno cited thresholdno cited thresholdno cited thresholdno cited threshold
exceedance_probabilityroc_aucmaximize0.8 (pulkkinen2019, foresti2016)0.85 (pulkkinen2019, foresti2016)0.9 (pulkkinen2019, foresti2016)pulkkinen2019, foresti2016
exceedance_probabilitybrier_skill_scoremaximize0.0 (foresti2016, flowerdew2014)no cited threshold1.0 (foresti2016, flowerdew2014)foresti2016, flowerdew2014

The synthetic runner reports CSI, POD, FAR at threshold 0.5 mm/h and FSS at scale 20 px. Its synthetic-test expectations are mean CSI > 0.95 and mean FSS > 0.99; these are runner checks, not convention pass-bands for motion_field.

Data Replacement Tracking

ComponentSourceStatusNotes
Radar forcingSynthetic analytical sequenceValidated for the default KI testReal radar archive is not mounted
Motion fieldpySTEPS Proesmans optical flowValidated by the synthetic runnerRank-1 dag output is motion_field
Nowcast framespySTEPS extrapolation nowcastValidated by the synthetic runnerWrites per-lead verification outputs
VerificationpySTEPS det_cat_fct and fssValidated by the synthetic runnerWrites diagnostics/last_run.json

12. Parameter Selection by Region

These are physically informed starting points for pySTEPS runs, not calibration against hydrocraft.db discharge or water-quality observations.

Motion Estimation Parameters

ParameterDefaultRangeDescription
motion_method'LK''LK', 'VET', 'proesmans', 'DARTS'Optical flow algorithm. Lucas-Kanade is robust for most cases; Proesmans is faster for uniform translation; VET and DARTS handle rotational or divergent flow.
fd_method (LK)'shitomasi''shitomasi', 'blob', 'tstorm'Feature detection method for Lucas-Kanade.
maxCorners (LK)500100 - 5000Maximum features to track.
qualityLevel (LK)0.10.001 - 0.5Minimum feature quality.
winSize (LK)(50, 50)(10,10) - (100,100)Lucas-Kanade window size in pixels.
n_iter (VET)31 - 10Number of VET optimization iterations.

Nowcast Parameters

ParameterDefaultRangeDescription
n_leadtimesvaries1 - 72Number of forecast time steps. At 5-min resolution, 12 is 1 hour, 36 is 3 hours, and 72 is 6 hours.
n_ens_members241 - 200Number of STEPS ensemble members. Minimum 20 for stable probability estimates; 48-100 for research; 1 means deterministic.
n_cascade_levels61 - 12Number of cascade decomposition levels. Should satisfy 2^n_levels <= min(nx, ny).
noise_method'nonparametric''nonparametric', 'parametric', NoneStochastic noise generation method for STEPS.
ar_order21 - 3Autoregressive model order for temporal evolution of cascade levels.
extrap_method'semilagrangian''semilagrangian', 'eulerian'Advection scheme.
interp_order11 - 3Interpolation order for semi-Lagrangian advection.
n_iter (extrap)11 - 5Sub-steps per advection step.

Verification Parameters

ParameterDescriptionCommon Values
thr (det_cat_fct)Rainfall threshold for categorical scores in mm/h0.1, 0.5, 1.0, 5.0, 10.0
scale (fss)Spatial scale for Fractions Skill Score in pixels1, 5, 10, 20, 50
scores (det_cat_fct)Categorical scores to compute['CSI', 'POD', 'FAR', 'HSS', 'ETS']

Domain Size Guidelines

Domain (pixels)Recommended n_cascade_levelsMax n_ens_members with 16 GB RAMTypical use case
128 x 1284-6200City-scale, single radar
256 x 2566-7100Regional composite
512 x 5126-848National composite
1024 x 10246-820Continental composite
2048 x 20486-810Full OPERA European composite
All versions