Output Structure

Where everything lands and what each file contains.

The output tree is organised by correction stage and by purpose. Every NetCDF carries CF-1.8 attributes - use ncdump -h <file> or xr.open_dataset(<file>).attrs to inspect.

Top-level layout

data/output/                                      (or data/example_bali/output/ for Bali)
  corrected_ls/
  corrected_lseqm/
  corrected_lseqmdl/
  trained_models/
  station_density/
  metrics_ls/
  metrics_lseqm/
  metrics_lseqmdl/
  quality_ls/
  quality_lseqm/
  quality_lseqmdl/
  station_validation/
  figures/

Corrected precipitation NetCDFs

File pattern:

{prefix}_{method_abbr}_corrected_imergl_month{MM}_dekad{D}.nc4

Variable	Type	Units	Description
`precipitation`	float32	mm/day	Corrected daily precipitation
`lat`, `lon`	float64	deg	Cell-centre coordinates, 0.1 deg grid
`time`	datetime64	UTC	Daily timestamps for the dekad x all years

Global attributes record the method (ls, lseqm, lseqmdl), the source IMERG / CPC files, the version of the framework, and the run timestamp.

Trained model artefacts

trained_models/
  bias_correction_model_month{MM}_dekad{DD}.keras

Loading: tf.keras.models.load_model(path). The model name carries no filename prefix, and there is no normalisation sidecar: each daily field is normalised by its own maximum at inference time and denormalised by the same value, so the .keras file is self-sufficient.

Confidence mask

station_density/confidence_mask_station_density.nc4

Variable: confidence (float32, 0-1). Computed once per AOI; cached and re-used across all dekads.

Metrics and QA

For each method:

metrics_{method}/
  {prefix}_metricssd_{ref}_imergl_{method}_month{MM}_dekad{DD}.nc4    # 31 metrics per pixel
quality_{method}/
  {prefix}_qualitysd_{ref}_imergl_{method}_month{MM}_dekad{DD}.nc4    # CQI + 3 components

{ref} is the reference dataset (cpc, imergl, or imergf), so each method gets nine metric files per dekad. The sd tag marks single-dekad mode; timeseries mode writes metricsts / qualityts instead.

The metrics NetCDF holds the 31-variable set drawn from WMO/TD-1485 + WMO-1317: relative bias, Pearson r, RMSE, MAE, reference and test standard deviation and their ratio, POD, FAR, CSI, wet-day frequency and mean wet-day precipitation (reference and test), NSE, maximum dry-spell length (reference and test), the KS statistic and its p-value, and the 25th / 50th / 75th / 90th / 95th / 99th percentile values for reference and test. The quality NetCDF is the aggregated CQI for plotting.

Station validation

Written by 05_station_validation.ipynb. These filenames carry no prefix:

station_validation/
  station_validation_{method}_month{MM}_dekad{DD}.csv          # one row per station, 31 metrics
  station_validation_summary_month{MM}_dekad{DD}.csv           # all methods, one dekad
  station_multi_threshold_{method}_month{MM}_dekad{DD}.csv      # WMO threshold scores per station
  multi_threshold_summary_{method}_month{MM}_dekad{DD}.csv      # threshold scores aggregated

Taylor diagram PNGs do not land here - they go to figures/taylor/.

Figures

figures/
  qa/                  QA spatial maps, one subdirectory per plot type
  taylor/              Pooled and per-dekad Taylor diagrams
  station_validation/  Per-station scatter, threshold curves, regional boxplots

Under figures/qa/ each plot type gets its own subdirectory named {quality_prefix}_{plot_type}/. Plot types include spatial comparison strips, CQI maps, and station scatter. The visualisation hub notebook (06_visualisation_hub.ipynb) generates all of these.

--- title: "Output Structure" description: "Where everything lands and what each file contains." --- The output tree is organised by correction stage and by purpose. Every NetCDF carries CF-1.8 attributes - use `ncdump -h <file>` or `xr.open_dataset(<file>).attrs` to inspect. ## Top-level layout ``` data/output/ (or data/example_bali/output/ for Bali) corrected_ls/ corrected_lseqm/ corrected_lseqmdl/ trained_models/ station_density/ metrics_ls/ metrics_lseqm/ metrics_lseqmdl/ quality_ls/ quality_lseqm/ quality_lseqmdl/ station_validation/ figures/ ``` ## Corrected precipitation NetCDFs File pattern: ``` {prefix}_{method_abbr}_corrected_imergl_month{MM}_dekad{D}.nc4 ``` | Variable | Type | Units | Description | |----------|------|-------|-------------| | `precipitation` | float32 | mm/day | Corrected daily precipitation | | `lat`, `lon` | float64 | deg | Cell-centre coordinates, 0.1 deg grid | | `time` | datetime64 | UTC | Daily timestamps for the dekad x all years | Global attributes record the method (`ls`, `lseqm`, `lseqmdl`), the source IMERG / CPC files, the version of the framework, and the run timestamp. ## Trained model artefacts ``` trained_models/ bias_correction_model_month{MM}_dekad{DD}.keras ``` Loading: `tf.keras.models.load_model(path)`. The model name carries no filename prefix, and there is no normalisation sidecar: each daily field is normalised by its own maximum at inference time and denormalised by the same value, so the `.keras` file is self-sufficient. ## Confidence mask ``` station_density/confidence_mask_station_density.nc4 ``` Variable: `confidence` (float32, 0-1). Computed once per AOI; cached and re-used across all dekads. ## Metrics and QA For each method: ``` metrics_{method}/ {prefix}_metricssd_{ref}_imergl_{method}_month{MM}_dekad{DD}.nc4 # 31 metrics per pixel quality_{method}/ {prefix}_qualitysd_{ref}_imergl_{method}_month{MM}_dekad{DD}.nc4 # CQI + 3 components ``` `{ref}` is the reference dataset (`cpc`, `imergl`, or `imergf`), so each method gets nine metric files per dekad. The `sd` tag marks single-dekad mode; timeseries mode writes `metricsts` / `qualityts` instead. The metrics NetCDF holds the 31-variable set drawn from WMO/TD-1485 + WMO-1317: relative bias, Pearson r, RMSE, MAE, reference and test standard deviation and their ratio, POD, FAR, CSI, wet-day frequency and mean wet-day precipitation (reference and test), NSE, maximum dry-spell length (reference and test), the KS statistic and its p-value, and the 25th / 50th / 75th / 90th / 95th / 99th percentile **values** for reference and test. The quality NetCDF is the aggregated CQI for plotting. ## Station validation Written by `05_station_validation.ipynb`. These filenames carry no prefix: ``` station_validation/ station_validation_{method}_month{MM}_dekad{DD}.csv # one row per station, 31 metrics station_validation_summary_month{MM}_dekad{DD}.csv # all methods, one dekad station_multi_threshold_{method}_month{MM}_dekad{DD}.csv # WMO threshold scores per station multi_threshold_summary_{method}_month{MM}_dekad{DD}.csv # threshold scores aggregated ``` Taylor diagram PNGs do not land here - they go to `figures/taylor/`. ## Figures ``` figures/ qa/ QA spatial maps, one subdirectory per plot type taylor/ Pooled and per-dekad Taylor diagrams station_validation/ Per-station scatter, threshold curves, regional boxplots ``` Under `figures/qa/` each plot type gets its own subdirectory named `{quality_prefix}_{plot_type}/`. Plot types include spatial comparison strips, CQI maps, and station scatter. The visualisation hub notebook (`06_visualisation_hub.ipynb`) generates all of these.