Methods and limitations

What the comparisons mean, and what they do not establish

Data and spatial support

The predictor is CFSv2 ensemble-mean precipitation from the IRI Data Library. The source archive was checked for the requested months, locations and valid ensemble members in the research preparation. Observations are local CHIRPS3 seasonal accumulations, aggregated from native 0.05° country-land pixels to 0.25° cells. The CHIRPS3 project provides product background; the exact local preprocessing and source hashes are recorded in the research archive.

The evaluation footprint contains 3,240 shared cells. Country-land weights account for the retained land support, and the same weights are used for all methods. Raw model forecasts are bilinearly interpolated to that grid; calibrated forecasts are produced at the predictand locations. This does not establish independent 0.25° dynamical predictability or replace station-based verification.

The lead-time case studies (monthly, seasonal, dekad, weekly and pentad) do not aggregate: their predictand is every native 0.05° CHIRPS v3.0 pixel whose centre lies inside Indonesia, 61,190 pixels, with area means weighted by the cosine of latitude. Each case page states its own data, training sample and checks.

The archived model source appends operational forecasts from April 2011. Complete data and valid member counts do not demonstrate a statistically homogeneous reforecast/operational record. The observation and source vintages are retrospective.

Training and validation

The main example uses 1991–2020, with 1983–2010 retained as a sensitivity check. The source begins with January 1982 initializations. Under the experiment’s calendar, some 1982-labelled target seasons require unavailable 1981 initializations; using 1983 as the common older start keeps sample availability consistent across all twelve seasons.

Both methods retain three predictor EOFs. CCA retains three predictand EOFs and two CCA modes. Counts are fixed, not searched. The fits use no observation transformation, tailoring, dry mask or skill mask. CPT automatically enables a zero bound for rainfall.

CPT uses a five-sample cyclic cross-validation window inside training: a target sample and two neighbours on each side are excluded, wrapping at the boundaries. The engine refits within each fold, then constructs a model from all training samples for the later forecast inputs. The withheld 2023-centred targets are outside their training intervals. The IRI CPT reference explains CCA/PCR and cross-validation. Checks made while preparing the tutorial (the CPT 17.8.3 source and eight perturbation runs; working records, not published) confirm this cross-validation behaviour.

Scores and calendar conventions

The notebook calculates country-land-weighted spatial RMSE, MAE and bias for each forecast window. Smaller RMSE/MAE is better; positive bias means too wet. Climatology is the corresponding training-period observed mean. Pooled errors over the original four-year context are computed before taking the square root.

The main story selects midpoint year 2023, including NDJ ending in January 2024. The context figure retains the original ending-year labels 2021–2024. Those conventions are deliberately distinguished. Adjacent seasons overlap and spatial cells are correlated; neither twelve windows nor thousands of cells provide that many independent annual validation samples. No significance tests are claimed.

Province-weighted verification

The province extension partitions the original native country-pixel weights using the unsimplified supplied 38-feature province file. Point-centre membership uses Shapely containment; points outside every province or inside multiple provinces remain explicit residual categories. The original national footprint is unchanged. The province provenance records source hashes and counts.

Within a coarse cell, province shares inherit the same saved forecast/observation error. These are province-weighted views of0.25° errors, not newly reconstructed province rainfall. For a province with little support, even a large percentage change may be driven by very few coarse cells. Cell counts and maximum cell-weight fractions are reported; neither is an independent validation sample count.

Each region is normalized separately. Combining region MSE, MAE and bias with their national weight fractions reproduces national errors, including the unassigned residual. Combine MSE before taking its square root. The four-year regional context uses the original ending-year2021–2024 labels and includes2023; it is not independent confirmation. Forecasts, fitting choices and masks are unchanged.

Why probability scores are absent here

Deterministic improvement does not establish probability reliability. Native CPT probabilities use an error distribution and information from cross-validated hindcast errors. The detailed CPT probability documentation describes Student’s t errors and configurable error-variance choices.

In the earlier research experiments, effective category-reference periods differ from some training periods: within-year older fits use 1991–2010 and recent fits 1991–2020; DJF/NDJ use ending years 1992–2010 and 1992–2020 respectively. These category periods are distinct from the full-training climatology baselines used in this deterministic tutorial. Probabilities are calculated before rainfall forecasts are zero-bounded, so a saved zero can conceal a negative distribution location. The saved prediction-error variance is the squared scale used by the engine, not simply the CV RMSE squared.

The official seasonal notebook labels some probability scores as in-sample. Its high-level driver re-dates training predictors to construct fitted probability series. Our direct-core benchmark did not use that pathway. The installed flexible-exceedance helper also uses different t parameters from native CPT17: a same-threshold OND check found a difference up to 0.882 percentage points. That establishes a difference in construction, not a ranking of empirical reliability.

Future probability comparisons need common events, explicit thresholds and distribution metadata, and a separately verified independent, retroactive or double-CV design. The CPT verification guide describes probabilistic verification as a separate task.

Research evidence

The source repository retains completed cases, with their NetCDF outputs and manifests, in research/cases/. Audits and reviews made while preparing the tutorial are working records and are not published. This tutorial’s input provenance identifies the exact fields and hashes used. The original files remain unchanged.

The next scientific work is a longer independent validation, deeper regional diagnostics and physical interpretation of the fitted EOF/CCA patterns. The present results should not be used directly to infer drought, planting dates, rainfall onset or operational probability reliability.

References

  1. IRI. Guide to PyCPT 2 for Seasonal Climate Forecasts (Jupyter Book, 2 September 2023 build): The Climate Predictability Tool (CPT). https://iri.columbia.edu/~awr/pycpt/seasonal-jupyterbook/html_2sep2023/cpt.html
  2. IRI. PyCPT operational seasonal notebook (Operations/pycpt-operational.ipynb). iri-pycpt/notebooks, commit 1c97531. https://github.com/iri-pycpt/notebooks/blob/1c975316910b9e047ce685d9b8cdefb2c4b73551/Operations/pycpt-operational.ipynb
  3. CPT+. Forecast Probabilities (CPT help). https://cptplus.org/Help/Tools_Forecast_Probs.html Same page name as on the former cpthelp.iri.columbia.edu, which did not resolve when checked on 10 November 2026.
  4. CPT+. 430: Verification (CPT help). https://cptplus.org/Help/Tools_Verify.html Same page name as on the former cpthelp.iri.columbia.edu, which did not resolve when checked on 10 November 2026.
  5. IRI Data Library. NMME NCEP-CFSv2 hindcasts, PENTAD_SAMPLES_FULL, precipitation. https://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/.NCEP-CFSv2/.HINDCAST/.PENTAD_SAMPLES_FULL/.prec/ Browsing Data Library pages requires signing in (checked 10 November 2026); PyCPT’s data requests in this tutorial succeeded without a key.
  6. Saha, S., Moorthi, S., Wu, X., et al. (2014). The NCEP Climate Forecast System Version 2. Journal of Climate, 27(6), 2185-2208. https://doi.org/10.1175/JCLI-D-12-00823.1
  7. Kirtman, B., Min, D., Infanti, J., et al. (2014). The North American Multimodel Ensemble: Phase-1 seasonal-to-interannual prediction; Phase-2 toward developing intraseasonal prediction. Bulletin of the American Meteorological Society, 95(4), 585-601. https://doi.org/10.1175/BAMS-D-12-00050.1
  8. Climate Hazards Center, UC Santa Barbara. CHIRPS: Rainfall Estimates from Rain Gauge and Satellite Observations (CHIRPS version 3.0 page). https://www.chc.ucsb.edu/data/chirps3
  9. Funk, C., Peterson, P., Landsfeld, M., et al. (2014). A quasi-global precipitation time series for drought monitoring. U.S. Geological Survey Data Series 832. https://doi.org/10.3133/ds832
  10. Shapely. shapely.contains_xy. https://shapely.readthedocs.io/en/stable/reference/shapely.contains_xy.html
Back to top