Issued 1 November 2023: the next six months

Monthly rainfall forecasts for Indonesia, December 2023 to May 2024, from one forecast date

import {selectControl, loadWeekData, issueTargets, skillByLead, weekSkillMaps, sameTargetSkill, weekIssueMaps, scoresByLead} from "../assets/leads/leads.js"
leadData = loadWeekData("../assets/monthly-2023/")
geo = ({indonesia: await FileAttachment("../data/boundaries/indonesia.geojson").json(),
  provinces: await FileAttachment("../data/boundaries/provinces.geojson").json(),
  neighbours: await FileAttachment("../data/boundaries/neighbours.geojson").json()})
leadChoices = leadData.targets.map(t => [String(t.lead), `Lead ${t.lead}: ${t.label}`])

Case 03 · Seasonal · Monthly rainfall · Full Indonesia

On 1 November 2023 a forecaster has one new CFSv2 run and six months to say something about: December, then January to May 2024. This case does what an outlook issued that day would do. It calibrates the run separately for each of those months, scores each calibration over 30 years, and checks every forecast against what happened.

The setup is the one used throughout the tutorial: CFSv2 as the predictor, CHIRPS v3.0 as the observations, the same CCA and PCR settings and 30 training years (1991-2020). The observations keep CHIRPS v3.0’s native 0.05° grid: 61,190 pixels over Indonesia. Only the target month changes from one lead to the next. All numbers on this page come from the executed notebook.

Hover, tap or use the arrow keys on any map or chart to read values; each view can be downloaded as it is set. Maps load when you choose them, one small file per map.

The short result

Issued on 1 November, only the December forecast has some cross-validated skill, and little: an area-mean CCA correlation of 0.10. For January to May it lies between -0.15 and 0.03, which is no skill.
The raw CFSv2 correlation is higher than the calibrated one at every lead, but it is also small: 0.18 for December, -0.02 to 0.21 afterwards.
The target month matters as much as the lead. One month ahead, November from a 1 October start scores 0.24; December from a 1 November start, 0.10.
December 2023 was dry and the forecast leaned dry. January to May 2024 were mostly wetter than normal, while the forecasts kept leaning dry: the hit rate fell to 29% for May against 65% for always forecasting “above normal”.

One issue date, six target months

The lead on this page is the number of whole months between the issue date and the first day of the target month: December is lead 1, May is lead 6. PyCPT counts to the middle of the target month, so its own lead values are half a month larger.

Lead	Target	PyCPT lead	Hindcast starts	Training targets
1	December 2023	1.5	1 Nov, 1991-2020	Dec 1991-2020
2	January 2024	2.5	1 Nov, 1990-2019	Jan 1991-2020
3	February 2024	3.5	1 Nov, 1990-2019	Feb 1991-2020
4	March 2024	4.5	1 Nov, 1990-2019	Mar 1991-2020
5	April 2024	5.5	1 Nov, 1990-2019	Apr 1991-2020
6	May 2024	6.5	1 Nov, 1990-2019	May 1991-2020

Training uses the target years 1991-2020, PyCPT’s target_first_year and target_final_year, so the observed normals and tercile boundaries of every target come from the same 30 years. For a target in January or later, the 1 November starts behind them are those of the year before.

Each lead has its own PyCPT case folder, downloaded with download_data and fitted with evaluate_models, using CCA (3 predictor and 3 predictand EOFs, 2 CCA modes) and PCR (3 predictor EOFs), cross-validation with a window of 5 years. These settings are the ones used in the Fit & diagnose case, fixed before any skill was computed. At 0.05° each fit took 17 to 18 minutes, most of it spent writing and reading CPT’s files for 61,190 pixels, so the eighteen fits ran side by side, each in its own process.

viewof timelineLead = selectControl("Highlight", leadChoices, "1")
issueTargets(leadData, timelineLead)

Monthly rainfall from dekads, at 0.05°

The observations are CHIRPS v3.0 monthly totals built from the local dekadal archive (days 1-10, 11-20 and 21 to the end of the month), with exact zeros kept. For each training target the notebook reads the first and last day from the hindcast file (Ti and Tf) and sums the dekads covering exactly those days, refusing to add them unless they run contiguously; the February totals therefore have 28 or 29 days as the calendar requires.

As a cross-check against a separately stored product, December-February built from nine dekads matches the archive’s own 3-month total to within 0.0005 mm for 1990-91 and 2023-24, with the same missing pixels. The predictand is every 0.05° pixel whose centre lies inside Indonesia (Natural Earth 1:10m): 61,190 pixels with data in all 30 training years of every target month.

The CFSv2 requests multiply the model’s daily mean by the days in the month: 31, 31, 28.25, 31, 30 and 31. PyCPT gives February 28.25 days in every year (its monthlengths table), while the observations have 28 or 29. Calibration regresses onto the observed totals, which absorbs a constant factor, and correlation ignores it.

Skill: December only, and little of it

The chart shows the area-weighted mean over Indonesia of two cross-validated scores for each lead; the maps further down show how they vary from place to place. Pearson correlation is 0 for no linear relationship. 2AFC is the chance that the forecast ranks two different years in the observed order, 50% for no skill (Mason and Weigel 2009).

viewof skillMetric = selectControl("Score", [["Pearson", "Pearson correlation"], ["2AFC (%)", "2AFC"]], "Pearson")
skillByLead(leadData, skillMetric)

Area-weighted means over 61,190 pixels, 1991-2020. Source: skill-by-lead.csv.
Lead	Target	CCA Pearson	PCR Pearson	Raw CFSv2 Pearson	CCA 2AFC (%)	PCR 2AFC (%)
1	Dec 2023	0.10	0.09	0.18	53.9	53.7
2	Jan 2024	-0.11	-0.09	-0.02	46.4	47.1
3	Feb 2024	-0.06	-0.05	0.09	48.5	49.1
4	Mar 2024	0.03	0.01	0.21	50.9	50.7
5	Apr 2024	-0.11	-0.09	0.12	47.1	47.5
6	May 2024	-0.15	-0.10	0.05	45.2	46.4

December is the only target with calibrated skill, and it is small: a mean correlation of 0.10 and a 2AFC of 54%.
For January to May the calibrated forecasts have none. Their mean correlations are between -0.15 and 0.03 and their 2AFC between 45% and 51%. Values below zero do not mean the forecast is reliably wrong: when there is little real skill, cross-validated correlations of regression forecasts are biased below zero (Barnston and van den Dool 1993).
The raw model correlates better at every lead, by 0.08 to 0.24 over CCA. It has nothing fitted, so it pays no cross-validation cost, and calibration sees the predictor only through 3 EOF patterns of a large domain (85-155°E, 20°S-20°N). Neither explanation was tested here. The raw model’s correlation is itself small, at most 0.21.

Lead or target month?

Each lead on this page is also a different month, so the table above mixes two effects: forecasting further ahead, and some months being harder than others. To separate them the notebook also fits the opposite design with the same settings: one target, November 2023, forecast from six earlier starts (1 October back to 1 May). There only the lead changes.

sameTargetSkill(leadData)

Area-weighted mean correlation, cross-validated for CCA, 1991-2020. Source: same-target-skill.csv.
Lead	Issued 1 Nov: target	CCA Pearson	Raw CFSv2	November 2023: start	CCA Pearson	Raw CFSv2
1	Dec 2023	0.10	0.18	1 Oct 2023	0.24	0.36
2	Jan 2024	-0.11	-0.02	1 Sep 2023	0.19	0.31
3	Feb 2024	-0.06	0.09	1 Aug 2023	0.23	0.20
4	Mar 2024	0.03	0.21	1 Jul 2023	0.04	0.22
5	Apr 2024	-0.11	0.12	1 Jun 2023	0.11	0.20
6	May 2024	-0.15	0.05	1 May 2023	0.01	0.23

At every lead, November scores higher than the month the 1 November run is forecasting, for CCA and for the raw model alike (for CCA only just at lead 4: 0.04 against 0.03). One month ahead, the change of target alone cuts the CCA correlation from 0.24 to 0.10. Three months ahead, November keeps 0.23 while February has none. So the low skill from 1 November is mostly a property of the target months, December to May, not only of the distance ahead. Why these months are harder is not tested here.

Where the skill is

Area means hide where skill sits. The maps show the cross-validated correlation (or 2AFC) at every 0.05° pixel for each target, lead 1 at the top to lead 6 at the bottom.

viewof mapMethod = selectControl("Forecast", [["CCA", "CCA"], ["PCR", "PCR"], ["raw", "Raw CFSv2"]], "CCA")
viewof mapMetric = selectControl("Score", mapMethod === "raw" ? [["pearson", "Pearson correlation"]] : [["pearson", "Pearson correlation"], ["twoafc", "2AFC"]], "pearson")
weekSkillMaps(leadData, geo, mapMethod, mapMethod === "raw" ? "pearson" : mapMetric)

The forecasts as issued, and what happened

Each forecast is checked against its own month. Categories use the 33rd and 67th percentiles of that month’s 1991-2020 totals at each pixel.

Area-weighted over the 61,190 pixels. Source: issued-2023-11-observed.csv.
Lead	Target	Observed mean (mm)	1991-2020 mean (mm)	Below normal (% of area)	Near normal	Above normal
1	Dec 2023	270	302	48.2	33.8	18.0
2	Jan 2024	313	283	22.7	33.3	44.0
3	Feb 2024	259	258	31.4	35.8	32.8
4	Mar 2024	299	290	30.6	33.2	36.2
5	Apr 2024	316	283	21.3	29.1	49.6
6	May 2024	304	243	12.2	22.6	65.2

December 2023 was dry. February was close to normal, with the three categories spread evenly. The other months of 2024 were wetter than normal over much of Indonesia, most of all April and May, with above normal over 50% and 65% of the area.

The CCA forecasts favoured below normal at every lead, over 51% to 73% of the area. The map column shows each lead’s target: choose the forecast’s most likely category, the observed category, the totals, or the difference.

viewof forecastMethod = selectControl("Calibration", [["CCA", "CCA"], ["PCR", "PCR"]], "CCA")
viewof forecastView = selectControl("Show", [["category", "Most likely category"], ["observedCategory", "Observed category"], ["total", "Forecast total"], ["observed", "Observed total"], ["difference", "Forecast minus observed"]], "category")
weekIssueMaps(leadData, geo, leadData.issue.date, forecastMethod, forecastView)

viewof scoreMetric = selectControl("Score", [["hit", "Hit rate"], ["rpss", "RPSS"], ["mean", "Mean forecast total"]], "hit")
scoresByLead(leadData, scoreMetric)

Each target scored against its own observation. Sources: issued-2023-11-forecasts.csv, issued-2023-11-observed.csv.
Lead	Target	Method	Hit rate (%)	Reference hit rate (%)	RPSS	RMSE (mm)	Mean forecast (mm)
1	Dec 2023	CCA	48.5	48.2	0.14	81.1	275
1	Dec 2023	PCR	49.3	48.2	0.14	80.6	270
2	Jan 2024	CCA	36.6	44.0	0.01	96.8	265
2	Jan 2024	PCR	38.0	44.0	-0.00	99.7	260
3	Feb 2024	CCA	47.0	35.8	0.11	71.8	240
3	Feb 2024	PCR	45.5	35.8	0.08	74.3	240
4	Mar 2024	CCA	39.9	36.2	0.05	86.5	269
4	Mar 2024	PCR	38.1	36.2	0.04	86.6	269
5	Apr 2024	CCA	40.3	49.6	0.03	97.8	273
5	Apr 2024	PCR	38.2	49.6	0.01	99.3	274
6	May 2024	CCA	28.8	65.2	-0.10	122.8	231
6	May 2024	PCR	33.6	65.2	-0.15	132.8	225

Hit rate is the share of area where the most likely category was the one observed. The reference is always forecasting the category that covered the most area that month, picked with hindsight, so it is a demanding comparison. December’s forecasts matched it; February’s and March’s beat it (47% against 36%, and 40% against 36% for CCA); January, April and May fell short, by most in May (29% against 65%).
RPSS compares the forecast probabilities with climatological odds (33/34/33%) using the ranked probability score (Epstein 1969). It was positive for December (0.14) and February (0.11 for CCA), close to zero for January, March and April, and negative for May: leaning dry was costly in the wettest month. One outcome per lead rewards being right about 2023, not skill in general.
The deterministic totals were close in December (275 mm against 270 mm for CCA) and too low in every later month: by 19 mm in February and by 30 to 73 mm in the others.

What this does and does not show

This is one issue date, one model, one set of fixed settings and one verifying year per target. It shows that from a 1 November CFSv2 start, with these settings, calibrated monthly forecasts for Indonesia carry a little skill for December and none for January to May, and that the same lead gives clearly more skill for November. It does not show how other start months, models or settings behave, and the 2023-24 scores describe single outcomes.

The seasonal case issues on the same date for the next six overlapping three-month seasons and compares each with its first month. The subseasonal cases look at the weeks before that with GEFSv12.

Files

Executed notebook: every number and figure on this page
Skill by lead (CSV) · Same target from earlier starts (CSV) · Forecasts as issued, scored (CSV) · Observed targets (CSV) · Run summary (JSON)
Maps at 0.05° (NetCDF): for every lead, the observed total and category, and each method’s correlation, forecast total and most likely category

References

IRI. PyCPT 2.10.4 source: pycpt/src/pycpt/notebook.py (setup, download_data, summarize_available_years, evaluate_models and the plotting functions). Tag pycpt-2.10.4, commit 084bb5c. https://github.com/iri-pycpt/pycpt/blob/084bb5c380c200b5f65f11ba224a965547b91da1/pycpt/src/pycpt/notebook.py
IRI. PyCPT 2.10.4 source: cptdl (data catalog, leads_from_target, evaluate_url, download, simple_download). Tag pycpt-2.10.4, commit 084bb5c. https://github.com/iri-pycpt/pycpt/tree/084bb5c380c200b5f65f11ba224a965547b91da1/pycpt/src/cptdl
IRI. CPT 17.8.3 source code archive. Columbia University Academic Commons. https://doi.org/10.7916/fdp6-v391
CPT+. Forecast Probabilities (CPT help). https://cptplus.org/Help/Tools_Forecast_Probs.html Same page name as on the former cpthelp.iri.columbia.edu, which did not resolve when checked on 10 November 2026.
IRI Data Library. NMME NCEP-CFSv2 hindcasts, PENTAD_SAMPLES_FULL, precipitation. https://iridl.ldeo.columbia.edu/SOURCES/.Models/.NMME/.NCEP-CFSv2/.HINDCAST/.PENTAD_SAMPLES_FULL/.prec/ Browsing Data Library pages requires signing in (checked 10 November 2026); PyCPT’s data requests in this tutorial succeeded without a key.
Saha, S., Moorthi, S., Wu, X., et al. (2014). The NCEP Climate Forecast System Version 2. Journal of Climate, 27(6), 2185-2208. https://doi.org/10.1175/JCLI-D-12-00823.1
Kirtman, B., Min, D., Infanti, J., et al. (2014). The North American Multimodel Ensemble: Phase-1 seasonal-to-interannual prediction; Phase-2 toward developing intraseasonal prediction. Bulletin of the American Meteorological Society, 95(4), 585-601. https://doi.org/10.1175/BAMS-D-12-00050.1
Climate Hazards Center, UC Santa Barbara. CHIRPS: Rainfall Estimates from Rain Gauge and Satellite Observations (CHIRPS version 3.0 page). https://www.chc.ucsb.edu/data/chirps3
Funk, C., Peterson, P., Landsfeld, M., et al. (2014). A quasi-global precipitation time series for drought monitoring. U.S. Geological Survey Data Series 832. https://doi.org/10.3133/ds832
Mason, S. J., and Weigel, A. P. (2009). A generic forecast verification framework for administrative purposes. Monthly Weather Review, 137(1), 331-349. https://doi.org/10.1175/2008MWR2553.1
Epstein, E. S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology, 8(6), 985-987. https://doi.org/10.1175/1520-0450(1969)008%3C0985:ASSFPF%3E2.0.CO;2
Barnston, A. G., and van den Dool, H. M. (1993). A degeneracy in cross-validated skill in regression-based forecasts. Journal of Climate, 6(5), 963-977. https://doi.org/10.1175/1520-0442(1993)006%3C0963:ADICVS%3E2.0.CO;2