Blending - Implementation

How LSEQM and the CNN output are combined per pixel.

For the theory, see Methodology > Blending. This page is the algorithm view.

Where it lives

The blend lives entirely inside apply_deeplearning_model() in src/deep_learning.py - CNN inference and blending are the same call. run_correction_pipeline() in src/bias_correction.py only invokes it after LS and EQM (LSEQM).

The formula in code

# Conceptual
confidence = load_confidence_mask(config.CONFIDENCE_MASK_FILE)
effective_alpha = 1.0 - confidence * (1.0 - DL_BLEND_ALPHA)

# Selective application: blend only above a per-pixel extreme threshold.
# The threshold is the GPD_THRESHOLD_PERCENTILE-th percentile of the LSEQM
# field itself, taken across the time dimension - not the CPC GPD threshold.
threshold_2d = lseqm_field.quantile(GPD_THRESHOLD_PERCENTILE / 100.0, dim='time')
extreme_mask = (lseqm_field > threshold_2d) & (lseqm_field > 0)
blended = xr.where(
    extreme_mask,
    effective_alpha * lseqm_field + (1.0 - effective_alpha) * cnn_field,
    lseqm_field
)

Three things to notice:

  1. effective_alpha is spatial, derived from confidence. Each pixel can have a different blend weight.
  2. The blend only fires above the per-pixel extreme threshold. Sub-threshold pixels keep LSEQM exactly. This is the “DL refines extremes, not the body” design decision.
  3. confidence == 0 -> effective_alpha == 1.0 -> pure LSEQM. Cells with no nearby stations get no DL contribution, regardless of DL_BLEND_ALPHA.

Where the inputs come from

Input Source
lseqm_field Output of the lseqm() call earlier in run_correction_pipeline()
cnn_field model.predict() on the per-sample-normalised daily slice, inside apply_deeplearning_model()
confidence get_or_create_confidence_mask(...) in run_correction_pipeline(), passed in as confidence_mask - cached after first call
DL_BLEND_ALPHA Module-level constant set by initialize_config() from deep_learning.blend_alpha
Extreme threshold Computed on the fly: lseqm_field.quantile(GPD_THRESHOLD_PERCENTILE / 100, dim='time'), giving one value per pixel

Save

The blended field is written via save_corrected_precip(blended, ..., method_abbr='lseqmdl', folder=lseqmdl_corrected_precip_path) to data/output/corrected_lseqmdl/. Land-sea mask applied at write time using _cfg.mask_file (dynamic, so the Bali config is picked up correctly mid-session).

Cost

Negligible. The blend is a per-pixel multiply and add. For Indonesia: <1 second per dekad. For Bali: instant.

Failure modes

  • use_confidence_mask: false -> no mask is built and confidence_mask=None is passed through -> apply_deeplearning_model takes its uniform-alpha branch and uses DL_BLEND_ALPHA everywhere, regardless of gauge density. Useful for ablation experiments.
  • DL_BLEND_ALPHA: 1.0 -> effective_alpha = 1.0 everywhere -> output equals LSEQM. Effectively disables DL while keeping the saved file path. Useful as a sanity check.
  • CNN output has NaN at some pixels (rare but possible if upstream had NaN in the input field): the blend propagates NaN at those pixels. Land-sea mask at save time turns it into a known NaN, not a silent bug.

What this gives you

  • A field that is identical to LSEQM in gauge-sparse cells.
  • A field that lifts toward CNN in gauge-dense cells on extreme days.
  • A field that never deviates from LSEQM below the per-pixel extreme threshold, regardless of DL output.

The QA Framework tutorial has a confidence_jan_d2.png figure that visualises where this transition happens for Bali.

Back to top