Climate Risk and Health Prediction Challenge · Oseni Sakariyau Oluwadamilare

Two findings from the data that cost us a lot of time, in case they save someone else theirs.

All numbers below come from out-of-fold predictions with folds grouped by village, scored the way the leaderboard scores: 0.6 x F1 + 0.4 x ROC-AUC, with the binary label fixed at probability > 0.5.

THERE IS A BREAK IN THE DATA AROUND MID-2017

Infant deaths (age 0 and 1) are 96.1% climate-sensitive before mid-2017 and 57.6% after. Widening the age band to under-5s gives 94.6% and 60.0%, so it is not an artefact of where the age cut sits. Over the same period, infants go from about a third of all recorded deaths to more than three quarters of them.

Year by year, as infant share of recorded deaths / share of those infant deaths that are climate-sensitive:

  2009 - 43% / 96%

  2013 - 30% / 96%

  2016 - 33% / 96%

  2018 - 48% / 60%

  2019 - 67% / 44%

  2021 - 74% / 60%

  2022 - 84% / 77%

More infant deaths recorded, and fewer of them climate-sensitive, at the same moment. That points to a change in which deaths were recorded rather than a change in the climate. It is also the most important interaction in the data: age on its own reaches an AUC of 0.73, but a model that cannot split on age AND date together misses most of what age can do.

CLIMATE ANOMALIES LOOK PREDICTIVE, AND IT IS THAT BREAK IN DISGUISE

Standardising a climate variable against its own village-and-month climatology produces features that look excellent among under-5s, and then collapse to chance the moment you look inside a single period. AUC among under-5s, then within each period, then the absolute correlation of the feature with year:

  Root-zone soil wetness, 180 days - 0.728, then 0.514 before and 0.535 after, corr 0.69

  Surface soil wetness, 180 days - 0.724, then 0.531 before and 0.538 after, corr 0.68

  Relative humidity, 180 days - 0.719, then 0.524 before and 0.548 after, corr 0.62

  Root-zone soil wetness, 30 days - 0.705, then 0.521 before and 0.531 after, corr 0.58

Soil moisture in this area rose steadily across 2007-2022, so "wetter than this village's normal" is largely a proxy for "later year", which is a proxy for the recording change. If you validate with shuffled folds you will not catch this.

The structural reason is worth stating plainly: the villages sit within a few kilometres of one another, so on any given date they share the same weather. That makes a climate variable close to a function of the date, and the date is already in the data. We fetched NASA POWER daily weather for every coordinate and built 84 features from 7, 30, 90 and 180-day windows. None of them improved a village-grouped CV score.

VALIDATE WITH FOLDS GROUPED BY VILLAGE

Ten of the eleven test villages never appear in the training data, and no test coordinate appears in train at all. Shuffled cross-validation therefore rewards memorising villages, which cannot transfer to the test set. Since switching to folds grouped by village, our local CV has ranked four submissions in exactly the leaderboard's order.

Also worth knowing: ten villages sit more than 30 km from the main cluster, several of them over 100 km away (Kikuube, Mbale, Entebbe, Sembabule, Kalangala and Mitooma among them), yet their records look statistically identical to the Iganga ones - same year coverage, same infant share, same positive rate. That suggests the coordinates are wrong rather than the records, which would mean the climate features attached to those rows describe the wrong place.

EXTERNAL DATA USED

  - NASA POWER daily meteorology, public and requiring no account:

    https://power.larc.nasa.gov/docs/services/api/temporal/daily/

  - The CHIRPS, ERA5-Land, MODIS MOD13Q1 and SRTM features supplied with the competition.

Happy to compare notes with anyone who has found signal that survives a period-conditioned check.