Hello Darth,

Thank you for flagging this, we will review and revert soon within the week.

Hi @Darth_Glennium, thanks for flagging this I want to add support to your observation, along with a couple of questions. On the sub-1.0 scores: I agree this is very hard to reconcile with legitimate modeling on the provided data. I've also experimented different things covering feature engineering, target encoding variants, CatBoost/XGBoost/LightGBM, ensemble of those three models and calibration, all validated on a fixed 10-fold CV and the practical ceiling I've converged to is around 1.04xx - 1.05xx. In my experience here, improvements of even 0.001 require significant work, and like you, I found persistent, irreducible confusion between adjacent cost classes that no modeling technique resolved. A jump of 0.04+ below that ceiling isn't the shape of a better model, it's the shape of additional information, most plausibly label information about the test rows themselves. A request: you mentioned that "the full labeled data from that 2022 edition including data that would overlap with the current test set is publicly available in GitHub repositories." Could you share those specific repository links, ideally directly with the Zindi team rather than in this thread(In as much i want to really see the label data) so they can verify the overlap with the current test set? That evidence would make the investigation concrete rather than circumstantial.

On the rules: the competition data page states: "You may use only the datasets provided for this competition. Automated machine learning tools such as automl are not permitted." So if the sub-1.0 solutions rely on external labeled data from the 2022 edition, they should not be accepted as valid final solutions under the current rules. That said, I want to keep the same door open that you did: neither of us can see their code, and it's possible they've found a legitimate technique the rest of us haven't considered. The winners' code review at the end of the competition should settle it either way but given that the 2022 labels would overlap the test set, I think it's worth the Zindi team looking into this before final rankings rather than after.

Basically you can't get sub 1.02 without using external data. for mine i have used external data of the other comptition to get that score. and i stopped submitting after i got that score. my best non exrternal data submission is 1.026 in private score but the comptition automatically selected the best score which have the external data. i think zindi will fix the leaderbored by removing those of us who used external data.

to be honest my best public leaderbored solution without the external data still be the best generalizing model from this comptition since on Public it was 1.042946216 and in private it is 1.026950578, which will be 2nd in the current economy :-)