Kaggle S6E8
A Playground competition run as a measurement exercise. The validation was frozen early instead of being tuned against the public leaderboard, and the standing moved up ninety-five places once the private split was scored.
// evidence
- public finish
- 183 / 3,532
- private finish
- 88 / 3,532
- percentile
- top 2.5%
- validation
- 5-fold OOF
- honest OOF
- 0.9700980
// The pipeline
A 192-member stack on a frozen five-fold split, with an honest out-of-fold score of 0.9700980. Leaderboard score moved from 0.97018 to 0.97115.
// Public to private
The standing moved from 183rd on the public leaderboard to 88th on the private one — up ninety-five places once the unseen split was scored.
That is the direction the movement should go when the validation was frozen early rather than tuned against the public board. The public rank was never the thing being optimised.
// Calibration before spending a submission
The offset between out-of-fold and leaderboard settled at +0.00112. That made it possible to predict the leaderboard score before using up a submission on finding out.
// Negative results
- max_bin
- +0.0020 on its own, 0.0000002 once inside the ensemble.
- Groupby aggregates
- +0.0012 alone, 0.0000034 inside the ensemble.
- Dead ends
- Marginalisation failed outright. Row order, duplicate detection and parent-row lookup all went nowhere.