Heretical Division

Pushing the limits of computational heresy

This division of I2 (not a serious division) works on writing short papers that push the limits of computational heresy. We publish to the heretical and satirical SIGBOVIK conference, hosted by the ACH (Association for Computational Heresy).

Current papers in the works:

Title	Abstract
“Train-Validation Split Optimization is All You Need (Literally)	The key tenet of model evaluation in supervised learning is the separation of the training dataset and the validation dataset. A machine learning model is trained on the training dataset and evaluated on the validation dataset. The separation of the data is key to the prevention of data leakage. This step of the modeling process is acknowledged by those in the field of machine learning – but, unfortunately, far less than it should be. Machine learning practitioners dedicate at most 15 seconds to deciding how to go about train-validation split. The random state seed and the train-size proportion are arbitrarily chosen. This arbitrariness poses serious problems for the objective evaluation of models. In this paper, we apply this principle to a quickly growing field that has completely neglected it: machine learning. Specifically, we apply Bayesian optimization to optimize the training proportion size and the random state of the train-validation split operation to maximize the validation performance. We find that optimizing the train-validation split consistently beats every single hitherto published machine learning technique on every supervised dataset.

Title

Abstract

“Train-Validation Split Optimization is All You Need (Literally)

The key tenet of model evaluation in supervised learning is the separation of the training dataset and the validation dataset. A machine learning model is trained on the training dataset and evaluated on the validation dataset. The separation of the data is key to the prevention of data leakage. This step of the modeling process is acknowledged by those in the field of machine learning – but, unfortunately, far less than it should be. Machine learning practitioners dedicate at most 15 seconds to deciding how to go about train-validation split. The random state seed and the train-size proportion are arbitrarily chosen. This arbitrariness poses serious problems for the objective evaluation of models. In this paper, we apply this principle to a quickly growing field that has completely neglected it: machine learning. Specifically, we apply Bayesian optimization to optimize the training proportion size and the random state of the train-validation split operation to maximize the validation performance. We find that optimizing the train-validation split consistently beats every single hitherto published machine learning technique on every supervised dataset.