The simplicity bubble effect as a zemblanitous phenomenon in learning systems

The ubiquity of Big Data and machine learning in society evinces the need of further investigation of their fundamental limitations. In this paper, we extend the ``too-much-information-tends-to-behave-like-very-little-information'' phenomenon to formal knowledge about lawlike universes and arbitrary collections of computably generated datasets. This gives rise to the simplicity bubble problem, which refers to a learning algorithm equipped with a formal theory that can be deceived by a dataset to find a locally optimal model which it deems to be the global one. However, the actual high-complexity globally optimal model unpredictably diverges from the found low-complexity local optimum. Zemblanity is defined by an undesirable but expected finding that reveals an underlying problem or negative consequence in a given model or theory, which is in principle predictable in case the formal theory contains sufficient information. Therefore, we argue that there is a ceiling above which formal knowledge cannot further decrease the probability of zemblanitous findings, should the randomly generated data made available to the learning algorithm and formal theory be sufficiently large in comparison to their joint complexity.