Divergent Predictive States: The Statistical Complexity Dimension of Stationary, Ergodic Hidden Markov Processes

Inferring models from samples of stochastic processes is challenging, even in the most basic setting in which processes are stationary and ergodic. A principle reason for this, discovered by Blackwell in 1957, is that finite-state generators produce realizations with arbitrarily-long dependencies that require an uncountable infinity of probabilistic features be tracked for optimal prediction. This, in turn, means predictive models are generically infinite-state. Specifically, hidden Markov chains, even if finite, generate stochastic processes that are irreducibly complicated. The consequences are dramatic. For one, no finite expression for their Shannon entropy rate exists. Said simply, one cannot make general statements about how random they are and finite models incur an irreducible excess degree of unpredictability. This was the state of affairs until a recently-introduced method showed how to accurately calculate their entropy rate and, constructively, to determine the minimal set of infinite predictive features. Leveraging this, here we address the complementary challenge of determining how structured hidden Markov processes are by calculating the rate of statistical complexity divergence -- the information dimension of the minimal set of predictive features.