Decentralized Stochastic Control in Standard Borel Spaces: Centralized MDP Reductions, Near Optimality of Finite Window Local Information, and Q-Learning

Decentralized stochastic control problems are intrinsically difficult to study because of the inapplicability of standard tools from centralized control such as dynamic programming and the resulting computational complexity. In this paper, we address some of these challenges for decentralized stochastic control with Borel spaces under three different but tightly related information structures under a unified theme: the one-step delayed information sharing pattern, the K-step periodic information sharing pattern, and the completely decentralized information structure where no sharing of information occurs. We will show that the one-step delayed and K-step periodic problems can be reduced to a centralized MDP, generalizing prior results which considered finite, linear, or static models, by addressing several measurability questions. The separated nature of policies under both information structures is then established. We then provide sufficient conditions for the transition kernels of both centralized reductions to be weak-Feller, which facilitates rigorous approximation and learning theoretic results. We will then show that for the completely decentralized control problem finite memory local policies are near optimal under a joint conditional mixing condition. This is achieved by obtaining a bound for finite memory policies which goes to zero as memory size increases. We will also provide a performance bound for the K-periodic problem, which results from replacing the full common information by a finite sliding window of information. The latter will depend on the condition of predictor stability in expected total variation, which we will establish. We finally show that under the periodic information sharing pattern, a quantized Q-learning algorithm converges asymptotically towards a near optimal solution. Each of the above, to our knowledge, is a new contribution to the literature.