There is a growing interest in the machine learning community in developing predictive algorithms that are "interpretable by design". Towards this end, recent work proposes to make interpretable decisions by sequentially asking interpretable queries about data until a prediction can be made with high confidence based on the answers obtained (the history). To promote short query-answer chains, a greedy procedure called Information Pursuit (IP) is used, which adaptively chooses queries in order of information gain. Generative models are employed to learn the distribution of query-answers and labels, which is in turn used to estimate the most informative query. However, learning and inference with a full generative model of the data is often intractable for complex tasks. In this work, we propose Variational Information Pursuit (V-IP), a variational characterization of IP which bypasses the need for learning generative models. V-IP is based on finding a query selection strategy and a classifier that minimizes the expected cross-entropy between true and predicted labels. We then demonstrate that the IP strategy is the optimal solution to this problem. Therefore, instead of learning generative models, we can use our optimal strategy to directly pick the most informative query given any history. We then develop a practical algorithm by defining a finite-dimensional parameterization of our strategy and classifier using deep networks and train them end-to-end using our objective. Empirically, V-IP is 10-100x faster than IP on different Vision and NLP tasks with competitive performance. Moreover, V-IP finds much shorter query chains when compared to reinforcement learning which is typically used in sequential-decision-making problems. Finally, we demonstrate the utility of V-IP on challenging tasks like medical diagnosis where the performance is far superior to the generative modelling approach.
翻译:机器学习界对开发预测算法的兴趣日益浓厚,这种算法是“设计可以解释的”。为此,最近的工作提议通过按顺序对数据进行可解释的查询作出解释性决定,直到能够根据获得的答案(历史)以高度信心作出预测。为了促进简短的问答链,采用了名为“信息追求”(IP)的贪婪程序,该程序适应性地选择查询以获得信息。采用了生成模型来学习查询-答案和标签的分布,而后者又被用来估计信息最丰富的查询。然而,对数据的全面归正模型的学习和推断往往难以解决复杂的任务。在这项工作中,我们提出“动态信息追求”(V-IP),对IP进行变异性描述,从而绕过学习基因模型的需要。 V-IP基于寻找查询选择策略和精度解析器,将真实和预测标签之间的预期交叉流度最小化。我们然后通过查询显示,IP战略是这一问题的最佳解决办法。因此,我们不用学习最实际的离正值的计算方法,而是用最精确的比正价的算方法,我们用一个最高级的算法来解释。