Driven by the key challenges of cell therapy manufacturing, including high complexity, high uncertainty, and very limited process observations, we propose a hybrid model-based reinforcement learning (RL) to efficiently guide process control. We first create a probabilistic knowledge graph (KG) hybrid model characterizing the risk- and science-based understanding of biomanufacturing process mechanisms and quantifying inherent stochasticity, e.g., batch-to-batch variation. It can capture the key features, including nonlinear reactions, nonstationary dynamics, and partially observed state. This hybrid model can leverage existing mechanistic models and facilitate learning from heterogeneous process data. A computational sampling approach is used to generate posterior samples quantifying model uncertainty. Then, we introduce hybrid model-based Bayesian RL, accounting for both inherent stochasticity and model uncertainty, to guide optimal, robust, and interpretable dynamic decision making. Cell therapy manufacturing examples are used to empirically demonstrate that the proposed framework can outperform the classical deterministic mechanistic model assisted process optimization.
翻译:在细胞疗法制造的关键挑战,包括高度复杂、高度不确定性和非常有限的流程观测的驱动下,我们提议采用基于混合模型的强化学习(RL)来有效指导流程控制。我们首先创建一种概率性知识图(KG)混合模型(KG)来描述基于风险和科学的对生物制造过程机制的理解,并量化固有的随机性,例如批量到批量的变异。它可以捕捉关键特征,包括非线性反应、非静止动态和部分观察状态。这种混合模型可以利用现有的机械化模型,便利从多种工艺数据中学习。我们使用一种计算抽样方法来生成对模型不确定性进行量化的后继样本。然后,我们引入一种基于混合模型的巴伊西亚RL,既考虑到内在的随机性和模型不确定性,又考虑到内在的随机性和模型不确定性,以指导最佳、稳健健和可解释的动态决策。使用细胞疗法的制造实例,从经验上证明拟议的框架能够超越典型的确定性机械模型的优化过程。