Interpreting the meaning of a visual scene requires not only identification of its constituent objects, but also a rich semantic characterization of object interrelations. Here, we study the neural mechanisms underlying visuo-semantic transformations by applying modern computational techniques to a large-scale 7T fMRI dataset of human brain responses elicited by complex natural scenes. Using semantic embeddings obtained by applying linguistic deep learning models to human-generated scene descriptions, we identify a widely distributed network of brain regions that encode semantic scene descriptions. Importantly, these semantic embeddings better explain activity in these regions than traditional object category labels. In addition, they are effective predictors of activity despite the fact that the participants did not actively engage in a semantic task, suggesting that visuo-semantic transformations are a default mode of vision. In support of this view, we then show that highly accurate reconstructions of scene captions can be directly linearly decoded from patterns of brain activity. Finally, a recurrent convolutional neural network trained on semantic embeddings further outperforms semantic embeddings in predicting brain activity, providing a mechanistic model of the brain's visuo-semantic transformations. Together, these experimental and computational results suggest that transforming visual input into rich semantic scene descriptions may be a central objective of the visual system, and that focusing efforts on this new objective may lead to improved models of visual information processing in the human brain.
翻译:在解释视觉场景的含义时,我们不仅需要识别其组成对象,还需要对对象的关联进行丰富的语义特征描述。在这里,我们通过将现代计算技术应用于由复杂的自然场景产生的大规模7T FMRI人类大脑反应数据集来研究相对语义变化背后的神经机制。使用语言深度学习模型对人造场景描述应用的语义嵌入,我们发现一个广泛分布的大脑区域网络,对语义场景描述进行编码。重要的是,这些语义嵌入比传统对象类别标签更好地解释这些区域的活动。此外,这些神经机制是活动的有效预测器,尽管参与者没有积极从事语义学任务,表明对大脑反应的7TFMMRI数据集是一种默认模式。为了支持这种观点,我们随后可以显示,非常准确的场景字幕重建可以直接线性地从大脑活动模式中解码。最后,一个经常性的演进线性网络,对语义嵌入比传统对象类别分类标签更能解释活动。此外,它们是活动的有效预测器,尽管参与者并没有积极从事一种语义性直观性视觉演进的视觉过程,在预测大脑的精度演进的大脑演进的大脑过程结果中,因此,我们可以预测大脑演进进进进进的大脑的大脑进进的大脑的大脑演进。