基于反馈型级联连接模型的多模态语义SFM方法研究

项目名称： 基于反馈型级联连接模型的多模态语义SFM方法研究

项目编号： No.61501451

项目类型： 青年科学基金项目

立项/批准年度： 2016

项目学科： 无线电电子学、电信技术

项目作者： 沈晔湖

作者单位： 苏州科技大学

项目金额： 19万元

中文摘要： 由运动恢复结构（SFM）是计算机视觉领域的基本问题之一。目前大多数研究基于对图像基本几何基元的几何分析，缺乏对语义信息的利用，因此稳定性不强、应用领域受限。本项目借鉴人脑并行分级处理以及反馈机制，拟研究反馈型级联连接模型框架对SFM 和图像分割、区域类别标记、物体识别等语义子模块进行整合，通过框架的反馈级联机制突破传统将各子问题割裂分析的做法，实现语义输出SFM 系统。本项目还将研究一种包含点、物体、区域等多模态输出SFM 新算法，提升稳定性，减少语义鸿沟。此外现有的基准测试数据库仅针对SFM或者图像理解与分析系统中的单个或部分子模块设计,因此本项目还将构建一个同时包含三维和语义分割信息的室内外SFM基准测试数据。该研究有望丰富SFM 和视频理解与分析算法理论，并推动机器人自主导航、增强现实、电影特效等领域的发展，因此具有重要的科学意义和广泛的应用前景。

中文关键词： 视觉信息处理；视觉信息获取

英文摘要： Structure from motion (SFM) is one of the basic research fields in computer vision community. However, researchers mainly focus on geometric analysis based on basic geometric elements until now. There are few researches about the combination with semantic information in the images. As a result, the performance of traditional SFM is not robust and its applications are restricted. Inspired by the parallel processing strategy of information and feedback function in our human brain, we plan to research about Cascaded Connection with Feedback Framework. In this framework, current sub-modules of semantic image understanding and SFM can be easily incorporated. This framework overcomes the defects of researches on each sub-module without considering the effects of other sub-modules. This project will propose multi-modality output SFM algorithm with information of points, objects and regions in the Cascaded Connection with feedback Framework in order to improve the robustness of SFM and diminish the semantic gaps in real applications. Current benchmark test databases are only designed according to single or some sub-modules of SFM and image understanding and analysis systems. To cope with the aforementioned defects, this project will build an indoor/outdoor test database with benchmarks about 3D structures and semantic information. This project is hoped to enrich the computational theory of SFM and video understanding and analysis. It will also advance the development of related areas such as autonomous robot navigation, augmented reality and special effects in movie industry etc.. In conclusion, this project is of important scientific meanings and has broad applications in the future.

英文关键词： Visual information processing;Visual information retrieval