【专知荟萃24】视频描述生成(Video Captioning)知识资料全集（入门/进阶/论文/综述/代码/专家，附查看）

2017 年 11 月 30 日 专知

点击上方“专知”关注获取专业AI知识!

【导读】主题荟萃知识是专知的核心功能之一，为用户提供AI领域系统性的知识学习服务。主题荟萃为用户提供全网关于该主题的精华（Awesome）知识资料收录整理，使得AI从业者便捷学习和解决工作问题！在专知人工智能主题知识树基础上，主题荟萃由专业人工编辑和算法工具辅助协作完成，并保持动态更新！另外欢迎对此创作主题荟萃感兴趣的同学，请加入我们专知AI创作者计划，共创共赢！专知为大家呈送专知主题荟萃知识资料大全集荟萃 （入门/进阶/综述/视频/代码/专家等），请大家查看！专知访问www.zhuanzhi.ai, 或关注微信公众号后台回复" 专知"进入专知，搜索感兴趣主题查看。此外，我们也提供该文网页桌面手机端（www.zhuanzhi.ai）完整访问，可直接点击访问收录链接地址，以及pdf版下载链接，请文章末尾查看！此为初始版本，请大家指正补充，欢迎在后台留言！欢迎大家转发分享~

视频描述生成（Video Captioning）专知荟萃

视频描述生成（Video Captioning）专知荟萃

入门学习
进阶文章

2015
2016
2017

Tutorial
代码
领域专家
Datasets

入门学习

Video Analysis 相关领域介绍之Video Captioning(视频to文字描述)

[https://zhuanlan.zhihu.com/p/26730181]

让机器读懂视频

[http://gitbook.cn/books/59192e91ceea8e6fe4504c74/index.html]

梅涛：“看图说话”——人类走开，我AI来

[https://mp.weixin.qq.com/s/shBHKh2emSIL5uWV7po8cw]

深度三维残差神经网络：视频理解新突破

[http://www.msra.cn/zh-cn/news/features/pseudo-3d-residual-networks-20171027]

Word2VisualVec for Video-To-Text Matching and Ranking

[http://www-nlpir.nist.gov/projects/tvpubs/tv16.slides/tv16.vtt.mediamill.slides.pdf]

进阶文章

2015

Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, Trevor Darrell, Long-term Recurrent Convolutional Networks for Visual Recognition and Description, CVPR, 2015.
- [http://arxiv.org/pdf/1411.4389.pdf]
Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko, Translating Videos to Natural Language Using Deep Recurrent Neural Networks, arXiv:1412.4729.

UT / UML / Berkeley [http://arxiv.org/pdf/1412.4729]

Yingwei Pan, Tao Mei, Ting Yao, Houqiang Li, Yong Rui, Joint Modeling Embedding and Translation to Bridge Video and Language, arXiv:1505.01861.

Microsoft [http://arxiv.org/pdf/1505.01861]

Subhashini Venugopalan, Marcus Rohrbach, Jeff Donahue, Raymond Mooney, Trevor Darrell, Kate Saenko, Sequence to Sequence--Video to Text, arXiv:1505.00487.

UT / Berkeley / UML [http://arxiv.org/pdf/1505.00487]

Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Ballas, Christopher Pal, Hugo Larochelle, Aaron Courville, Describing Videos by Exploiting Temporal Structure, arXiv:1502.08029

Univ. Montreal / Univ. Sherbrooke [http://arxiv.org/pdf/1502.08029.pdf]]

Anna Rohrbach, Marcus Rohrbach, Bernt Schiele, The Long-Short Story of Movie Description, arXiv:1506.01698

MPI / Berkeley [http://arxiv.org/pdf/1506.01698.pdf]]

Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, Sanja Fidler, Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books, arXiv:1506.06724

Univ. Toronto / MIT [[http://arxiv.org/pdf/1506.06724.pdf]]

Kyunghyun Cho, Aaron Courville, Yoshua Bengio, Describing Multimedia Content using Attention-based Encoder-Decoder Networks, arXiv:1507.01053

Univ. Montreal [http://arxiv.org/pdf/1507.01053.pdf]

2016

Multimodal Video Description

[https://dl.acm.org/citation.cfm?id=2984066]

Describing Videos using Multi-modal Fusion

[https://dl.acm.org/citation.cfm?id=2984065]

Andrew Shin , Katsunori Ohnishi , Tatsuya Harada Beyond caption to narrative: Video captioning with multiple sentences

[http://ieeexplore.ieee.org/abstract/document/7532983/]

Jianfeng Dong, Xirong Li, Cees G. M. Snoek Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction

[https://pdfs.semanticscholar.org/de22/8875bc33e9db85123469ef80fc0071a92386.pdf]

2017

Dotan Kaufman, Gil Levi, Tal Hassner, Lior Wolf, Temporal Tessellation for Video Annotation and Summarization, arXiv:1612.06950.

TAU / USC [[https://arxiv.org/pdf/1612.06950.pdf]]

Chiori Hori, Takaaki Hori, Teng-Yok Lee, Kazuhiro Sumi, John R. Hershey, Tim K. Marks Attention-Based Multimodal Fusion for Video Description

[https://arxiv.org/abs/1701.03126]

Weakly Supervised Dense Video Captioning（CVPR2017）
Multi-Task Video Captioning with Video and Entailment Generation（ACL2017）
Multimodal Memory Modelling for Video Captioning, Junbo Wang, Wei Wang, Yan Huang, Liang Wang, Tieniu Tan

[https://arxiv.org/abs/1611.05592]

Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, Eric P. Xing Recurrent Topic-Transition GAN for Visual Paragraph Generation

[https://arxiv.org/abs/1703.07022]

MAM-RNN: Multi-level Attention Model Based RNN for Video Captioning Xuelong Li1 , Bin Zhao2 , Xiaoqiang Lu1

[https://www.ijcai.org/proceedings/2017/0307.pdf]

Tutorial

“Bridging Video and Language with Deep Learning,” Invited tutorial at ECCV-ACM Multimedia, Amsterdam, The Netherlands, Oct. 2016.

[https://www.microsoft.com/en-us/research/publication/tutorial-bridging-video-language-deep-learning/]

ICIP-2017-Tutorial-Video-and-Language-Pub

[https://www.microsoft.com/en-us/research/wp-content/uploads/2017/09/ICIP-2017-Tutorial-Video-and-Language-Pub.pdf]

代码

neuralvideo

[https://github.com/olivernina/neuralvideo]

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

[ https://www.cs.utexas.edu/~vsub/naacl15_project.html#code\]

Describing Videos by Exploiting Temporal Structure

[https://github.com/yaoli/arctic-capgen-vid]

SA-tensorflow: Soft attention mechanism for video caption generation

[https://github.com/tsenghungchen/SA-tensorflow]

Sequence to Sequence -- Video to Text

[https://github.com/jazzsaxmafia/video_to_sequence\]

领域专家

梅涛微软亚洲研究院资深研究员梅涛博士，微软亚洲研究院资深研究员，国际模式识别学会会士，美国计算机协会杰出科学家，中国科技大学和中山大学兼职教授博导。主要研究兴趣为多媒体分析、计算机视觉和机器学习。 - [https://www.microsoft.com/en-us/research/people/tmei/]
Xirong Li 李锡荣中国人民大学数据工程与知识工程教育部重点实验室副教授、博士生导师。

[http://lixirong.net/]

Jiebo Luo IEEE/SPIE Fellow、长江讲座美国罗彻斯特大学教授

[http://www.cs.rochester.edu/u/jluo/]

Subhashini Venugopalan

[https://www.cs.utexas.edu/~vsub/\]

Datasets

MSR-VTT dataset 该数据集为ACM Multimedia 2016 的 Microsoft Research - Video to Text (MSR-VTT) Challenge。地址为 Microsoft Multimedia Challenge 。该数据集包含10000个视频片段（video clip），被分为训练，验证和测试集三部分。每个视频片段都被标注了大概20条英文句子。此外，MSR-VTT还提供了每个视频的类别信息（共计20类），这个类别信息算是先验的，在测试集中也是已知的。同时，视频都是包含音频信息的。该数据库共计使用了四种机器翻译的评价指标，分别为：METEOR, BLEU@1-4,ROUGE-L,CIDEr。

[https://www.microsoft.com/en-us/research/publication/msr-vtt-large-video-description-dataset-bridging-video-language-supplementary-material/]
[http://ms-multimedia-challenge.com/]

YouTube2Text dataset(or called MSVD dataset) 该数据集同样由Microsoft Research提供，地址为 Microsoft Research Video Description Corpus 。该数据集包含1970段YouTube视频片段（时长在10-25s之间），每段视频被标注了大概40条英文句子。