机器学习的一个分支,它基于试图使用包含复杂结构或由多重非线性变换构成的多个处理层对数据进行高层抽象的一系列算法。

知识荟萃

Deep Learning 深度学习 专知荟萃

入门学习

  1. 《一天搞懂深度学习》台大 李宏毅 300页PPT

  2. Deep Learning(深度学习)学习笔记整理系列之(1-8)

  3. 深层学习为何要“Deep”(上,下)

  4. 《神经网络与深度学习》 作者:邱锡鹏 中文图书 2017

  5. 深度学习基础 206页PPT 邱锡鹏 复旦大学 2017年8月17日

  6. 《Neural Networks and Deep Learning》 By Michael Nielsen / Aug 2017

    - 原文:[http://neuralnetworksanddeeplearning.com/index.html]
    
  7. 李宏毅机器学习视频和笔记。

  8. 吴恩达(AndrewNg)深度学习视频和笔记

  9. 动手学深度学习pytorch版

综述

  1. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436-444. (Three Giants' Survey)

  2. Representation Learning: A Review and New Perspectives, Yoshua Bengio, Aaron Courville, Pascal Vincent, Arxiv, 2012.

  3. Deep learning in neural networks: An overview(2015)

  4. Text summarization using unsupervised deep learning(2017 - Elsevier)

进阶文章

Deep Belief Network(DBN)(Milestone of Deep Learning Eve)

ImageNet Evolution(Deep Learning broke out from here)

Model

Optimizations

  • Sutskever, Ilya, et al. "On the importance of initialization and momentum in deep learning." ICML (3) 28 (2013): 1139-1147.
  • Kingma, Diederik, and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).
  • Andrychowicz, Marcin, et al. "Learning to learn by gradient descent by gradient descent." arXiv preprint arXiv:1606.04474 (2016).
  • Han, Song, Huizi Mao, and William J. Dally. "Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding." CoRR, abs/1510.00149 2 (2015).
  • Iandola, Forrest N., et al. "SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size." arXiv preprint arXiv:1602.07360 (2016).

Unsupervised Learning / Deep Generative Model

RNN / Sequence-to-Sequence Model

Neural Turing Machine

  1. Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014).
  2. Zaremba, Wojciech, and Ilya Sutskever. "Reinforcement learning neural Turing machines." arXiv preprint arXiv:1505.00521 362 (2015).

Deep Reinforcement Learning

Deep Transfer Learning / Lifelong Learning / especially for RL

One Shot Deep Learning

NLP(Natural Language Processing)

Object Detection

Visual Tracking

Image Caption

Machine Translation

  • Luong, Minh-Thang, et al. "Addressing the rare word problem in neural machine translation." arXiv preprint arXiv:1410.8206 (2014).

  • Sennrich, et al. "Neural Machine Translation of Rare Words with Subword Units". In arXiv preprint arXiv:1508.07909, 2015.

  • Luong, Minh-Thang, Hieu Pham, and Christopher D. Manning. "Effective approaches to attention-based neural machine translation." arXiv preprint arXiv:1508.04025 (2015).

  • Chung, et al. "A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation". In arXiv preprint arXiv:1603.06147, 2016.

  • Lee, et al. "Fully Character-Level Neural Machine Translation without Explicit Segmentation". In arXiv preprint arXiv:1610.03017, 2016.

  • Wu, Schuster, Chen, Le, et al. "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation". In arXiv preprint arXiv:1609.08144v2, 2016.

  • Surabhi Punjabi, Harish Arsikere, Sri Garimella "Language Model Bootstrapping Using Neural Machine Translation For Conversational Speech Recognition"(2019)

Robotics

  • Koutník, Jan, et al. "Evolving large-scale neural networks for vision-based reinforcement learning." Proceedings of the 15th annual conference on Genetic and evolutionary computation. ACM, 2013.
  • Levine, Sergey, et al. "End-to-end training of deep visuomotor policies." Journal of Machine Learning Research 17.39 (2016): 1-40.
  • Pinto, Lerrel, and Abhinav Gupta. "Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours." arXiv preprint arXiv:1509.06825 (2015).
  • Levine, Sergey, et al. "Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection." arXiv preprint arXiv:1603.02199 (2016).
  • Zhu, Yuke, et al. "Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning." arXiv preprint arXiv:1609.05143 (2016).
  • Yahya, Ali, et al. "Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Search." arXiv preprint arXiv:1610.00673 (2016).
  • Gu, Shixiang, et al. "Deep Reinforcement Learning for Robotic Manipulation." arXiv preprint arXiv:1610.00633 (2016).
  • A Rusu, M Vecerik, Thomas Rothörl, N Heess, R Pascanu, R Hadsell."Sim-to-Real Robot Learning from Pixels with Progressive Nets." arXiv preprint arXiv:1610.04286 (2016).
  • Mirowski, Piotr, et al. "Learning to navigate in complex environments." arXiv preprint arXiv:1611.03673 (2016).
  • Zehui Meng, Qi Heng Ho, Zefan Huang, Hongliang Guo, Marcelo H. Ang Jr., Daniela Rus "Online Multi-Target Tracking for Maneuvering Vehicles in Dynamic Road Context"(2019)

Object Segmentation

Medical Image Analysis

Tutorial

  1. UFLDL Tutorial 1
  2. UFLDL Tutorial 2
  1. Deep Learning for NLP (without Magic)
  1. A Deep Learning Tutorial: From Perceptrons to Deep Networks
  1. Deep Learning from the Bottom up
  1. Theano Tutorial
  1. Neural Networks for Matlab
  1. Using convolutional neural nets to detect facial keypoints tutorial
  1. Pytorch Tutorials
  1. The Best Machine Learning Tutorials On The Web
  1. VGG Convolutional Neural Networks Practical
  1. TensorFlow tutorials
  1. More TensorFlow tutorials
  1. TensorFlow Python Notebooks
  1. Keras and Lasagne Deep Learning Tutorials
  1. Classification on raw time series in TensorFlow with a LSTM RNN
  1. Using convolutional neural nets to detect facial keypoints tutorial
  1. TensorFlow-World
  1. Deep Learning NIPS’2015 Tutorial Geoff Hinton, Yoshua Bengio & Yann LeCun 深度学习三巨头共同主持

视频教程

Courses

  1. Machine Learning - Stanford
  1. Machine Learning - Caltech
  1. Machine Learning - Carnegie Mellon
  1. Neural Networks for Machine Learning
  1. Neural networks class
  1. Deep Learning Course
  1. A.I - Berkeley
  1. A.I - MIT
  1. Vision and learning - computers and brains
  1. Convolutional Neural Networks for Visual Recognition - Stanford
  1. Convolutional Neural Networks for Visual Recognition - Stanford
  1. Deep Learning for Natural Language Processing - Stanford
  1. Neural Networks - usherbrooke
  1. Machine Learning - Oxford
  1. Deep Learning - Nvidia
  1. Graduate Summer School: Deep Learning, Feature Learning
  1. Deep Learning - Udacity/Google
  1. Deep Learning - UWaterloo
  1. Statistical Machine Learning - CMU
  1. Deep Learning Course
  1. Bay area DL school
  • [http://www.bayareadlschool.org/] by Andrew Ng, Yoshua Bengio, Samy Bengio, Andrej Karpathy, Richard Socher, Hugo Larochelle and many others @ Stanford, CA (2016)
  1. Designing, Visualizing and Understanding Deep Neural Networks-UC Berkeley
  1. UVA Deep Learning Course
  1. MIT 6.S094: Deep Learning for Self-Driving Cars
  1. MIT 6.S191: Introduction to Deep Learning
  1. Berkeley CS 294: Deep Reinforcement Learning
  1. [Keras in Motion video course
  1. Practical Deep Learning For Coders

Videos and Lectures

  1. How To Create A Mind
  1. Deep Learning, Self-Taught Learning and Unsupervised Feature Learning
  1. Recent Developments in Deep Learning
  1. The Unreasonable Effectiveness of Deep Learning
  1. Deep Learning of Representations
  1. Principles of Hierarchical Temporal Memory
  1. Machine Learning Discussion Group - Deep Learning w/ Stanford AI Lab
  1. Making Sense of the World with Deep Learning
  1. Demystifying Unsupervised Feature Learning
  1. Visual Perception with Deep Learning
  1. The Next Generation of Neural Networks
  1. The wonderful and terrifying implications of computers that can learn
  1. Unsupervised Deep Learning - Stanford
  1. Natural Language Processing
  1. A beginners Guide to Deep Neural Networks
  1. Deep Learning: Intelligence from Big Data
  1. Introduction to Artificial Neural Networks and Deep Learning
  1. NIPS 2016 lecture and workshop videos

代码

  1. Caffe
  1. Torch7
  1. Theano
  1. cuda-convnet
  1. convetjs
  1. Ccv
  1. NuPIC -[http://numenta.org/nupic.html]
  2. DeepLearning4J
  1. Brain
  1. DeepLearnToolbox
  1. Deepnet
  1. Deeppy -[https://github.com/andersbll/deeppy]
  2. JavaNN
  1. hebel
  1. Mocha.jl
  1. OpenDL
  1. cuDNN
  1. MGL
  1. Knet.jl
  1. Nvidia DIGITS - a web app based on Caffe
  1. Neon - Python based Deep Learning Framework
  1. Keras - Theano based Deep Learning Library
  1. Chainer - A flexible framework of neural networks for deep learning
  1. RNNLM Toolkit
  1. RNNLIB - A recurrent neural network library
  1. char-rnn
  1. MatConvNet: CNNs for MATLAB
  1. Minerva - a fast and flexible tool for deep learning on multi-GPU
  1. Brainstorm - Fast, flexible and fun neural networks.
  1. Tensorflow - Open source software library for numerical computation using data flow graphs
  1. DMTK - Microsoft Distributed Machine Learning Tookit
  1. Scikit Flow - Simplified interface for TensorFlow [mimicking Scikit Learn]
  1. MXnet - Lightweight, Portable, Flexible Distributed/Mobile Deep Learning framework
  1. Veles - Samsung Distributed machine learning platform
  1. Marvin - A Minimalist GPU-only N-Dimensional ConvNets Framework
  1. Apache SINGA - A General Distributed Deep Learning Platform
  1. DSSTNE - Amazon's library for building Deep Learning models
  1. SyntaxNet - Google's syntactic parser - A TensorFlow dependency library
  1. mlpack - A scalable Machine Learning library
  1. Torchnet - Torch based Deep Learning Library
  1. Paddle - PArallel Distributed Deep LEarning by Baidu
  1. NeuPy - Theano based Python library for ANN and Deep Learning
  1. Lasagne - a lightweight library to build and train neural networks in Theano
  1. nolearn - wrappers and abstractions around existing neural network libraries, most notably Lasagne
  1. Sonnet - a library for constructing neural networks by Google's DeepMind
  1. PyTorch - Tensors and Dynamic neural networks in Python with strong GPU acceleration
  1. CNTK - Microsoft Cognitive Toolkit

领域专家

  1. Aaron Courville
  2. Abdel-rahman Mohamed
  3. Adam Coates
  4. Alex Acero
  5. Alex Krizhevsky
  6. Alexander Ilin
  7. Amos Storkey
  8. Andrej Karpathy
  9. Andrew M. Saxe
  10. Andrew Ng
  1. Andrew W. Senior
  1. Andriy Mnih
  1. Ayse Naz Erkan
  1. Benjamin Schrauwen
  1. Bernardete Ribeiro
  1. Bo David Chen
  1. Boureau Y-Lan
  1. Brian Kingsbury
  1. Christopher Manning
  1. Clement Farabet
  1. Dan Claudiu Cireșan
  1. David Reichert
  1. Derek Rose
  1. Dong Yu
  1. Drausin Wulsin
  1. Erik M. Schmidt
  1. Eugenio Culurciello
  1. Frank Seide
  1. Galen Andrew
  1. Geoffrey Hinton
  1. George Dahl
  1. Graham Taylor
  1. Grégoire Montavon
  1. Guido Francisco Montúfar
  1. Guillaume Desjardins
  1. Hannes Schulz
  1. Hélène Paugam-Moisy
  1. Honglak Lee
  1. Hugo Larochelle
  1. Ilya Sutskever
  1. Itamar Arel
  1. James Martens
  1. Jason Morton
  1. Jason Weston
  1. Jeff Dean
  1. Jiquan Mgiam
  1. Joseph Turian
  1. Joshua Matthew Susskind
  1. Jürgen Schmidhuber
  1. Justin A. Blanco
  1. Koray Kavukcuoglu
  1. KyungHyun Cho
  1. Li Deng
  1. Lucas Theis
  1. Ludovic Arnold
  1. Marc'Aurelio Ranzato
  1. Martin Längkvist
  1. Misha Denil
  1. Mohammad Norouzi
  1. Nando de Freitas
  1. Navdeep Jaitly
  1. Nicolas Le Roux
  1. Nitish Srivastava
  1. Noel Lopes
  1. Oriol Vinyals
  1. Pascal Vincent
  1. Patrick Nguyen
  1. Pedro Domingos
  1. Peggy Series
  1. Pierre Sermanet
  1. Piotr Mirowski
  1. Quoc V. Le
  1. Reinhold Scherer
  1. Richard Socher
  1. Rob Fergus
  1. Robert Coop
  1. Robert Gens
  1. Roger Grosse
  1. Ronan Collobert
  1. Ruslan Salakhutdinov
  1. Sebastian Gerwinn
  1. Stéphane Mallat
  1. Sven Behnke
  1. Tapani Raiko
  1. Tara Sainath
  1. Tijmen Tieleman
  1. Tom Karnowski
  1. Tomáš Mikolov
  1. Ueli Meier
  1. Vincent Vanhoucke
  1. Volodymyr Mnih
  1. Yann LeCun
  1. Yichuan Tang
  1. Yoshua Bengio
  1. Yotaro Kubo
  1. Youzhi [Will] Zou
  1. Fei-Fei Li
  1. Ian Goodfellow
  1. Robert Laganière

重要网站收藏

  1. deeplearning.net
  2. deeplearning.stanford.edu
  3. nlp.stanford.edu
  4. ai-junkie.com
  5. cs.brown.edu/research/ai
  6. eecs.umich.edu/ai
  7. cs.utexas.edu/users/ai-lab
  8. cs.washington.edu/research/ai
  9. aiai.ed.ac.uk
  10. www-aig.jpl.nasa.gov
  1. csail.mit.edu
  1. cgi.cse.unsw.edu.au/~aishare
  1. cs.rochester.edu/research/ai
  1. ai.sri.com
  1. isi.edu/AI/isd.htm
  1. nrl.navy.mil/itd/aic
  1. hips.seas.harvard.edu
  1. AI Weekly
  1. stat.ucla.edu
  1. deeplearning.cs.toronto.edu
  1. jeffdonahue.com/lrcn/
  1. visualqa.org
  1. www.mpi-inf.mpg.de/departments/computer-vision...
  1. Deep Learning News
  1. Machine Learning is Fun! Adam Geitgey's Blog

免费在线图书

  1. Deep Learning
  2. Neural Networks and Deep Learning
  3. Deep Learning
  4. Deep Learning Tutorial
  5. neuraltalk
  6. An introduction to genetic algorithms
  7. Artificial Intelligence: A Modern Approach
  8. Deep Learning in Neural Networks: An Overview

Datasets

  1. MNIST
  2. Google House Numbers
  3. CIFAR-10 and CIFAR-100
  4. IMAGENET
  5. Tiny Images
  6. Flickr Data
  7. Berkeley Segmentation Dataset 500
  8. UC Irvine Machine Learning Repository
  9. Flickr 8k
  10. Flickr 30k
  11. Microsoft COCO
  12. VQA
  13. Image QA
  14. AT&T Laboratories Cambridge face database
  15. AVHRR Pathfinder
  16. Air Freight
    • [http://www.anc.ed.ac.uk/~amos/afreightdata.html] - The Air Freight data set is a ray-traced image sequence along with ground truth segmentation based on textural characteristics. [455 images + GT, each 160x120 pixels]. [Formats: PNG]
  17. Amsterdam Library of Object Images
    • [http://www.science.uva.nl/~aloi/] - ALOI is a color image collection of one-thousand small objects, recorded for scientific purposes. In order to capture the sensory variation in object recordings, we systematically varied viewing angle, illumination angle, and illumination color for each object, and additionally captured wide-baseline stereo images. We recorded over a hundred images of each object, yielding a total of 110,250 images for the collection. [Formats: png]
  18. Annotated face, hand, cardiac & meat images
    • [http://www.imm.dtu.dk/~aam/] - Most images & annotations are supplemented by various ASM/AAM analyses using the AAM-API. [Formats: bmp,asf]
  19. Image Analysis and Computer Graphics
  20. Brown University Stimuli
  21. CAVIAR video sequences of mall and public space behavior
  22. Machine Vision Unit
  23. CCITT Fax standard images
  24. CMU CIL's Stereo Data with Ground Truth[cil-ster.html] - 3 sets of 11 images, including color tiff images with spectroradiometry [Formats: gif, tiff]
  25. CMU PIE Database
  26. CMU VASC Image Database
  27. Caltech Image Database
  28. Columbia-Utrecht Reflectance and Texture Database
    • [http://www.cs.columbia.edu/CAVE/curet/] - Texture and reflectance measurements for over 60 samples of 3D texture, observed with over 200 different combinations of viewing and illumination directions. [Formats: bmp]
  29. Computational Colour Constancy Data
    • [http://www.cs.sfu.ca/~colour/data/index.html] - A dataset oriented towards computational color constancy, but useful for computer vision in general. It includes synthetic data, camera sensor data, and over 700 images. [Formats: tiff]
  30. Computational Vision Lab
  31. Content-based image retrieval database
  32. Efficient Content-based Retrieval Group
  33. Densely Sampled View Spheres
  34. Computer Science VII [Graphical Systems]
  35. Digital Embryos
  36. Univerity of Minnesota Vision Lab
  37. El Salvador Atlas of Gastrointestinal VideoEndoscopy
  38. FG-NET Facial Aging Database
  39. FVC2000 Fingerprint Databases
    • [http://bias.csr.unibo.it/fvc2000/] - FVC2000 is the First International Competition for Fingerprint Verification Algorithms. Four fingerprint databases constitute the FVC2000 benchmark [3520 fingerprints in all].
  40. Biometric Systems Lab
  41. Face and Gesture images and image sequences
    • [http://www.fg-net.org] - Several image datasets of faces and gestures that are ground truth annotated for benchmarking
  42. German Fingerspelling Database
  43. Language Processing and Pattern Recognition
  44. Groningen Natural Image Database
  45. ICG Testhouse sequence
  46. Institute of Computer Graphics and Vision
  47. IEN Image Library
  48. INRIA's Syntim images database
  49. INRIA
  50. INRIA's Syntim stereo databases
  51. Image Analysis Laboratory
  52. Image Analysis Laboratory
  53. Image Database
  54. JAFFE Facial Expression Image Database
    • [http://www.mis.atr.co.jp/~mlyons/jaffe.html] - The JAFFE database consists of 213 images of Japanese female subjects posing 6 basic facial expressions as well as a neutral pose. Ratings on emotion adjectives are also available, free of charge, for research purposes. [Formats: TIFF Grayscale images.]
  55. ATR Research, Kyoto, Japan
  56. JISCT Stereo Evaluation
    • [ftp://ftp.vislist.com/IMAGERY/JISCT/] - 44 image pairs. These data have been used in an evaluation of stereo analysis, as described in the April 1993 ARPA Image Understanding Workshop paper ``The JISCT Stereo Evaluation'' by R.C.Bolles, H.H.Baker, and M.J.Hannah, 263--274 [Formats: SSI]
  57. MIT Vision Texture
  58. MIT face images and more
  59. Machine Vision
  60. Mammography Image Databases
  61. ftp://ftp.cps.msu.edu/pub/prip
  62. Middlebury Stereo Data Sets with Ground Truth
    • [http://www.middlebury.edu/stereo/data.html] - Six multi-frame stereo data sets of scenes containing planar regions. Each data set contains 9 color images and subpixel-accuracy ground-truth data. [Formats: ppm]
  63. Middlebury Stereo Vision Research Page
  64. Modis Airborne simulator, Gallery and data set
  65. NIST Fingerprint and handwriting
  66. NIST Fingerprint data
  67. NLM HyperDoc Visible Human Project
  68. National Design Repository
    • [http://www.designrepository.org] - Over 55,000 3D CAD and solid models of [mostly] mechanical/machined engineerign designs. [Formats: gif,vrml,wrl,stp,sat]
  69. Geometric & Intelligent Computing Laboratory
  70. OSU [MSU] 3D Object Model Database
  71. OSU [MSU/WSU] Range Image Database
  72. OSU/SAMPL Database: Range Images, 3D Models, Stills, Motion Sequences
  73. Signal Analysis and Machine Perception Laboratory
  74. Otago Optical Flow Evaluation Sequences
  75. Vision Research Group
  76. ftp://ftp.limsi.fr/pub/quenot/opflow/testdata/piv/
    • [ftp://ftp.limsi.fr/pub/quenot/opflow/testdata/piv/] - Real and synthetic image sequences used for testing a Particle Image Velocimetry application. These images may be used for the test of optical flow and image matching algorithms. [Formats: pgm [raw]]
  77. LIMSI-CNRS/CHM/IMM/vision
  78. LIMSI-CNRS
  79. Photometric 3D Surface Texture Database
  80. SEQUENCES FOR OPTICAL FLOW ANALYSIS [SOFA]
    • [http://www.cee.hw.ac.uk/~mtc/sofa] - 9 synthetic sequences designed for testing motion analysis applications, including full ground truth of motion and camera parameters. [Formats: gif]
  81. Computer Vision Group
  82. Sequences for Flow Based Reconstruction
  83. Stereo Images with Ground Truth Disparity and Occlusion
    • [http://www-dbv.cs.uni-bonn.de/stereo_data/] - a small set of synthetic images of a hallway with varying amounts of noise added. Use these images to benchmark your stereo algorithm. [Formats: raw, viff [khoros], or tiff]
  84. Stuttgart Range Image Database
  85. Department Image Understanding
  86. The AR Face Database
  87. Purdue Robot Vision Lab
  1. The MIT-CSAIL Database of Objects and Scenes
  • [http://web.mit.edu/torralba/www/database.html] - Database for testing multiclass object detection and scene recognition algorithms. Over 72,000 images with 2873 annotated frames. More than 50 annotated object classes. [Formats: jpg]
  1. The RVL SPEC-DB [SPECularity DataBase]
  • [http://rvl1.ecn.purdue.edu/RVL/specularity_database/] - A collection of over 300 real images of 100 objects taken under three different illuminaiton conditions [Diffuse/Ambient/Directed]. -- Use these images to test algorithms for detecting and compensating specular highlights in color images. [Formats: TIFF ]
  1. Robot Vision Laboratory
  1. The Xm2vts database
  • [http://xm2vtsdb.ee.surrey.ac.uk] - The XM2VTSDB contains four digital recordings of 295 people taken over a period of four months. This database contains both image and video data of faces.
  1. Centre for Vision, Speech and Signal Processing
  1. Traffic Image Sequences and 'Marbled Block' Sequence
  1. IAKS/KOGS
  1. U Bern Face images
  1. U Michigan textures
  1. U Oulu wood and knots database
  1. UCID - an Uncompressed Colour Image Database
  1. UMass Vision Image Archive
  1. UNC's 3D image database
  1. USF Range Image Data with Segmentation Ground Truth
  1. University of Oulu Physics-based Face Database
  1. Machine Vision and Media Processing Unit
  1. University of Oulu Texture Database
  • [http://www.outex.oulu.fi] - Database of 320 surface textures, each captured under three illuminants, six spatial resolutions and nine rotation angles. A set of test suites is also provided so that texture segmentation, classification, and retrieval algorithms can be tested in a standard manner. [Formats: bmp, ras, xv]
  1. Machine Vision Group
  1. Usenix face database
  1. View Sphere Database
  1. PRIMA, GRAVIR
  1. Vision-list Imagery Archive
  1. Wiry Object Recognition Database
  • [http://www.cs.cmu.edu/~owenc/word.htm] - Thousands of images of a cart, ladder, stool, bicycle, chairs, and cluttered scenes with ground truth labelings of edges and regions. [Formats: jpg]
  1. 3D Vision Group
  1. Yale Face Database
  1. Yale Face Database B
  1. Center for Computational Vision and Control
  2. DeepMind QA Corpus
  3. YouTube-8M Dataset
    • [https://research.google.com/youtube8m/] - YouTube-8M is a large-scale labeled video dataset that consists of 8 million YouTube video IDs and associated labels from a diverse vocabulary of 4800 visual entities.
  4. Open Images dataset

初步版本,水平有限,有错误或者不完善的地方,欢迎大家提建议和补充,会一直保持更新,本文为专知内容组原创内容,未经允许不得转载,如需转载请发送邮件至fangquanyi@gmail.com 或 联系微信专知小助手(Rancho_Fang)

敬请关注http://www.zhuanzhi.ai 和关注专知公众号,获取第一手AI相关知识

最近更新:2019-12-10

VIP内容

【导读】关于《深度学习系统优化》综述论文

深度学习(Deep Learning, DL)模型在视觉、语言、医疗、商业广告、娱乐等许多应用领域都取得了优异的表现。随着DL应用和底层服务硬件的快速发展,都显示出了强大的扩展趋势,即模型扩展和计算扩展,例如,最近的预训练模型具有数千亿参数,内存消耗约TB级,以及提供数百个TFLOPS的最新GPU加速器。随着规模化趋势的出现,DL推理服务系统出现了新的问题和挑战,逐步向大规模深度学习服务系统发展。本综述旨在总结和分类大规模深度学习服务系统出现的挑战和优化机会。通过提供一种新颖的分类方法,总结计算范式,阐述最新的技术进展,我们希望本综述能够揭示新的优化视角,激发大规模深度学习系统优化的新工作。

https://www.zhuanzhi.ai/paper/9ee7ca2cf6457080794f9b6608f09e7a

深度学习(DEEP Learning, DL)模型,如CNN[15,36,44],Transformers[2,7,10,29]和推荐模型[31,41]在许多认知任务,如视觉、语音和语言应用中取得了优异的表现,这在许多领域产生重要的应用,如医学图像分析[38],照片造型[34],机器翻译[40],产品推荐[31]、定制广告[13]、游戏[21]等。这种广泛的DL应用带来了巨大的市场价值,也带来了大量的DL服务流量。例如,FB有18.2亿的日活跃用户[11]。广告推荐查询的数量可以达到每秒10M查询。消费者生成数据的巨大增长和DL服务的使用也推动了对以人工智能为中心的数据中心(如亚马逊AWS[27]和微软Azure[6])的需求,以及对GPU等强大的DL加速器的日益采用。根据[35]的报告,2018年,GPU在全球数据中心加速器市场上以298300万美元的份额占据了85%的主要份额。到2025年,该产品将达到298.19亿美元。

随着市场需求的不断增长,DL应用和底层服务硬件在计算可扩展(例如,增加计算并行性、内存和存储以服务于更大的模型)和模型扩展(例如,更高的结构复杂性、计算工作量、参数大小以获得更好的精度),这大大复杂化了服务系统的管理和优化。一方面,如图1 (a)所示,在计算扩展趋势下,具有大规模计算并行性的GPU已成为近年来数据中心DL计算加速器的主要类型之一,并保持着持续的指数级性能缩放。最近的GPU如NVIDIA Tesla V100提供每秒130拉浮点运算(TFLOPS),和900 GB / s内存带宽, 和这些数字进一步增加到312 TFLOPS和1.6 TB / s内存带宽,可以提供数万DL模型如ResNet50[15]同时提供更高的效率(性能/瓦特)。另一方面,如图1 (b)所示,模型规模已经被证明是获得更好的精度的最重要的因素之一,其有效性在实践中一致显示在所有领域的工业超大模型,如视觉模型BiT [22], NLP模型BERT [7],GPT3[2]和深度学习推荐模型DLRM[31]。例如,最近的超大型模型MT-NLG[29]已经实现了5300亿参数。工业级商用DLRM[31]已达到~ TB模型大小,大大超过了单机存储能力,需要多个设备才能进行协同计算。

在这样的背景下,我们观察到目前的DL系统社区对大规模深度学习系统(LDS)仍然缺乏足够的认识和关注,忽视了出现的挑战和机遇: 传统的DL系统优化通常集中在单模型单机推理设置(即一对一映射)。然而,LDS具有更大的DL模型和更强大的硬件,能够实现更灵活的推理计算,将多实例到单设备、一实例到多设备、甚至多实例到多设备映射变为现实。例如,计算缩放(如GPU、TPU)促使许多研究工作在单个设备上进行多模型推理,例如将一个GPU划分为多个容器化vGPU或多实例GPU (MIG),以获得更好的硬件利用率、更高的服务吞吐量和成本效率。考虑到实际的成本管理(例如,总拥有成本,TCO),服务大量推理查询的数据中心也倾向于迁移到多租户推理服务,例如,将多个推理查询放置在同一设备上,从而产生新的优化目标(例如,每秒服务的总查询,以及来自传统单租户推断的约束(例如,服务水平协议、SLA)。类似地,模型扩展也提出了新的一对多推理场景的要求。目前的超大型模型(如DLRM)在推理过程中需要耗费大量的内存(∼TB不量化),这需要新的协同计算范式,如异构计算或分布式推理。这种协作服务涉及远程进程调用(RPC)和低带宽通信,这带来了与传统的单设备推理截然不同的瓶颈。由于涉及到以上所有场景,现代数据中心面临更复杂的多对多场景,需要专门的推理查询调度,如服务路由器和计算设备管理,以获得更好的服务性能,如延迟、吞吐量和成本等。

在本文中,我们提出了一种新的计算范式分类法,总结了新的优化目标,阐述了新的技术设计视角,并为未来的LDS优化提供了见解。

  • 多对多计算范式以DNN实例(I)和计算设备(D)之间的关系为特征,新兴的LDS计算范式除了单实例单设备(SISD)外,还可以分为三个新的类别,即多实例单设备(MISD),单实例多设备(SIMD)和多实例多设备(MIMD),如图2所示。与专注于单模型性能的SISD不同,LDS工作有不同的优化目标,包括推理延迟、服务吞吐量、成本、可扩展性、服务质量等。例如,多租户推理(multi-tenant inference, MISD)的目标是提高服务吞吐量和电力效率,而超大规模模型推理服务的目标是以低成本提高硬件可伸缩性。

  • 大规模设计和技术由于推理服务的规模,LDS工作也在算法创新、运行时调度和资源管理方面面临许多优化挑战和机遇。例如,多租户推理优化寻求细粒度的硬件资源分区和作业调度,例如空间/时间共享,以提供QoS保证。由于延迟通信瓶颈,分布式推理需要专门的模型-硬件协同优化,例如高效的模型分片和平衡协作等。

通过对现有工作的总结,我们旨在对出现的挑战、机遇和创新提供一个全面的调研,从而推动LDS运营和优化的新创新。调研的其余部分组织如下:第2节介绍了研究的初步内容,包括我们对LDS的分类,并说明了本次调研的范围。第3节总结了在多实例单设备(MISD)优化方面面临的挑战和最近的工作;第4节总结了单实例多设备(SIMD)优化方面的研究工作;第5节总结了这项工作。

成为VIP会员查看完整内容
0
9

最新论文

Propelled by new designs that permit to circumvent the spectral bias, implicit neural representations (INRs) have recently emerged as a promising alternative to classical discretized representations of signals. Nevertheless, despite their practical success, we still lack a proper theoretical characterization of how INRs represent signals. In this work, we aim to fill this gap, and we propose a novel unified perspective to theoretically analyse INRs. Leveraging results from harmonic analysis and deep learning theory, we show that most INR families are analogous to structured signal dictionaries whose atoms are integer harmonics of the set of initial mapping frequencies. This structure allows INRs to express signals with an exponentially increasing frequency support using a number of parameters that only grows linearly with depth. Afterwards, we explore the inductive bias of INRs exploiting recent results about the empirical neural tangent kernel (NTK). Specifically, we show that the eigenfunctions of the NTK can be seen as dictionary atoms whose inner product with the target signal determines the final performance of their reconstruction. In this regard, we reveal that meta-learning the initialization has a reshaping effect of the NTK analogous to dictionary learning, building dictionary atoms as a combination of the examples seen during meta-training. Our results permit to design and tune novel INR architectures, but can also be of interest for the wider deep learning theory community.

0
0
下载
预览
Top