Deep Learning and Its Applications


Research Seminar
Deep Learning Research & Application Centre (DLC)
Hang Seng Management College




Albert Au Yeung
20th, July, 2017

Agenda

  • What is deep learning?
  • What can deep learning be used for?
  • How can we use deep learning?
  • Applications of deep learning
  • Machine learning and Social Science

About Me

  • Lead Engineer in Machine Learning at Zwoop
  • Co-founder of Axon Labs Ltd.
  • Research positions at NTT CSLab, ASTRI, Huawei Noah's Ark Lab
  • Research interests and experience:
    • Machine learning, natural language processing and social network analysis
    • Consumer behaviour analysis and recommendation systems
    • Image search engine based on deep learning
    • Computational history and social science

Trend

What is Deep Learning?

What can DL be Used for?

Deep learning is mainly used to perform one of the following two tasks:

Computer Vision

  • Computer vision in the old days rely heavily on specific feature extraction methods
  • Examples of feature detection methods:


Ref: Image Classification with Bag of Visual Words - MathWorks

Convolutional Neural Network (CNN)

  • Instead of fully connected layers, a CNN also involves layers that are only partially connected to the previous or the next (kernels or filters or feature detectors)
  • Filters of different sizes are applied to the whole image to extract features useful for the task
  • Representation learning (e.g. automatic feature extraction from images)

Ref: Gradient-Based Learning Applied to Document Recognition (LeCun et al., 1998)

Convolutional Neural Network (CNN)

  • Huge improvement in the image classification task on ImageNet

Ref: Musings on Deep Learning. Li Jiang, 2014

Recurrent Neural Network (RNN)

  • Extending neural network to handle sequential inputs (e.g. language, time-series data, speech and audio signals, etc.)
  • Outputs depend on previous computations
  • Drastically outperform traditional approaches in certain domains (e.g. Deep Speech (Hannun et al., 2014))

Ref: Understanding LSTM Networks (Colab, 2015)

Attention Mechanism

  • Focusing on a certain area / period of an input when trying to predict the outputs
  • Found particularly useful when performing machine translation or image caption generation

Ref: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Xu et al., 2015)

Latest Research in DL - Vision

Colourisation of black and white images/videos (Iizuka et al. 2016)

Latest Research in DL - Vision

Neural Style / Style Transfer (Gatys et al. 2016) Source code available on Github

Latest Research in DL - Speech

Synthesizing Obama: Learning Lip Sync from Audio (Suwajankorn et al. 2017)

Latest Research in DL - Language

Character Sequence Models for Colorful Words (Kawakami et al. 2016)

Latest Research in DL - Language

A Neural Attention Model for Sentence Summarization (Rush et al. 2016)

How can we use DL?

Training a deep neural network usually requires a huge amount of data. For example, the CNNs such as VGG16 and Inception V3 are trained on the ImageNet dataset with ~1.2 million images of 1,000 different classes.

  • How can one start with DL:
    • collect enough data to start with!
    • use pre-trained models directly
    • use pre-trained models and do fine-tuning, a.k.a. transfer learning

Transfer Learning

  • For common tasks such as image classification, it is likely that filters (feature extractors) trained on one datasets are useful on many other datasets as well
  • Transfer learning refers to the task of applying knowledge gained in one problem to another different but related problem
  • Transfer learning in deep learning
    1. Using a pre-trained neural nework as a feature extractor
    2. Perform fine-tuning on a pre-trained network
  • Ref: http://cs231n.github.io/transfer-learning/

Transfer Learning (2)


Ref: Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks (Oquab et al., 2014)

Example of Transfer Learning

  • TourLens - Recognize ~50 objects for tourists in Hong Kong
  • Around ~100 training images for each class
  • Transfer learning using pre-trained AlexNet on ImageNet

Example of Transfer Learning - Pre-trained Word Embeddings

  • Word embeddings are dense vector representations of words that reflect their contextual similarity
  • Different pre-trained word vectors are available, e.g. word2vec, GloVe, fastText
  • Can be used to initialise the embedding layer when training NLP models

Ref: GloVe: Global Vectors for Word Representation: https://nlp.stanford.edu/projects/glove/

Common Deep Learning Frameworks

Machine learning and Social Science

Machine learning is not only for solving computational problems, but can also be applied to study problems in humanities and social science

Summary

  • Deep learning is a powerful machine learning method for solving different tasks, especially when the problem can be defined as a classification or transformation problem
  • Moving from feature engineering to 'neural network engineering'
  • There are many open source deep learning frameworks that allow research and development in deep learning to be done more efficiently
  • Can deep learning be useful in solving problems in humanities and social science?

The End

Thank you for attending the seminar! Questions are welcome.

Contact me at:

======= Deep Learning and Its Applications | Albert Au Yeung

Deep Learning and Its Applications


Research Seminar
Deep Learning Research & Application Centre (DLC)
Hang Seng Management College




Albert Au Yeung
20th, July, 2017

Agenda

  • What is deep learning?
  • What can deep learning be used for?
  • How can we use deep learning?
  • Applications of deep learning
  • Machine learning and Social Science

About Me

  • Lead Engineer in Machine Learning at Zwoop
  • Co-founder of Axon Labs Ltd.
  • Research positions at NTT CSLab, ASTRI, Huawei Noah's Ark Lab
  • Research interests and experience:
    • Machine learning, natural language processing and social network analysis
    • Consumer behaviour analysis and recommendation systems
    • Image search engine based on deep learning
    • Computational history and social science

Trend

What is Deep Learning?

What can DL be Used for?

Deep learning is mainly used to perform one of the following two tasks:

Computer Vision

  • Computer vision in the old days rely heavily on specific feature extraction methods
  • Examples of feature detection methods:


Ref: Image Classification with Bag of Visual Words - MathWorks

Convolutional Neural Network (CNN)

  • Instead of fully connected layers, a CNN also involves layers that are only partially connected to the previous or the next (kernels or filters or feature detectors)
  • Filters of different sizes are applied to the whole image to extract features useful for the task
  • Representation learning (e.g. automatic feature extraction from images)

Ref: Gradient-Based Learning Applied to Document Recognition (LeCun et al., 1998)

Convolutional Neural Network (CNN)

  • Huge improvement in the image classification task on ImageNet

Ref: Musings on Deep Learning. Li Jiang, 2014

Recurrent Neural Network (RNN)

  • Extending neural network to handle sequential inputs (e.g. language, time-series data, speech and audio signals, etc.)
  • Outputs depend on previous computations
  • Drastically outperform traditional approaches in certain domains (e.g. Deep Speech (Hannun et al., 2014))

Ref: Understanding LSTM Networks (Colab, 2015)

Attention Mechanism

  • Focusing on a certain area / period of an input when trying to predict the outputs
  • Found particularly useful when performing machine translation or image caption generation

Ref: Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (Xu et al., 2015)

Latest Research in DL - Vision

Colourisation of black and white images/videos (Iizuka et al. 2016)

Latest Research in DL - Vision

Neural Style / Style Transfer (Gatys et al. 2016) Source code available on Github

Latest Research in DL - Speech

Synthesizing Obama: Learning Lip Sync from Audio (Suwajankorn et al. 2017)

Latest Research in DL - Language

Character Sequence Models for Colorful Words (Kawakami et al. 2016)

Latest Research in DL - Language

A Neural Attention Model for Sentence Summarization (Rush et al. 2016)

How can we use DL?

Training a deep neural network usually requires a huge amount of data. For example, the CNNs such as VGG16 and Inception V3 are trained on the ImageNet dataset with ~1.2 million images of 1,000 different classes.

  • How can one start with DL:
    • collect enough data to start with!
    • use pre-trained models directly
    • use pre-trained models and do fine-tuning, a.k.a. transfer learning

Transfer Learning

  • For common tasks such as image classification, it is likely that filters (feature extractors) trained on one datasets are useful on many other datasets as well
  • Transfer learning refers to the task of applying knowledge gained in one problem to another different but related problem
  • Transfer learning in deep learning
    1. Using a pre-trained neural nework as a feature extractor
    2. Perform fine-tuning on a pre-trained network
  • Ref: http://cs231n.github.io/transfer-learning/

Transfer Learning (2)


Ref: Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks (Oquab et al., 2014)

Example of Transfer Learning

  • TourLens - Recognize ~50 objects for tourists in Hong Kong
  • Around ~100 training images for each class
  • Transfer learning using pre-trained AlexNet on ImageNet

Example of Transfer Learning - Pre-trained Word Embeddings

  • Word embeddings are dense vector representations of words that reflect their contextual similarity
  • Different pre-trained word vectors are available, e.g. word2vec, GloVe, fastText
  • Can be used to initialise the embedding layer when training NLP models

Ref: GloVe: Global Vectors for Word Representation: https://nlp.stanford.edu/projects/glove/

Common Deep Learning Frameworks

Machine learning and Social Science

Machine learning is not only for solving computational problems, but can also be applied to study problems in humanities and social science

Summary

  • Deep learning is a powerful machine learning method for solving different tasks, especially when the problem can be defined as a classification or transformation problem
  • Moving from feature engineering to 'neural network engineering'
  • There are many open source deep learning frameworks that allow research and development in deep learning to be done more efficiently
  • Can deep learning be useful in solving problems in humanities and social science?

The End

Thank you for attending the seminar! Questions are welcome.

Contact me at: