Deep Neural Networks

This course will aim to introduce students to the core fundamentals of modern deep multi-layered neural networks, while still remaining grounded in practice. The underpinning assumption in its design is that while students may have experience (especially hands-on experience) in machine learning, data science or general software engineering — they have not worked with deep learning or taken prior courses in the area.

Frequency

This course normally runs twice a year.

Course dates

Objectives

At the conclusion of this course students should understand:

1. The principles and approaches for learning with deep neural networks.
2. The main variants of deep learning (such convolutional and recurrent architectures), and their typical applications.
3. The key concepts, issues and practices when training and modeling with deep architectures; as well as have hands-on experience in using deep learning frameworks for this purpose.
4. How to implement basic versions of some of the core deep network algorithms (such as back-propagation)
5. How deep learning fits within the context of other machine learning approaches, and what tasks it is considered to be suited and not well suited to perform.

Contents

Introduction to Neural Networks and Deep Learning

Background, History and Intuition

Supervised Training Methods, and Basic Architectures

Multi-layer Perceptrons, Backpropagation, Stochastic Gradient Descent

Convolutional Networks and Image Applications

Unsupervised Methods and Related Architectures

Autoencoders and Generative Adversarial Networks

Recurrent Networks and Time-series/Sequential Applications

Modeling Audio, and Case studies in Speech and Sounds

Data Augmentation and Transfer Learning

Practical Implications due to Hardware, Systems and the Cloud

Scalability and Efficiency of Training and Inference

Requirements

Software Tools: PyTorch and TensorFlow

Prerequisites: Students registering for this course must satisfy three area of background knowledge.

1. Mathematical Foundations — Undergraduate courses in the following topics: calculus, probability and linear algebra
2. Programming Skills — Undergraduate level programming courses or experience that indicates programming proficiency
3. Machine Learning Experience — Students are required to have taken at least one prior course in machine learning at an introductory level during earlier study. Such a course should have provided basic machine learning concepts and ideas, as well as described some popular modelling approaches (at the time the course was taken). There is no expectation this course would have covered any content related to deep learning or types of neural networks.

Compute Resources and Reading Material:

• Compute resources offered via Google Cloud (free)
• Recommended reading: Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning (http://www.deeplearningbook.org/)