This is the primary course website for Mathematical Foundations of Deep Neural Networks (심층신경망의 수학적 기초), M1407.001200, Spring 2024. We will also use eTL as a secondary website.

- No lecture on 05/27.
- On Tuesday, 05/28, 5:00–6:35 pm, I will give a talk entitled "세상을 바꿀 LLM의 사고력" in the Mathematics department's Gauss Colloquium. Those of you who are interested are encouraged to attend.
- No lecture on 06/03 and 06/05.

- Python Tutorial 1, Jupyter notebook code.
- Python Tutorial 2, Jupyter notebook code.
- Python Tutorial 3, Jupyter notebook code.
- Chapter 0: Prologue
- Chapter 1: Optimization and Stochastic Gradient Descent, Jupyter notebook code.
- Chapter 2: Shallow Neural Networks, Jupyter notebook code.
- Chapter 3: Convolutional Neural Networks, Jupyter notebook code.
- Chapter 4: CNNs for Non-Classification Supervised Learning, Jupyter notebook code.
- Chapter 5: Unsupervised Learning, Jupyter notebook code.
- Chapter A: Basics of Monte Carlo, Jupyter notebook code.

This class will have weekly homework assignments. Submit completed assignments through eTL.

- Homework 1, Due 03/11, 5pm, wideMinima.py, conv1D.py.
- Homework 2, Due 03/18, 5pm, twolayerSGD.py.
- Homework 3, Due 03/25, 5pm, threelayerSGD.py.
- Homework 4, Due 04/01, 5pm, lenet_original.py.
- Homework 5, Due 04/08, 5pm, mlp_backprop.py, label_memorization.py.
- Homework 6, Due 04/22, 5pm, ddescent.py.
- Homework 7, Due 04/29, 5pm, counting_params.py, bn_remove.py, smallNetSaved, LNiN.py.
- Homework 8, Due 05/06, 5pm.
- Homework 9, Due 05/20, 5pm, anomaly_detection.py, normalizingFlow1d.py.
- Homework 10, Due 05/27, 5pm, flow_inpainting.py, nice.pt.
- Homework 11, Due 06/10, 5pm, flow_anomaly.py, nice.pt.
- Homework 12, Due 06/18, 5pm, minimax_logistic.py, swiss_roll.py.

- [Week 1] Optimization and stochastic gradient descent
- [Week 2] Shallow neural networks and logistic regression.
- [Week 3] Multi-layer perceptron. Softmax regression.
- [Week 4] Convolutional layers, pooling layers, GPU computing, LeNet
- [Week 5] Data augmentation, regularization techniques: dropout, weight decay, early stopping
- [Week 6] Weight initialization, VGGNet, backprop
- [Week 7] Optimizers (ADAM, RMSProp), NiN network, GoogLeNet
- [Week 8] Batch normalization, ResNet, DenseNet
- [Week 9] ResNext, SENet, DNCNN, super-resolution, inverse problem
- [Week 10-11] Flow models
- [Week 12-13] Variational auto-encoders
- [Week 14-15] Generative adversarial networks

Course material will be posted on this website. eTL will be used for announcements, homework submission, and receiving homework and exam scores.

Ernest K. Ryu, 27-205,

Mondays and Wednesdays 5:00–6:15pm at 28-102.

The midterm and final exams will be hand-written (no computers), in-person, and 4 hours long.

- Midterm exam: Saturday, 04/13, 9:00am–1:00pm.
- Final exam: Friday, 06/21, 9:00am–1:00pm.
- Sample Exam: 2022 Midterm.
- Sample Exam: 2022 Final.
- Sample Exam: 2021 Midterm.
- Sample Exam: 2021 Final.

Homework 30%, midterm exam 30%, final exam 40%.

Good knowledge of the following subjects is required.

- Vector calculus: profeciency with gradients and chain rules.
- Linear algebra: profeciency with matrix-vector products.
- Basic probability: random variables, expectation, mean, variance, discrete and continuous random variables, Gaussian random variables.

No prior backround in machine learning or deep learning is required.

Although programming is an essential part of deep learning, I do not want to make substantial Python programming background a prerequisite for this course. Therefore, I will record and upload a Python tutorial for students to study before the start of the semester. I expect this tutorial to be about 5 hours long, and it will cover the necessary Python programming concepts such as variables, loops, functions, classes, objects, inheritance, exceptions, Python lists, Python tuples, Python decorators, Python iterators, the NumPy library, and the Matplotlib library. Of course, students already familiar with these concepts need not spend time with the tutorial.

- Python Tutorial 1, Jupyter notebook code.
- Python Tutorial 2, Jupyter notebook code.
- Python Tutorial 3, Jupyter notebook code.

Programming in deep learning tends to be relatively simple if one utilizes the frameworks such as PyTorch. These frameworks do utilize advanced programming concepts such as inheritance, decorators, and iterators, but if you are roughly familiar with these concepts, I do not expect the Python programming to be a major challenge in this course.

The course does not have a designated reference or a textbook. However, you may find the following references useful: Fei Fei Li's CS 231n at Stanford, Pieter Abbeel's Deep Unsupervised Learning at Berkeley, *Dive into Deep Learning* by Zhang, Lipton, Li, and Smola, and *Deep Learning* by Goodfellow, Bengio, and Courville.

Throughout this class, you will spend many hours computing gradients. While the chain rule is taught in any vector calculus curriculum, chain rule formulae in vector and matrix forms are not. Fundamentally, you should first carry out the gradient calculations elementwise and then collect the result into a vectorized form. Nevertheless, you may find the following references useful: Matrix calculus notes 1, matrix calculus notes 2, and matrix calculus notes 3 (Chapters 1–3).

- Python Tutorial 1, Jupyter notebook code.
- Python Tutorial 2, Jupyter notebook code.
- Python Tutorial 3, Jupyter notebook code.
- Chapter 0: Prologue
- Chapter 1: Optimization and Stochastic Gradient Descent, Jupyter notebook code.
- Chapter 2: Shallow Neural Networks, Jupyter notebook code.
- Chapter 3: Convolutional Neural Networks, Jupyter notebook code.
- Chapter 4: CNNs for Non-Classification Supervised Learning, Jupyter notebook code.
- Chapter 5: Unsupervised Learning, Jupyter notebook code.
- Chapter A: Basics of Monte Carlo, Jupyter notebook code.

- Homework 1, wideMinima.py, conv1D.py.
- Homework 2, twolayerSGD.py.
- Homework 3, threelayerSGD.py.
- Homework 4, lenet_original.py.
- Homework 5, mlp_backprop.py, label_memorization.py.
- Homework 6, ddescent.py.
- Homework 7, counting_params.py, bn_remove.py, smallNetSaved, LNiN.py.
- Homework 8.
- Homework 9, anomaly_detection.py, normalizingFlow1d.py.
- Homework 10, flow_inpainting.py, nice.pt.
- Homework 11, flow_anomaly.py, nice.pt.
- Homework 12, minimax_logistic.py, swiss_roll.py.