RMSProp and ADAM
| Date: 03-27-2024 |
| Lecturer: Charles Guille-Escuret |
Download notes
- Semesters
-
Fall 2024 (current)
- 1. Introduction to Convex Analysis
- 2. Introduction to Convex Analysis pt.2
- 3. Subgradient Method and Dual Averaging
- 4. Nesterov's Acceleration
- 5. Adaptivity: Line-search, backtracking, Polyak Step Size
- 6. Projected and Proximal Gradient Method
- 7. Bregman divergence and Mirror Descent
- 8. Introduction to Higher-Order Methods
- 9. Constrained Optimization Lagrangian
- 10. Surrogate Optimization
-
Winter 2024
- 1. Basic concepts, convexity, gradient descent
- 2. Subgradient Descent, Dual averaging
- 3. Convergence proof: gradient, subgradient descent
- 4. Acceleration: Nesterov, heavy ball
- 5. Stochastic gradient descent
- 6. Adaptive methods: line search, Polyak step
- 7. RMSProp and ADAM
- 8. Automated proof techniques
- 9. New adaptive techniques for deep learning