Unit 2: Multivariate Problems, Gradient-Based Methods
Feb 17:
Bracketing
Contents:
- Unimodality
- Finding an Initial Bracket
- Fibonacci Search
- Golden Section Search
- Quadratic Fit Search
- Shubert-Piyavskii Method
- Bisection Method
Resources
- Slides
Local Descent
Contents:
- Descent Direction Iteration
- Line Search
- Approximate Line Search
- Trust Region Methods
- Termination Conditions
Resources:
- Chapter 4 and Appendix of [KW]
Feb 18: Exercise Session 2
Feb 19: First-Order Methods
Contents:
- Gradient Descent
- Conjugate Gradient
- Momentum
- Nesterov Momentum
- Adagrad
- RMSProp
- Adadelta
- Adam
- Hypergradient Descent
Resources:
- Chapter 5 of [KW]
Feb 25: Exercise Session 3
March 3: Second-Order Methods and Direct Methods
Second-Order Methods:
- Newton’s Method
- Secant Method
- Quasi-Newton Methods
Resources:
- Chapter 6 of [KW]
Direct Methods:
- Cyclic Coordinate Search
- Powell’s Method
- Hooke-Jeeves
- Generalized Pattern Search
- Nelder-Mead Simplex Method
- (Divided Rectangles)
Resources:
- Chapter 7 of [KW]