Unit 2: Unconstrained Optimization, Local Descent Methods

Contents:

  • Unimodality
  • Finding an Initial Bracket
  • Fibonacci Search
  • Golden Section Search
  • Quadratic Fit Search
  • Shubert-Piyavskii Method
  • Bisection Method

Resources

Feb 18: Line Search in Local Descent Methods

Contents:

  • Descent Direction Iteration
  • Line Search
  • Approximate Line Search
  • Trust Region Methods
  • Termination Conditions

Resources:

  • Slides

  • Chapter 4 and Appendix of [KW]

Feb 19: First-Order Methods

Contents:

  • Gradient Descent
  • Conjugate Gradient

Resources:

Feb 24: Exercise Session 2

Feb 25: Exercise Session 3

March 3: Second-Order Methods

First-Order Methods (Cntd):

  • Momentum
  • Nesterov Momentum
  • Adagrad
  • RMSProp
  • Adadelta
  • Adam
  • Hypergradient Descent

Resources:

  • Slides from class on February 19
  • Chapter 5 of [KW]

Second-Order Methods:

  • Newton’s Method
  • Secant Method
  • Quasi-Newton Methods

Resources:

March 4: Exercise Session 4

March 5: Direct Methods

Direct Methods:

  • Cyclic Coordinate Search
  • Powell’s Method
  • Hooke-Jeeves
  • Generalized Pattern Search
  • Nelder-Mead Simplex Method
  • (Divided Rectangles)

Resources: