List of Topics for Assignment 1

# Derivatives

- Derivatives in Multiple Dimensions
- Numerical Differentiation
- Automatic Differentiation

# Bracketing

Finding an Initial Bracket:
  - Fibonacci Search
  - Golden Section Search
  - Quadratic Fit Search
  - Bisection Method

# Descent Direction Iteration

Line Search:
  - Exact Line Search
  - Approximate Line Search:
    - Backtracking line search (Armijo line search)
    - Strong backtracking line search (bracketing + zoom)
    - Trust Region Methods
  - Termination Conditions

First-Order Methods:
  - Gradient Descent
  - Conjugate Gradient
  - Momentum:
    - Nesterov Momentum
    - Adagrad
    - RMSProp
    - Adadelta
    - Adam
    - Hypergradient Descent

Second-Order Methods:
  - Newton’s Method
  - Secant Method
  - Quasi-Newton Methods:
    - DFP
    - BFGS
    - L-BFGS

Direct Methods:
  - Cyclic Coordinate Search
  - Powell's Method
  - Hooke-Jeeves
  - Generalized Pattern Search
  - Nelder-Mead Simplex Method
  - (Divided Rectangles)