Unit 7: Special Topics in Gradient Descent

Special Topics in Gradient Descent

April 27: Online Gradient Based Optimization

• Recap of Basic Constrained Convex Optimization
• Introduction to Online Learning Framework
• Regret Minimization
• Online Gradient Descent (OGD)
• Online Support Vector Machines
• Stochastic Gradient Descent via OGD

Resources:

• Slides
• Elad Hazan. Introduction to Online Convex Optimization (2019)

May 4: Distributed Gradient Descent

• Recap of Previous Lecture
• Motivation for Large-scale Distributed Learning
• Parallel SGD
• Synchronous SGD

Resources:

• Slides
• Léon Bottou, Frank E. Curtis, Jorge Nocedal. Optimization Methods for Large-Scale Machine Learning (2018)
• Dong Yin, Ashiwn Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, Peter L. Bartlett. Gradient Diversity: A Key Ingredient for Scalable Distributed Learning (2018)
• Martin A. Zinkevich, Markus Weimer, Alex Smola, Lihong Li. Parallelized Stochastic Gradient Descent (2010)

May 11: Communication-Efficient Gradient Descent

Resources:

• Recap of Previous Lecture
• Towards Communication-Efficient Gradient Descent
• Local SGD
• FedProx for Heterogeneous Clients

Resources:

• Slides
• H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas. communication-Efficient Learning of Deep Networks from Decentralized Data (2016)
• Sebastian U. Stich. Local SGD Converges Fast and Communicates Little (2019)
• Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, Virginia Smith. Federated Optimization in Heterogeneous Networks (2023)

May 31: Obligatory Assignment 3

• Assignment

• SDU rules on AI assistants https://mitsdu.dk/en/aiatsdu apply to this exam. In particular:

  • you have to add the Generative AI declaration form from the Faculty or an Appendix where you declare to which extent you have used AI tools. The Form or the Appendix must be present also if you did not use AI and hence declare so.

  • if you do not declare but the teachers have suspects that you used AI tools, you will be reported for exam cheating

  • if you use AI, the teachers will be allowed to take this in consideration and to lower consequently the final grade if they suspect that the learning goals have not been achieved.

  In declaring the use you can share the conversation you had with the tools and provide the link or copying it in the Appendix or in the declaration form. The Appendix and the declaration form are not subject to page limits.