ITI42726 Advanced Optimization for Machine Learning (Spring 2027)

• at least 80 ECTS credits in computer science, of which at least 20 ECTS credits in programming.
• strong programming skills (Python), linear algebra, multi-variable calculus, probability theory, statistics, and basic machine learning knowledge.

Second semester (spring) in the full-time programme.

Fourth semester (spring) in the part-time programme.

The topics covered in this course are:

• Mathematical foundations of optimization: convex analysis, duality theory, optimality conditions
• Numerical optimization for machine learning: gradient descent variants, stochastic optimization, variance reduction
• Constrained optimization methods: projected gradient, augmented Lagrangian, ADMM
• Large-scale optimization and distributed optimization for modern ML systems
• Regularization and structured sparsity in learning problems
• Second-order methods: quasi-Newton, Hessian approximations, natural gradient methods
• Non-convex optimization in deep learning: landscape analysis, saddle points, implicit bias
• Proximal algorithms for non-smooth optimization: proximal gradient, forward-backward splitting, ISTA/FISTA
• Primal-dual methods and operator splitting techniques
• Subgradient methods for non-smooth and large-scale learning problems
• Modern applications: sparse learning, compressed sensing, deep networks, inverse problems, generative models

Special Focus (at the end of course):

• Advanced proximal algorithms in ML

Sub-gradient methods for non-differentiable objectives

Knowledge

The student:

• understands convex and non-convex optimization theory relevant to modern machine learning
• understands proximal operators and splitting methods for non-smooth optimization
• understands theoretical and practical aspects of sub-gradient methods
• understands large-scale optimization challenges in modern ML systems

Skills

The student can:

• formulate ML problems as optimization problems with constraints and regularization
• implement proximal optimization algorithms for large-scale learning tasks
• design optimization pipelines for non-smooth and composite objectives
• analyze convergence and stability of optimization algorithms

General Competence

The student:

• can design scalable optimization solutions for modern machine learning systems
• can evaluate trade-offs between accuracy, stability, and computational cost
• can implement research-level optimization algorithms used in modern ML literature

The student must deliver:

• Up to 3 mandatory optimization programming assignments
• One research-oriented mini-project with oral presentation

Coursework requirements must be accepted to qualify for the exam.

Individual portfolio and oral exam

The exam is divided into two parts:

• Portfolio: Project report and code implementation (counts for 40% of final grade)
• Oral exam: Theory and problem solving, questions related to the project (counts for 60% of final grade)

Both parts must be passed in order to pass the exam as a whole. The student will get an individual joint grade for the entire course. Grades: A - F.

Resit / rescheduled exam is carried out at the same time as the next ordinary examination. The department management can nevertheless decide to arrange a resit exam in the following semester.

Upon re-examination, each part of the examination can be retaken. Upon re-examination, a new project must be carried out.

This course is evaluated by a:

• Final course evaluation

The responsible for the course compiles a report based on the feedback from the students and his/her own experience with the course. The report is discussed by the study quality committee at the Department of Computer Science and Communication.