Optimization for Deep Learning
Training and adapting neural networks places demands on optimizer memory, update geometry, and parameter efficiency. I study algorithms that address these constraints alongside a theoretical analysis of their convergence.
This includes compressed adaptive optimizers, low-rank adaptation, sparsified training, and adaptive scaling for Muon. The aim is to connect practical training methods with a clearer understanding of when and why they converge.
Research focus
- Reducing the memory cost of adaptive optimization.
- Analyzing low-rank adaptation and sparsified training.
- Adapting step scales in normalized optimizers.
Selected papers
MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
The Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024)
Reduces adaptive-optimizer memory overhead through compression and error feedback.
Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation
The 17th International OPT Workshop on Optimization for Machine Learning (NeurIPS Workshop 2025)
Develops a convergence framework for randomized low-rank adaptation.
MAST: Model-Agnostic Sparsified Training
The 13th International Conference on Learning Representations (ICLR 2025)
Analyzes training formulations that sparsify both models and gradients.
Paper · arXiv · NOPTA 2024 · ICLR 2025 · Cite
Distance-Aware Muon: Adaptive Step Scaling for Normalized Optimization
arXiv · 2026
Studies adaptive step scaling for Muon and related normalized optimizers.