Zeroth-Order Machine Learning: Fundamental Principles and Emerging Applications in Foundation Models

Abstract

Part 1: Introduction of ZO-ML Preliminary Concepts and Mathematical Foundations Basic mathematical tools and formulations Why ZO over FO: Limitations of Traditional Gradient-Based Optimization Emerging challenges and drawbacks of relying solely on FO gradientbased methods Survey of Practical Applications and Use Cases Overview of applications that benefit from ZOML Part 2: Foundations of ZO-ML Algorithmic Landscape of ZO-ML A rundown of primary algorithms and methods in ZOML Convergence and Query Complexity Understanding the provable properties of ZOML Scaling ZO-ML: Practical Techniques and Implementations Tips and tricks for ZOML algorithms at scale Extending ZO-ML across Learning Paradigms How does ZOML adapt to various ML paradigms? Break Part 3: Applications of ZO-ML Prompt Learning in FMs Fine-tuning and Personalization in FMs via ZO-ML ZO-ML in the Context of AI Robustness, Efficiency, and Automation Part 4: Demo Expo Introducing the ZO-ML Toolbox A guided tour of our specialized toolbox for ZOML Benchmarking with ZO algorithms An introduction to ZO performance metrics and benchmark applications Practical Demos: Utilizing ZOT for Parameter-Efficient Fine-Tuning (PEFT), and Adversarial Defense Live demonstrations showcasing the utility of ZOML Part 5: Conclusion and Q&A Wrap-Up: Key Takeaways from the Tutorial Future Horizons: SP and ML Opportunities and Challenges Resources for Deeper Exploration A curated list of essential ZOML resources