Zeroth-Order Machine Learning: Fundamental Principles and Emerging Applications in Foundation Models

Abstract

The overarching goal of this tutorial is twofold: The first aim is to conduct a comprehensive assessment of the latest advancements in the gradient-free learning paradigm, also referred to as zeroth-order machine learning (ZO-ML). This involves an exploration of the theoretical and methodological foundations that support ZO-ML. The second goal is to illustrate the effective integration of ZO-ML techniques with emerging ML/AI applications. This step aims to bridge the theoretical and practical aspects of ZO-ML, demonstrating its potential to overcome design limitations in current foundation model (FM)-oriented applications.