Properties of federated averaging on highly distributed data

Abstract

Tactical edge environments are highly distributed with a large number of sensing, computational, and communication nodes spread across large geographical regions, governments, and situated in unique operational environments. In such settings, a large number of observations and actions may occur across a large number of nodes but each node may only have a small number of these data locally. Further, there may be technical as well as policy constraints in aggregating all observations to a single node. Learning from all of the data may uncover critical correlations and insights. However, without having access to all the data, this is not possible. Recently proposed federated averaging approaches allow for learning a single model from data spread across multiple nodes and achieve good results on image classification tasks. However, this still assumes a sizable amount of data on each node and a small number of nodes. This paper investigates the properties of federated averaging for neural networks relative to batch sizes and number of nodes. Experimental results on a human activity dataset finds that (1) accuracy indeed drops as the number of nodes increase but only slightly, however (2) accuracy is highly sensitive to the batch size only in the federated averaging case.