A reactive language for analyzing cloud logs

Abstract

Log analysis is required in many domains, and especially in the emerging field of cloud computing. Cloud applications are often built by composing diverse services. When something goes wrong, finding the root cause of the problem can be difficult. Many services are only reachable through their Application Programming Interfaces (APIs) with no possibility for live introspection. In this context, logs become an essential tool for monitoring and debugging. Cloud services typically generate very large quantities of log messages, with formats that may not be well specified and may vary over time. In this paper, we present CloudLens, a language for the analysis of semi-structured textual data as found in logs, and specify its formal semantics. CloudLens is a reactive language and views logs as streams of objects. Our objective is to facilitate exploring the contents of logs interactively and to write reusable analyses succinctly, using familiar constructs. We implemented an interpreter for the Apache Zeppelin notebook to provide an interactive IDE. Our prototype implementation is open source and we report on a detailed case study using logs from the Apache OpenWhisk project.