Decoding Logs for Automatic Metric Identification

Abstract

Automated Log Analysis tasks such as root cause analysis, fault prediction, etc. play a pivotal role in maintaining the overall application health. These tasks employ log parsers that extract the dynamic (variable) and constant (template) parts of a log line. However, our observations indicate that not all of them carry equal significance. Hence, there is a need to prioritize which templates/variables to use for log analysis. In this paper, we introduce LogMId, a Logs-based Metric Identification method, which is designed to extract critical IT metrics from logs. Through LogMId, we aim to enhance monitoring, observability tools and in turn Site Reliability Engineers to mine better insights from log data. We showcase the effectiveness of LogMId on a popular log analysis task of anomaly detection. Our experiments indicate that integrating previously used benchmark tools with LogMId features leads to improved results. Additionally, LogMId demonstrates effectiveness even with a smaller amount of training data, emphasising its utility.