Exploring Large Language Models for Low-Resource IT Information Extraction

Abstract

Information Extraction (IE) in IT is an important foundational task that is needed for many AIOps applications. A major challenge of IE in IT is that we often do have not have sufficient labelled data for training machine learning algorithms since acquiring labels is labor-intensive and costly. In this paper, we propose to leverage Large Language Models (LLMs) to address this challenge of low resources and study two data augmentation strategies, i.e., using LLMs to generate pseudo labels and generate synthetic data. We use multiple IE tasks and datasets, including a new Semantic Troubleshooting-Segment Extraction task and Named Entity Recognition, to evaluate the benefits of LLMs. Our experiment results suggest that data augmentation using LLMs, specifically, using SeqMix model that combines active labeling with synthetic data samples generated in the embedding-vector space, is a promising approach IT domain IE. Our study also shows that although data augmentation and direct labeling with the state-of-the-art, ChatGPT model achieves a high performance on general domain IE, there is a need to adapt it for IE from IT text data. Moreover, our initial exploration of two label weighting and selection strategies (confidence and consistency-based) suggests that they could be used to improve data augmentation with ChatGPT for IT domain IE. Finally, we also suggest directions for future research on the new STSE task, including developing better evaluation metrics.