Automated Generation of Competency Questions Using Large Language Models and Knowledge Graphs

Abstract

This research presents a novel approach to automated competency question generation by integrating Large Language Models (LLMs) with Knowledge Graphs (KGs), particularly within the context of sustainability assessment standards like BREEAM. The study develops a comprehensive methodology combining natural language processing and knowledge representation to address the challenges of manual question generation in competency-based assessments. The methodology begins with text extraction from BREEAM standards, followed by preprocessing, transformation into graph documents, and the construction of a structured KG. Advanced LLMs, including GPT-4o and Mistral, are employed to generate competency questions based on entity-specific and community-focused retrieval methods. The system is rigorously evaluated using quantitative metrics such as cosine similarity scores and qualitative assessments using the "LLM-as-a-Judge"method. Results demonstrate that GPT-4 and Mistral models generate highly relevant, clear, and complex questions, highlighting the potential for scalable, domainspecific competency assessments. This research opens avenues for improving AI-driven educational technologies and personalised learning through automated, adaptive assessment tools.