Adapting LLMs for Structured Natural Language API Integration

Abstract

Integrating APIs is crucial for enterprise systems, enabling seamless application interaction within workflows. However, the vast and diverse API landscape makes combining calls based on user intent a significant challenge. Existing methods rely on Named Entity Recognition (NER) and knowledge graphs, but struggle with control flow structures like conditionals and loops. We propose a novel framework that leverages the success of Large Language Models (LLMs) in code generation for natural language API integration. Our approach involves fine-tuning an LLM on automatically generated API flows derived from services' OpenAPI specifications. This aims to surpass NER-based methods and compare the effectiveness of different tuning strategies. Specifically, we investigate the impact of enforcing syntax through constrained generation or retrieval-augmented generation. To facilitate systematic comparison, we introduce targeted test suites that assess the generalization capabilities and ability of these approaches to retain structured knowledge. We expect to observe that fine-tuned LLMs can: (a) learn structural constraints implicitly during training, and (b) achieve significant improvements in both in-distribution and out-of-distribution performance.