Towards a No-code Resilient Intelligent Automation System

Abstract

The advancement of Large Language Models (LLMs) has opened new avenues for addressing the complexities of UI automation, presenting solutions that cater not only to developers but also to non-technical users. This paper discusses IDA, an innovative no-code UI automation system that leverages LLMs to address two major challenges in UI automation: resilient element matching and the management of conditional UI logic complexities. IDA's effectiveness is evidenced by its success in the IPA challenge, demonstrating remarkable resilience by adapting to dynamic UI changes and making UI automation accessible to non-developers. Through an examination of IDA's components and its application in automating tasks with varying degrees of complexity, we illustrate the potential of LLMs to simplify UI automation processes and enhance the accessibility of no-code tools for non-technical users. Our findings underscore the significance of integrating AI to overcome technical barriers and extend the benefits of UI automation across a wider range of enterprise applications.