Semantic UI Understanding in the Era of AI

Abstract

In recent years, there has been a growing interest in the development of AI agents that can automate the interaction with user interfaces (UIs) and carry out actions on behalf of user. The ability to understand the semantics of a user interface plays a crucial role in effectively performing such automation-related activities. Semantic UI understanding is the process of extracting the meaning of a UI, including the purpose of each element and the relationships between elements. Attaining high-quality results in this domain remains a challenging task. Nevertheless, this pursuit is critical in order to comply with necessary business quality standards and in order to establish trust in AI agents operating autonomously on the UI. This paper discusses the key challenges in developing Semantic UI understanding, as well as the notable uses for it. We review the key methods in this field, while presenting their pros and cons. Finally, we propose a combined approach for solving this challenge.