Abstract
Human-in-the-loop data labeling is generally considered a tedious, error-prone and expensive activity. Automation of the labeling task is desirable, but current approaches can conflict with principles of trust and human agency. We are developing a data labeling experience where the human labeler transparently interacts with an AI assistant to reach automation readiness, at which point the remainder of the labeling task can be delegated to the AI assistant. Our approach combines semi-supervised learning, active learning, and human-AI decision tracking to reduce labeling effort and support reliable automation.