A Collaborative, Immersive Language Learning Environment Using Augmented Panoramic Imagery

Abstract

Augmenting immersive technologies with AI for foreign language learning is a relatively unexplored, multidisciplinary, and complex research paradigm. The Cognitive and Immersive Room at Rensselaer Polytechnic Institute facilitates cultural and foreign language learning beyond a traditional classroom setting. Exploration of virtual environments and real-world renderings occur at human-scale by groups of students and teachers simultaneously, without the need for head-mounted displays. Students travel to true-to-life Panoramic Scenes to learn authentic cultural knowledge and practice vocabulary within the same surroundings they would use such knowledge while traveling abroad. Multimodal input allows students to engage with the system through gesture, voice, and spatial positioning, creating a dynamic language learning experience. This work underwent initial assessment in a six-week summer course, AI-Assisted Immersive Chinese, and preliminary qualitative results find a majority of surveyed students indicate the system is useful, engaging, and fun.