VASP: Virtualization assisted security monitor for cross-platform protection

Abstract

Numerous operating systems have been designed to manage and control system resources with large and complicated features, so they need high security protection. However, previous security applications can not provide adequate protection due to the untrusted execution environment. Furthermore, these security strategies cannot support a universal cross-platform system security requirements. This paper presents VASP, a hypervisor based monitor which allows a trusted execution environment to monitor various malicious behaviors in the operating system. This is achieved by taking advantage of ×86 hardware virtualization and self-transparency technology, and providing a unified security protection to unmodified operating systems such as Linux and Windows. Our design is targeted at establishing a security monitor which resides completely outside of the target OS environment with a negligible overhead. According to the security analysis and performance experiment result, our approach can effectively protect applications and the kernel at a modest overhead of only 0.9% average in Windows XP and 2.6% average in Linux. © 2011 ACM.