RandHyp: Preventing attacks via Xen hypercall interface

Abstract

Virtualization plays a key role in constructing cloud environments and providing services. Although the main jobs of the hypervisors are to guarantee proper isolation between domains and provide them services, the hypercall interface provided by the hypervisor for cross-layer interactions with domains gives attackers the possibility to breach the isolation or cause denial of service from inside the domains. In this paper, we propose a transparent approach that uses randomization technique to protect the hypercall interface. In our approach, even facing a total compromise of a domain, the security of the virtualization platforms can be guaranteed. We have built a prototype called RandHyp based on Xen. Our experimental results show that RandHyp can effectively prevent attacks via Xen hypercall interface with a small overhead. © 2012 IFIP International Federation for Information Processing.