Using Hardware Isolated Execution Environments for Securing Systems

Fengwei Zhang
George Mason University


With the rapid proliferation of malware attacks on the Internet, malware detection and analysis plays a critical role in crafting effective defenses. Advanced malware detection and analysis relies on virtualization and emulation technologies to introspect the malware in an isolated environment and analyze malicious activities by instrumenting code execution. Virtual Machine Introspection (VMI) systems have been widely adopted for malware detection and analysis. VMI systems use hypervisor technology to create an isolated execution environment for system introspection and to expose malicious activity. However, recent malware can detect the presence of virtualization or corrupt the hypervisor state and thus avoid detection and debugging. In my talk, I will first introduce tools that use hardware isolated execution environments for attack detection, malware debugging, and executing sensitive operations. These tools leverage System Management Mode (SMM), a special CPU mode existing in the x86 architecture, as a trusted execution environment. Then, I will focus on two uses of SMM for building defensive tools. One is attack detection that introspects all layers of system software; the other is transparent malware debugging, which achieves a higher level of transparency than state-of-the-art systems. Lastly, I will talk about other hardware isolated execution environments and my future research directions.