Automated Detection and Containment of Stealth Attacks on the Operating System Kernel

Arati Baliga
Rutgers University


The operating system kernel serves as the root of trust for all applications running on the computer system, which rely on it for services. An attack on the operating system kernel that alters its state is critical because it puts all applications at risk. A compromised system can be exploited by remote attackers stealthily, in several ways, such as exfiltration of sensitive information, wasteful usage of the system's resources, adversely affecting system performance or involving it in fraudulent or malicious activities without the user's knowledge or permission. The lack of appropriate detection tools allows such systems to stealthily lie within the attackers realm for indefinite periods of time. Stealth attacks on the kernel are carried out by malware commonly known as rootkits. The goal of the rootkit is to conceal the presence of the attacker on the victim system, so that the attacker can continue to control the system undetected. Early attacks achieved this goal by installing trojaned system binaries and shared libraries on the victim system. As detection mechanisms matured to check the integrity of system binaries on disk, attacks evolved to modify code and data structures in the kernel instead. Conventionally, kernel rootkits modified the kernel to achieve stealth, while most functionality was provided by accompanying user space programs. The newer kernel rootkits achieve the malice and stealth solely by modifying kernel data. My work explores the threat posed by both types of kernel rootkits and proposes novel automated techniques for their detection and containment. In this talk, I will discuss the three main contributions of my research. The first contribution that I will talk about is an automated containment technique built using the virtualization architecture. This technique counters the ongoing damage done to the system by the conventional kernel rootkits. It is well suited for attacks that employ kernel or user mode stealth but provide most of the malicious functionality as user space programs. It also effectively contains other malware, such as viruses and worms that are bundled with rootkits. My second contribution is to identify a new class of stealth attacks on the kernel, which do not exhibit explicit hiding behavior but are stealthy by design. They achieve their malicious objectives by solely modifying data within the kernel. These attacks demonstrate that the threat posed to kernel data is systemic requiring comprehensive protection. It also highlights the shortcomings of available detection techniques; since these attacks do not explicitly hide objects, they cannot be detected by tools that use the hiding behavior as a symptom for detection. The final contribution is a novel automated technique that can be used for detection of such stealth data-centric attacks. The key idea behind this technique is to automatically identify and extract invariants exhibited by kernel data structures during its normal operation. The hypothesis being, rootkits that manipulate kernel data violate some of these invariants and therefore, can be detected. Our system automatically infers invariants on kernel data during a training phase. These invariants are used as specifications of data structure integrity and are enforced during runtime. Our technique successfully detects all rootkits that are publicly available. It also detects more recent stealth attacks developed by us and proposed by other recent research literature.