AbstractsComputer Science

These days all kinds of malware are pervasive on the Internet. Compared to their ances- tors that were commonly used for vandalism or demonstration of skills, modern malware, such as Bots, are driven by the underground economics. Often consisting of hundreds to thousands of bots, botnets are one of the most serious threats on the Internet, responsible for various attacks, such as spamming and distributed denial of service (DDoS). As web browsers are the main interface for the majority of Internet users to surf the Internet today, many of such stealthy malware seek to invade via web browsers in the form of browser helper objects (BHO) and browser toolbars. To defend against Internet malware, existing schemes mainly rely on either signature- based or anomaly-based detection approaches. Signature-based detection is e®ective for known malware if the malware signature has been generated. However, the e®ectiveness of signature-based schemes is challenged by polymorphism, metamorphism, obfuscation, en- cryption, and other techniques. Moreover, signature-based schemes do not work for zero-day (or unknown) malware. On the other hand, anomaly-based detection schemes seek to detect behavior patterns that do not conform to the established normal patterns. Anomaly-based detection schemes do not require malware signatures. However, modern computer software and systems are often complicated, building and analyzing a comprehensive behavior model is time consuming and even impractical. To overcome these challenges, we propose a novel execution-based approach for stealthy malware detection. In order to facilitate such run-time detection, we aim to design and implement multi-level sandboxing techniques to create controlled running environments to execute testing programs so that their behaviors can be closely observed and analyzed. First, we leverage virtual machines for OS-level sandboxing to detect bots on individ- ual hosts. By cloning the host image to a virtual machine and screening user input on the virtual machine, the detection noise is signi¯cantly reduced. We ¯nd that a typical bot exhibits three invariant features along its onset: (1) the startup of a bot is automatic without requiring any user actions; (2) a bot must establish a command and control chan- nel with its botmaster; and (3) a bot will perform local or remote attacks sooner or later. xi These invariants indicate three indispensable phases (startup, preparation, and attack) for a bot attack. Thus, we propose BotTracer to detect these three phases with the assistance of OS-level sandboxing techniques. To validate BotTracer, we implement a prototype of BotTracer based on VMware. The results show that BotTracer can successfully detect all the bots in the experiments. However, BotTracer may slightly degrade the user per- formance. Furthermore, advanced malware could evade BotTracer by performing virtual machine ¯ngerprinting. Second, to overcome the limitations of OS-level sandboxes, we build Malyzer based on process-level…