Graduation Semester and Year
2017
Language
English
Document Type
Dissertation
Degree Name
Doctor of Philosophy in Computer Science
Department
Computer Science and Engineering
First Advisor
David C Kung
Second Advisor
Jiang Ming
Abstract
The number of smartphones has increased greatly during the last few years. Among the popular mobile operating systems (such as iOS and Android) installed on these devices, Android captures most of the mobile market share. This also puts Android OS in a spotlight to attract malware attacks. A recent study shows that for the last two years, more than ~99% of the mobile malware targeted Android OS. Examples of such attacks are leakage of privacy-sensitive data available on the devices (such as phone number, contacts, photos, and SMS and call logs), recording audio and video files, silently making phone calls in the background, and encrypting device files. Many of them are profit-oriented (i.e., sending SMS to premium rate numbers can cause unexpected higher monthly bills for the users). Driven by the rich profit, the malware attacks are also becoming stealthier over time to maximize the long-term payoffs. A stealthy attack typically takes extra precautionary measures to stay undetected for a longer period of time. There are two types of stealthy attacks based on how stealth is achieved: (1) Type 1 attacks use hidden or uncommon program flows of Android apps to exhibit their malicious behaviors. (2) Type 2 attacks launch additional actions to hide their intended malicious behaviors. For example, the infamous Android.HeHe malware carefully takes three actions to silently block incoming phone calls: that is, it mutes ringer just before the incoming call is notified on the device, blocks the phone call, and unmutes the ringer after call blocking. To combat such attacks, researchers have developed numerous techniques based on static analysis. Static analysis detects malicious behaviors by analyzing the app code without execution. It represents program logic in some model (such as a control flow graph) and analyzes the model to detect possible attacks. The effectiveness of a static analysis tool relies on three key elements: (i) the app model representing app behaviors, (ii) the attack model representing attack behaviors, and (iii) the attack detection algorithm which analyzes the app model. If any of the models and/or the algorithm is inadequate, then sophisticated attacks such as stealthy attacks discussed above cannot be detected. To this end, this dissertation develops methods to accurately model app and attack behaviors, and, based on those models, improves analysis algorithms to effectively detect malicious behaviors in Android apps. More specifically, the dissertation proposes two static analysis frameworks called Dexteroid and StateDroid to achieve these goals. The former identifies many hidden program flows and based on them, analyzes Android apps to detect malicious behaviors including type 1 stealthy attacks. The latter focuses on modeling of malware attacks and developing analysis techniques to detect the attacks such as type 2 stealthy attacks. Dexteroid identifies hidden program flows in Android apps by performing reverse-engineering on life cycle models of Android components. The components are building blocks of Android apps and life cycle models describe components’ behaviors. Dexteroid represents the reverse-engineered life cycle models as state machines and drives from them all program flows which consist of component callback methods. The callback methods are analyzed to detect malicious behaviors, including those that are launched through hidden program flows. A prototype of Dexteroid is implemented as a static taint analysis tool. A novel implementation of taint analysis which maintains up to date values and states of program variables through symbol tables allows Dexteroid to detect many attacks with high accuracy. Current implementation detects two attacks: (1) leakage of private information, and (2) sending SMS to premium-rate numbers. Evaluation results on a Google Play and Genome Malware apps show that the proposed framework is effective and efficient in terms of precision, recall, and execution time. StateDroid focuses on detecting type 2 stealthy attacks which typically execute multiple actions to launch and hide their malicious behaviors. To detect them, the framework presents novel techniques, based on state machines, to construct accurate attack behaviors. An attack, represented by an attack state machine (ASM), has states and transitions; state represents status of the attack, and transition represents the executed action. The framework first detects actions of an attack, and then uses them with an ASM to detect the attack. Given an Android app as an input, StateDroid performs fine-grained static analysis and reports various detected stealthy behaviors (in one pass), including but not limited to sending SMS message, blocking phone call, removing app icon from launcher menu, recording an audio or video file, and setting device ringer to silent mode. A prototype of StateDroid framework is implemented, and evaluated extensively with ground truth dataset, 1505 Google Play apps, and 1369 malicious apps including 94 notorious ransomware apps. The experimental results demonstrate the efficacy and generality of StateDroid. The success of StateDroid will enable broader adoptions of formal methods in cyber defense.
Keywords
Privacy, Android malware, Stealthy behaviors, Android apps security analysis, Static analysis, Attack state machine, Life cycle models, Formal methods
Disciplines
Computer Sciences | Physical Sciences and Mathematics
License
This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License.
Recommended Citation
Junaid, Mohsin, "STATEFUL DETECTION OF STEALTHY BEHAVIORS IN ANDROID APPS" (2017). Computer Science and Engineering Dissertations. 343.
https://mavmatrix.uta.edu/cse_dissertations/343
Comments
Degree granted by The University of Texas at Arlington