Searching and Classifying Mobile Application Screenshots

Author

Adis Kovacevic

Graduation Semester and Year

2016

Language

English

Document Type

Thesis

Degree Name

Master of Science in Computer Science

Department

Computer Science and Engineering

First Advisor

Christoph Csallner

Second Advisor

Bahram Khalili

Abstract

Searching for a particular application layout image is a challenging task. No search provider gives an adequate method to filter the query results to the look of a mobile application. Searching for a particular style of application requires lots of manual time sifting through the results returned. Search engines such as Google are too broad and return too many unrelated results without providing sufficient filters on things like category or layout type. This paper proposes a technique that would allow the searching and classifying of mobile application screenshots based on the layout of the content, the category of the application, and the text in the image. It was originally conceived to support REMAUI (Reverse Engineering Mobile Application User Interfaces), an active research project headed up by Dr. Csallner. REMAUI has the ability to automatically reverse engineer the User Interface layer of an application by being given input Images. The long term goal of this work is to create a full search framework for any UI image. In this paper, we introduced the first steps to this framework by focusing on mobile UI screenshots. We discuss 3 techniques to classify the layout of the image, Block Analysis, Interval Encoding, and Bag of Visual Words. We continue on to discuss a method to classify the category of the application based on the text in the image. Finally, we put all the information together in a single REST API. The API can search input images by the image content and filter by type and layout. The results are ranked by Solr for relevance and returned as json by the API.

Keywords

Mobile, Screenshots, Classifying

Disciplines

Computer Sciences | Physical Sciences and Mathematics

License

This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License.

Comments

Degree granted by The University of Texas at Arlington

Recommended Citation

Kovacevic, Adis, "Searching and Classifying Mobile Application Screenshots" (2016). Computer Science and Engineering Theses. 421.
https://mavmatrix.uta.edu/cse_theses/421