Document Type
Article
Source Publication Title
Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
DOI
http://dx.doi.org/10.1145/3611643.3613100
Abstract
The lack of diverse UI element representations in publicly available datasets hinders the scalability of sketch-based interactive mobile search. This paper introduces D2S2, a novel approach that addresses this limitation via drag-and-drop mobile screen search, accommodating visual and text-based queries. D2S2 searches 58k Rico screens for relevant UI examples based on UI element attributes, including type, position, shape, and text. In an evaluation with 10 novice software developers D2S2 successfully retrieves target screens within the top-20 search results in 15/19 attempts within a minute. The tool offers interactive and iterative search, updating its search results each time the user modifies the search query. Interested users can freely access D2S2 (http://pixeltoapp.com/D2S2), build on D2S2 or replicate results via D2S2’s open-source implementation (https://github.com/toni-tang/D2S2), or watch D2S2’s video demonstration (https://youtu.be/fdoYiw8lAn0).
Publication Date
11-30-2023
Language
English
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Recommended Citation
Mohian, Soumik; Tang, Tony; Trinh, Tuan; Dang, Don; and Csallner, Christoph, "D2S2: Drag ’n’ Drop Mobile App Screen Search" (2023). Association of Computing Machinery Open Access Agreement Publications-Archive. 81.
https://mavmatrix.uta.edu/utalibraries_acmoapubs/81