The layout of a mobile screen is a critical data source for UI design research and semantic understanding of the screen. However, UI layouts in existing datasets are often noisy, have mismatches with their visual representation, or consists of generic or app-specific types that are difficult to analyze and model. In this paper, we propose the CLAY pipeline that uses a deep learning approach for denoising UI layouts, allowing us to automatically improve existing mobile UI layout datasets at scale. Our pipeline takes both the screenshot and the raw UI layout, and annotates the raw layout by removing incorrect nodes and assigning a semantically meaningful type to each node. To experiment with our data-cleaning pipeline, we create the CLAY dataset of 59,555 human-annotated screen layouts, based on screenshots and raw layouts from Rico, a public mobile UI corpus. Our deep models achieve high accuracy with F1 scores of 82.7% for detecting layout objects that do not have a valid visual representation and 85.9% for recognizing object types, which significantly outperforms a heuristic baseline. Our work lays a foundation for creating large-scale high quality UI layout datasets for data-driven mobile UI research and reduces the need of manual labeling efforts that are prohibitively expensive.
翻译:移动屏幕的布局是界面设计研究和屏幕语义理解的关键数据源。 但是,现有数据集的界面布局往往噪音,与视觉显示不匹配,或由难以分析和建模的通用或特定软件类型组成。 在本文中,我们建议CLAY管道使用深层学习方法来拆分界面布局,允许我们自动改进现有的移动界面布局数据集。 我们的管道使用屏幕截图和原始界面布局,并通过删除不正确的节点和为每个节点指定一个具有语义意义的类型来说明原始布局。 为了试验我们的数据清理管道,我们创建了59,555个基于截图和来自波多黎各的原始布局的带有人文说明的屏幕布局。 我们的深层模型达到了82.7%的F1分,用于探测没有有效视觉显示的布局对象和85.9%的原始布局,这些布局明显超越了目标类型。 我们的CLAY数据集以高压压模的标度模型为基础,为高压模版图制作了高压的模板。