When designing Machine Learning (ML) enabled solutions, designers often need to simulate ML behavior through the Wizard of Oz (WoZ) approach to test the user experience before the ML model is available. Although reproducing ML errors is essential for having a good representation, they are rarely considered. We introduce Wizard of Errors (WoE), a tool for conducting WoZ studies on ML-enabled solutions that allows simulating ML errors during user experience assessment. We explored how this system can be used to simulate the behavior of a computer vision model. We tested WoE with design students to determine the importance of considering ML errors in design, the relevance of using descriptive error types instead of confusion matrix, and the suitability of manual error control in WoZ studies. Our work identifies several challenges, which prevent realistic error representation by designers in such studies. We discuss the implications of these findings for design.
翻译:在设计机器学习(ML)启用的解决方案时,设计师往往需要通过Oz向导(WoZ)方法模拟ML行为,以测试用户在ML模型出现之前的经验。虽然复制 ML错误对于具有良好的代表性至关重要,但却很少被考虑。我们引入了错误精华(WoE)工具,用于WoZ对ML驱动的解决方案进行研究,从而在用户经验评估中模拟 ML错误。我们探索了如何利用这个系统模拟计算机视觉模型的行为。我们用设计学生测试WoE,以确定在设计中考虑 ML错误的重要性、使用描述性错误类型而不是混淆矩阵的相关性以及WoZ研究中人工控制错误的适宜性。我们的工作找出了几项挑战,防止设计师在这类研究中真实地代表错误。我们讨论了这些发现对设计的影响。