Using human and robot synthetic data for training smart hand tools

The future of work does not require a choice between human and robot. Aside from explicit human-robot collaboration, robotics can play an increasingly important role in helping train workers as well as the tools they may use, especially in complex tasks that may be difficult to automate or effectively roboticize. This paper introduces a form of smart tool for use by human workers and shows how training the tool for task recognition, one of the key requirements, can be accomplished. Machine learning (ML) with purely human-based data can be extremely laborious and time-consuming. First, we show how data synthetically-generated by a robot can be leveraged in the ML training process. Later, we demonstrate how fine-tuning ML models for individual physical tasks and workers can significantly scale up the benefits of using ML to provide this feedback. Experimental results show the effectiveness and scalability of our approach, as we test data size versus accuracy. Smart hand tools of the type introduced here can provide insights and real-time analytics on efficient and safe tool usage and operation, thereby enhancing human participation and skill in a wide range of work environments. Using robotic platforms to help train smart tools will be essential, particularly given the diverse types of applications for which smart hand tools are envisioned for human use.