ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models

By combining voice and touch interactions, multimodal interfaces can surpass the efficiency of either modality alone. This paper targets complex interactions, where users can issue multimodal commands that translate into one of the possible exponential combinations of actions/function invocations. This paper presents ReactGenie, a programming framework where developers can code with simple object-oriented abstractions and labeled user-invocable primitives. ReactGenie translates multimodal user commands into ReactGenieDSL, a domain-specific language we created for this purpose, using a neural semantic parser based on large-language models. The ReactGenie runtime interprets the parsed ReactGenieDSL and composes primitives to implement complex user commands. As a result, ReactGenie provides an unprecedented level of richness in user interactions. Our evaluation showed that 12 developers can learn and build a ReactGenie application in under 2.5 hours on average. In addition, compared with a traditional GUI, end users can complete tasks faster and with less task load using ReactGenie apps.