LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System

Interactive user interfaces have increasingly explored AI's role in enhancing communication efficiency and productivity in collaborative tasks. AI tools such as chatbots and smart replies aim to enhance conversation quality and improve team performance. Early AI assistants, were limited by predefined knowledge bases and decision trees. However, the advent of Large Language Models (LLMs) such as ChatGPT has revolutionized AI assistants, employing advanced deep learning architecture to generate context-aware, coherent, and personalized responses. Consequently, ChatGPT-based AI assistants provide a more natural and efficient user experience across various tasks and domains. In this paper, we study how LLM models such as ChatGPT can be used to improve work efficiency in collaborative workplaces. Specifically, we present an LLM-based Smart Reply (LSR) system utilizing the ChatGPT to generate personalized responses in daily collaborative scenarios, while adapting to context and communication style based on prior responses. Our two-step process involves generating a preliminary response type (e.g., Agree, Disagree) to provide a generalized direction for message generation, thus reducing response drafting time. We conducted an experiment in which participants completed simulated work tasks, involving a Dual N-back test and subtask scheduling through Google Calendar while interacting with researchers posing as co-workers. Our findings indicate that the proposed LSR reduces overall workload, as measured by the NASA TLX, and improves work performance and productivity in the N-back task. We also provide qualitative feedback on participants' experiences as well as design recommendations so as to provide future directions for the design of these technologies.