LLM-based Smart Reply (LSR): Enhancing Collaborative Performance with ChatGPT-mediated Smart Reply System

CSCW studies have increasingly explored AI's role in enhancing communication efficiency and productivity in collaborative tasks. AI tools such as chatbots, smart replies, and language models aim to optimize conversation management and improve team performance. Early AI assistants, such as Gmail smart reply, were limited by predefined knowledge bases and decision trees. However, the advent of large language models (LLMs) such as ChatGPT has revolutionized AI assistants, employing advanced deep learning architecture to generate context-aware, coherent, and personalized responses. Consequently, ChatGPT-based AI assistants provide a more natural and efficient user experience across various tasks and domains. In this paper, we formalize the concept of AI Collaborative Tools (ACT) as AI technologies in human collaborative work and discuss how the emergence of ChatGPT has transformed the AI landscape and increased focus on ACT for improving team performance. Meanwhile, we present an LLM-based Smart Reply (LSR) system utilizing the ChatGPT API to generate personalized responses in daily collaborative scenarios, considering context, tone, and communication style. Our two-step process involves generating a preliminary response type (e.g., Agree, Disagree) to provide a generalized direction for message generation, thus reducing response drafting time. We conducted an experiment in which participants completed simulated work tasks, involving Google Calendar manipulation and a double-back N-back test, while interacting with researchers posing as teammates requesting scheduling changes. Our findings indicate that the AI teammate increases perceived performance and reduces mental demand, as measured by the NASA TLX, and improves performance in the N-back task. We also provide qualitative feedback on participants' experiences working with the AI teammate.