In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world's languages, be equitable, i.e., not unduly biased towards any particular language, and be inclusive of all users, particularly in low-resource settings where compute constraints are common. In this paper, we propose an evaluation paradigm that assesses NLP technologies across all three dimensions. While diversity and inclusion have received attention in recent literature, equity is currently unexplored. We propose to address this gap using the Gini coefficient, a well-established metric used for estimating societal wealth inequality. Using our paradigm, we highlight the distressed state of current technologies for Indian (IN) languages (a linguistically large and diverse set, with a varied speaker population), across all three dimensions. To improve upon these metrics, we demonstrate the importance of region-specific choices in model building and dataset creation, and more importantly, propose a novel, generalisable approach to optimal resource allocation during fine-tuning. Finally, we discuss steps to mitigate these biases and encourage the community to employ multi-faceted evaluation when building linguistically diverse and equitable technologies.
翻译:为了使自然语言处理技术具有广泛的适用性、公平性和实用性,它需要为世界各种语言的发言人提供服务,不偏向任何特定的语言,并包容所有用户,特别是在计算能力有限的低资源环境中。在本文中,我们提出了一种评估范式,该范式评估了NLP技术在所有三个维度上的表现。虽然多样性和包容性在最近的文献中得到了重视,但公平性目前尚未得到探讨。我们建议使用基尼系数,这是一种广泛采用的用于估计社会财富不平等的度量标准,来解决这一问题。使用我们的评估范式,我们突显了当前印度语言技术在三个维度上的困境(印度语是一个语言数量庞大,种类丰富,讲话人口位于各个地区的语言族群)。为了改进这些指标,我们展示了在模型构建和数据集创建时进行区域特定选择的重要性,更重要的是,提出了一种新颖的、具有普遍适用性的优化资源分配方法。最后,我们讨论了减轻这些偏差的步骤,并鼓励社区在构建语言多样和平等的技术时采用多维度评估。