On Diverse Preferences for Large Language Model Alignment

The alignment of large language models (LLMs) with human values is crucial for the development of artificial general intelligence (AGI). One promising approach to achieve this alignment is reinforcement learning from human feedback, which employs a reward model (RM) learned from human preference datasets to guide LLMs in generating text that aligns with human preferences. Through intensive experiments and analysis of reward distribution, this paper finds that preference datasets are diverse from each other, even though they are all proposed to align human preference. Hence, mixing diverse human preference datasets to increase data size for enhancing reward modeling could fail. To address the issue and capture the shared human values from diverse preferences, a new training policy called MORE is introduced, which minimizes preference bias by adaptively adjusting the preference objective across diverse preferences. Experiments with the Pythia-1.4B model and five mixed preference datasets show that MORE achieves superior reward accuracy and lower calibration error, highlighting its ability to leverage diverse human preference data.