The limits of open-ended generative models are unclear, yet increasingly important. What causes them to succeed and what causes them to fail? In this paper, we take a prompt-centric approach to analyzing and bounding the abilities of open-ended generative models. We present a generic methodology of analysis with two challenging prompt constraint types: structural and stylistic. These constraint types are categorized into a set of well-defined constraints that are analyzable by a single prompt. We then systematically create a diverse set of simple, natural, and useful prompts to robustly analyze each individual constraint. Using the GPT-3 text-davinci-002 model as a case study, we generate outputs from our collection of prompts and analyze the model's generative failures. We also show the generalizability of our proposed method on other large models like BLOOM and OPT. Our results and our in-context mitigation strategies reveal open challenges for future research. We have publicly released our code at https://github.com/SALT-NLP/Bound-Cap-LLM.
翻译:开放型基因模型的局限性并不明确,但却越来越重要。 是什么原因导致它们成功,又是什么原因导致它们失败? 在本文件中,我们采取以迅速为中心的方法来分析和约束开放型基因模型的能力。 我们提出了一种通用的分析方法,它具有两种具有挑战性的迅速制约类型:结构性和文体性。这些制约类型被归类为一套定义明确的制约因素,可以通过单一的及时方法加以分析。然后,我们系统地建立一套简单、自然和有用的提示,以有力地分析每一种制约因素。我们用GPT-3文本-davinici-002模型作为案例研究,从我们收集的提示中产生产出,并分析模型的基因失败。我们还展示了我们所提议的方法在BLOOM和ALM等其他大型模型上的可概括性。我们的结果和我们的文中减缓战略揭示了未来研究的公开挑战。我们已经在https://github.com/SALP/Bound-CAP-LM上公开公布了我们的代码。