Using GPT-4 Prompts to Determine Whether Articles Contain Functional Evidence Supporting or Refuting Variant Pathogenicity

Purpose: To assess Generative Pre-trained Transformer version 4's (GPT-4) ability to classify articles containing functional evidence relevant to assessments of variant pathogenicity. Results: GPT-4 settings and prompts were trained on a set of 45 articles and genetic variants. A final test set of 72 manually classified articles and genetic variants were then processed using two prompts. The prompts asked GPT-4 to supply all functional evidence present in an article for a variant or indicate that no functional evidence is present. For articles with having functional evidence, a second prompt asked GPT-4 to classify the evidence into pathogenic, benign, intermediate, and inconclusive categories. The first prompt identified articles with variant-level functional evidence with 87% sensitivity and 89% positive predictive value (PPV). Five of 26 articles with no functional data were indicated as having functional evidence by GPT-4. For variants with functional assays present as determined by both manual review and GPT-4, the sensitivity and PPV of GPT-4 prompt concordance was: Pathogenic (92% sensitive and 73% PPV), Intermediate or Inconclusive (67% sensitive and 93% PPV), Benign (100% sensitive and 73% PPV). Conclusion: The GPT-4 prompts detected the presence or absence of a functional assay with high sensitivity and PPV, and articles with unambiguous evidence supporting a benign or pathogenic classification with high sensitivity and reasonable PPV. Our prompts detected papers with intermediate or inconclusive evidence with lower sensitivity but high PPV. Our results support that GPT-4 may be useful in variant classification workflows by enabling prioritization of articles for review that are likely to have functional evidence supporting or refuting pathogenicity, but not that GPT-4 is capable of fully automating the genetics literature review component of variant classification.