Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty

/in /by

Publication Date: 5/11/2024

Event: ICLR 2024 Workshop on Reliable and Responsible Foundation Models

Reference: pp. 1-6, 2024

Authors: Chen Ling, Emory University; Xujiang Zhao, NEC Laboratories America, Inc.; Xuchao Zhang, Microsoft Research; Yanchi Liu, NEC Laboratories America, Inc.; Wei Cheng, NEC Laboratories America, Inc.; Haoyu Wang, NEC Laboratories America, Inc.; Zhengzhang Chen, NEC Laboratories America, Inc.; Mika Oishi, NEC Corporation; Takao Osaki, NEC Corporation; Haifeng Chen, NEC Laboratories America, Inc.; Katsushi Matsuda, NEC Corporation; Liang Zhao, Emory University

Abstract: Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text, typically in the form of (subject, relation, object) triples. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant context from relevant relations and generate structured output due to the restrictions on fine-tuning the model. Second, LLMs generate responses autoregressively based on probability, which makes the predicted relations lack confidence. In this paper, we assess the capabilities of LLMs in improving the OIE task. Particularly, we propose various in-context learning strategies to enhance LLM’s instruction-following ability and a demonstration uncertainty quantification module to enhance the confidence of the generated relations. Our experiments on three OIE benchmark datasets show that our approach holds its own against established supervised methods, both quantitatively and qualitatively.

Publication Link: https://openreview.net/forum?id=7TOPn1CPuP