Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models

/in /by

Publication Date: 11/12/2024

Event: The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024)

Reference: pp. 15299-15312, 2024

Authors: Zi’ou Zheng, Queen’s University; Christopher Malon, NEC Laboratories America, Inc.; Martin Renqiang Min, NEC Laboratories America, Inc.; Xiaodan Zhu, Queen’s University

Abstract: When performing complex multi-step reasoning tasks, the ability of Large Language Models (LLMs) to derive structured intermediate proof steps is important for ensuring that the models truly perform the desired reasoning and for improving models explainability. This paper is centered around a focused study: whether the current state-of-the-art generalist LLMs can leverage the structures in a few examples to better construct the proof structures with in context learning. Our study specifically focuses on structure-aware demonstration and structure aware pruning. We demonstrate that they both help improve performance. A detailed analysis is provided to help understand the results.

Publication Link: https://aclanthology.org/2024.emnlp-main.854.pdf

Additional Publication Link: https://arxiv.org/abs/2410.08436