Natural Language Inference (NLI) is a task in natural language processing that involves determining the logical relationship between two given text segments. Typically, these segments are a premise and a hypothesis. The goal is to classify the relationship as either “entailment,” “contradiction,” or “neutral.” NLI is often used to evaluate the understanding of language semantics and reasoning capabilities of language models. Models pretrained on large language datasets, such as BERT, have demonstrated strong performance on NLI tasks by capturing contextual information and relationships between sentences.

Posts

Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models

/in /by

When performing complex multi-step reasoning tasks, the ability of Large Language Models (LLMs) to derive structured intermediate proof steps is important for ensuring that themodels truly perform the desired reasoning and for improving models explainability. This paper is centred around a focused study: whether the current state-of-the-art generalist LLMs canleverage the structures in a few examples to better construct the proof structures with incontext learning. Our study specifically focuses on structure-aware demonstration and structureawarepruning. We demonstrate that they both help improve performance. A detailed analysis is provided to help understand the results.

Read more

Overcoming Poor Word Embeddings with Word Definitions

/in /by

Modern natural language understanding models depend on pretrained subword embeddings, but applications may need to reason about words that were never or rarely seen during pretraining. We show that examples that depend critically on a rarer word are more challenging for natural language inference models. Then we explore how a model could learn to use definitions, provided in natural text, to overcome this handicap. Our model’s understanding of a definition is usually weaker than a well-modeled word embedding, but it recovers most of the performance gap from using a completely untrained word.

Read more

Teaching Syntax by Adversarial Distraction

/in /by

Existing entailment datasets mainly pose problems which can be answered without attention to grammar or word order. Learning syntax requires comparing examples where different grammar and word order change the desired classification. We introduce several datasets based on synthetic transformations of natural entailment examples in SNLI or FEVER, to teach aspects of grammar and word order. We show that without retraining, popular entailment models are unaware that these syntactic differences change meaning. With retraining, some but not all popular entailment models can learn to compare the syntax properly.

Read more