Publication Date: 12/7/2022
Event: 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Reference: pp. 338-350, 2023
Authors: Haris Widjaja, Carnegie Mellon University; Kiril Gashteovski, NEC Laboratories Europe; Wiem Ben Rim, NEC Laboratories Europe; Pengfei Liu, Carnegie Mellon University; Christopher Malon, NEC Laboratories America, Inc.; Daniel Ruffinelli, University of Mannheim; Carolin Lawrence, NEC Laboratories Europe; Graham Neubig, Carnegie Me
Abstract: Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples. To augment KGs with new knowledge, researchers proposed models for KG Completion (KGC) tasks such as link prediction, i.e., answering (h, p, ?) or (?, p, t) queries. Such models are usually evaluated with averaged metrics on a held-out test set. While useful for tracking progress, averaged single-score metrics cannotreveal what exactly a model has learned — or failed to learn. To address this issue, we propose KGxBoard: an interactive framework for performing fine-grained evaluation on meaningful subsets of the data, each of which tests individual and interpretable capabilities of a KGC model. In our experiments, we highlight the findings that we discovered with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
Publication Link: https://aclanthology.org/2022.emnlp-demos.34/