Fast Few-shot Debugging for NLU Test Suites

/in /by

Publication Date: 5/26/2022

Event: Deep Learning Inside Out workshop at ACL 2022

Reference: pp. 79-86, 2022

Authors: Christopher Malon, NEC Laboratories America, Inc.; Kai Li, NEC Laboratories America, Inc.; Erik Kruus, NEC Laboratories America, Inc.

Abstract: We study few-shot debugging of transformer based natural language understanding models, using recently popularized test suites to not just diagnose but correct a problem. Given a few debugging examples of a certain phenomenon, and a held-out test set of the same phenomenon, we aim to maximize accuracy on the phenomenon at a minimal cost of accuracy on the original test set. We examine several methods that are faster than full epoch retraining. We introduce a new fast method, which samples a few in-danger examples from the original training set. Compared to fast methods using parameter distance constraints or Kullback-Leibler divergence, we achieve superior original accuracy for comparable debugging accuracy.

Publication Link: https://aclanthology.org/2022.deelio-1.8/

Secondary Publication Link: https://arxiv.org/pdf/2204.06555.pdf