Matching Confidences and Softened Target Occurrences for Calibration

Publication Date: 11/27/2024

Event: Digital Image Computing: Techniques & Applications (DICTA 2024)

Reference: pp. 1-18, 2024

Authors: Vinith Kugathasan, Mohamed bin Zayed University of Artificial Intelligence; Honglu Zhou, Salesforce Research; Zachary Izzo, NEC Laboratories America, Inc.; Gayal Kuruppu, Mohamed bin Zayed University of Artificial Intelligence; Sejong Yoon, The College of New Jersey; Muhammad Haris Khan, Mohamed bin Zayed University of Artificial Intelligence

Abstract: The problem of calibrating deep neural networks (DNNs) is gaining attention, as these networks are becoming central to many real-world applications. Different attempts have been made to counter the poor calibration of DNNs. Amongst others, train-time calibration methods have unfolded as an effective class for improving model calibration. Motivated by this, we propose a novel train-time calibration method that is built on a new auxiliary loss formulation, namely multiclass alignment of confidences with the gradually softened ground truth occurrences (MACSO). It is developed on the intuition that, for a class, the gradually softened ground truth occurrences distribution is a suitable non-zero entropy signal whose better alignment with the predicted confidences distribution is positively correlated with reducing the model calibration error. In our train-time approach, besides simply aligning the two distributions, e.g., via their means or KL divergence, we propose to quantify the linear correlation between the two distributions, which preserves the relations among them, thereby further improving the calibration performance. Finally, we also reveal that MACSO posses desirable theoretical properties. Extensive results on several challenging datasets, featuring in and out-of-domain scenarios, class imbalanced problem, and a medical image classification task, validate the efficacy of our method against state-of-the-art train-time calibration methods.

Publication Link: