Prototype Retrieval is a task to retrieve or identify representative examples or prototypes from a given dataset. A prototype refers to a data point that encapsulates the essential characteristics or features of a particular category or class. Prototype retrieval is a useful concept for handling scenarios where there is a need to recognize and understand classes with limited examples, or even classes that were not present in the training data. It plays a key role in enhancing the generalization capabilities of machine learning models.

Posts

Retrieval, Analogy, and Composition: A framework for Compositional Generalization in Image Captioning

/in /by

Image captioning systems are expected to have the ability to combine individual concepts when describing scenes with concept combinations that are not observed during training. In spite of significant progress in image captioning with the help of the autoregressive generation framework, current approaches fail to generalize well to novel concept combinations. We propose a new framework that revolves around probing several similar image caption training instances (retrieval), performing analogical reasoning over relevant entities in retrieved prototypes (analogy), and enhancing the generation process with reasoning outcomes (composition). Our method augments the generation model by referring to the neighboring instances in the training set to produce novel concept combinations in generated captions. We perform experiments on the widely used image captioning benchmarks. The proposed models achieve substantial improvement over the compared baselines on both composition-related evaluation metrics and conventional image captioning metrics.

Read more