A Masked Language Model (MLM) is a type of pretraining technique used in natural language processing (NLP) where certain tokens in a given text sequence are randomly masked or replaced with a special token. The model is then trained to predict the original identity of the masked tokens based on the context provided by the surrounding words. BERT (Bidirectional Encoder Representations from Transformers) is a well-known example of a pretrained model that uses masked language modeling during its training phase. MLM helps the model learn contextualized representations of words within the broader context of a sentence or document.

Posts

Domain oriented Language Modeling with Adaptive Hybrid Masking and Optimal Transport Alignment

/in /by

Motivated by the success of pre-trained language models such as BERT in a broad range of natural language processing (NLP) tasks, recent research efforts have been made for adapting these models for different application domains. Along this line, existing domain-oriented models have primarily followed the vanilla BERT architecture and have a straightforward use of the domain corpus. However, domain-oriented tasks usually require accurate understanding of domain phrases, and such fine-grained phrase-level knowledge is hard to be captured by existing pre-training scheme. Also, the word co-occurrences guided semantic learning of pre-training models can be largely augmented by entity-level association knowledge. But meanwhile, there is a risk of introducing noise due to the lack of ground truth word-level alignment. To address the issues, we provide a generalized domain-oriented approach, which leverages auxiliary domain knowledge to improve the existing pre-training framework from two aspects. First, to preserve phrase knowledge effectively, we build a domain phrase pool as auxiliary knowledge, meanwhile we introduce Adaptive Hybrid Masked Model to incorporate such knowledge. It integrates two learning modes, word learning and phrase learning, and allows them to switch between each other. Second, we introduce Cross Entity Alignment to leverage entity association as weak supervision to augment the semantic learning of pre-trained models. To alleviate the potential noise in this process, we introduce an interpretable Optimal Transport based approach to guide alignment learning. Experiments on four domain-oriented tasks demonstrate the superiority of our framework.

Read more