SentiBERT: Pre-training Language Model Combining Sentiment Information

SentiBERT: Pre-training Language Model Combining Sentiment Information

Blog Article

Pre-training language models on large-scale unsupervised corpus are attracting the attention of researchers in the field of natural language processing.The existing model mainly extracts the semantic and structural features of the text in the pre-training stage.Aiming at sentiment task and complex emotional features, a pre-training method focusing on learning Washing Machine Interface Module sentiment features is proposed on the basis of the latest pre-training language model BERT(bidirectional encoder representations from transformers).

In the further pre-training stage, this paper improves pre-training task of BERT with the help of sentiment dictionary.At the same time, this paper uses context-based word sentiment prediction task to classify the sentiment of masked words to acquire the textual representation biased towards sentiment features.Finally, fine-tuning is performed on a small labeled data sets.

Experimental results show that, compared with the original BERT model, the accuracy of sentiment tasks can be improved by 1 percentage point.More advanced results can be achieved at Inlet Valve Seal small training sets.