用于文本分类的多探测任务语言模型微调

doi:10.13190/j.jbupt.2019-149

北京邮电大学学报 ›› 2019, Vol. 42 ›› Issue (6): 76-83.doi: 10.13190/j.jbupt.2019-149

用于文本分类的多探测任务语言模型微调

傅群超, 王枞

1. 北京邮电大学软件学院, 北京 100876;
2. 北京邮电大学可信分布式计算与服务教育部重点实验室, 北京 100876

收稿日期:2019-11-22 出版日期:2019-12-28 发布日期:2019-11-15
作者简介:傅群超(1992-),男,博士生,E-mail:fuqunchao@bupt.edu.cn;王枞(1958-),女,教授,博士生导师.
基金资助:
国家重点研发计划项目（2017YFC1307705）

Based on Multiple Probing Tasks Fine-Tuning of Language Models for Text Classification

FU Qun-chao, WANG Cong

1. School of Software Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China;
2. Key Laboratory of Trustworthy Distributed Computing and Service(Beijing University of Posts and Telecommunications), Ministry of Education, Beijing 100876, China

Received:2019-11-22 Online:2019-12-28 Published:2019-11-15

摘要/Abstract

摘要： 预训练语言模型被广泛运用在多项自然语言处理任务中，但是对于不同的任务没有精细的微调.针对文本分类任务，提出基于探测任务的语言模型微调方法，利用探测任务训练模型特定的语言学知识，可提高模型在文本分类任务上的性能.设计了6个探测任务，覆盖句子浅层、语法和语义三方面信息.最后在6个文本分类数据集上验证了本文的方法，使分类错误率得到改善.

关键词: 探测任务, 语言模型, 多任务学习, 文本分类

Abstract: Pre-trained language models are widely used in many natural language processing tasks, but there is no fine-tuning for different tasks. Therefore, for text classification task, the author proposes a method of fine-tuning language model based on probing task, which utilizes the specific linguistic knowledge of probing task training model, and improves the performance of the model in text classification task. Six probing tasks are given to cover the shallow information of sentences, grammar and semantics. The method is shown validated on six text classification datasets, and classification error rate is improved.

Key words: probing task, language model, multiple task, text classification

中图分类号:

TN929.53

傅群超, 王枞. 用于文本分类的多探测任务语言模型微调[J]. 北京邮电大学学报, 2019, 42(6): 76-83.

FU Qun-chao, WANG Cong. Based on Multiple Probing Tasks Fine-Tuning of Language Models for Text Classification[J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2019, 42(6): 76-83.

参考文献

[1] Dai A M, Le Q V. Semi-supervised sequence learning[C]//Advances in Neural Information Processing Systems. Montréal, Canada:[s.n.], 2015:3079-3087.
[2] Howard J, Ruder S. Universal language model fine-tuning for text classification[C]//Proceedings of the 56^th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers). Stroudsburg, PA, USA:Association for Computational Linguistics, 2018:328-339.
[3] Peters M, Ammar W, Bhagavatula C, et al. Semi-supervised sequence tagging with bidirectional language models[C]//Proceedings of the 55^th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers). Stroudsburg, PA, USA:Association for Computational Linguistics, 2017:1756-1765.
[4] Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations[C]//Proceedings of NAACL-HLT. New Orleans:[s.n.], 2018:2227-2237.
[5] Radford A, Narasimhan K, Salimans T, et al. Improving language understanding by generative pre-training[EB/OL]. (2018-06-11)[2019-06-17]. https://s3-us-west-2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/language understanding paper.pdf.
[6] Devlin J, Chang M, Lee K, et al. Bert:pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018-01-15)[2019-06-17]. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
[7] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. Long Beach, America:[s.n.], 2017:5998-6008.
[8] Liu P J, Saleh M, Pot E, et al. Generating wikipedia by summarizing long sequences[C]//Sixth International Conference on Learning Representations Vancouver. Canada:[s.n.], 2018:557-573.
[9] Kitaev N, Klein D. Constituencyparsing with a self-attentive encoder[C]//Proceedings of the 56^th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg. PA, USA:Association for Computational Linguistics, 2018:2676-2686.
[10] Suzuki J, Isozaki H. Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data[C]//Proceedings of ACL-08:HLT. Columbus, Ohio:[s.n.], 2008:665-673.
[11] Kamal N, Andrew M, Tom M. Semi-supervised text classification using EM[M]//Semi-Supervised Learning. Boston:The MIT Press, 2006:32-55.
[12] Liang P. Semi-supervised learning for natural language[D]. Massachusetts:Massachusetts Institute of Technology, 2005.
[13] Chen Danqi, Manning C. A fast and accurate dependency parser using neural networks[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Stroudsburg, PA, USA:Association for Computational Linguistics, 2014:740-750.
[14] Qi Ye, Sachan D, Felix M, et al. When and why are pre-trained word embeddings useful for neural machine translation?[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 2(Short Papers), Stroudsburg, PA, USA:Association for Computational Linguistics, 2018:529-535.
[15] Logeswaran L, Lee H. An efficient framework for learning sentence representations[C]//Sixth International Conference on Learning Representations Vancouver. Canada:[s.n.], 2018:1884-1891.
[16] Erhan D, Bengio Y, Courville A, et al. Why does unsupervised pre-training help deep learning?[J]. Journal of Machine Learning Research, 2010, 11(2):625-660.
[17] Hochreiter S, Schmidhuber J. Long short-term memory[J]. NeuralComputation, 1997, 9(8):1735-1780.
[18] Min S, Seo M, Hajishirzi H. Question answering through transfer learning from large fine-grained supervision data[C]//Proceedings of the 55^th Annual Meeting of the Association for Computational Linguistics(Volume 2:Short Papers). Stroudsburg, PA, USA:Association for Computational Linguistics, 2017:510-517.
[19] Severyn A, Moschitti A. UNITN:training deep convolutional neural network for twitter sentiment classification[C]//Proceedings of the 9^th International Workshop on Semantic Evaluation (SemEval 2015). Stroudsburg, PA, USA:Association for Computational Linguistics, 2015:464-469.
[20] Sennrich R, Haddow B, Birch A. Improving neural machine translation models with monolingual data[C]//Proceedings of the 54^th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg, PA, USA:Association for Computational Linguistics, 2016:86-96.
[21] Mou Lili, Meng Zhao, Yan Rui, et al. How transferable are neural networks in NLP applications?[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, PA, USA:Association for Computational Linguistics, 2016:479-489.
[22] Adi Y, Kermany E, Belinkov Y, et al. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks[J]. arXiv preprint arXiv:1608. 04207. 2016.
[23] Conneau A, Kruszewski G A N, Lample G, et al. What you can cram into a single vector:Probing sentence embeddings for linguistic properties[J]. CoRR, 2018:01070.
[24] Linzen T, Dupoux E, Goldberg Y. Assessing the ability of LSTMs to learn syntax-sensitive dependencies[J]. Transactions of the Association for Computational Linguistics, 2016(4):521-535.
[25] Gulordava K, Bojanowski P, Grave E, et al. Colorless green recurrent networks dream hierarchically[C]//Proceedings of NAACL-HLT. New Orleans, Louisiana:[s.n.], 2018:1195-1205.
[26] Kingma D P, Ba J L. Adam:Amethod for stochastic optimization[C]//the 3^rd International Conference on Learning Representations. San Diego, America:[s.n.], 2015:351-365.
[27] Zhu Yukun, Kiros R, Zemel R, et al. Aligning books and movies:towards story-like visual explanations by watching movies and reading books[C]//2015 IEEE International Conference on Computer Vision (ICCV). New York:IEEE Press, 2015:19-27.
[28] Yosinski J, Clune J, Bengio Y, et al. How transferable are features in deep neural networks?[C]//Advances in Neural Information Processing Systems. Montréal Canada:[s.n.], 2014:3320-3328.
[29] Maas A L, Daly R E, Pham P T, et al. Learning word vectors for sentiment analysis[C]//Proceedings of the 49^th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies. Portland, Oregon:[s.n.], 2011:142-150.
[30] Zhang X, Zhao J, Lecun Y. Character-level convolutional networks for text classification[C]//Advances in Neural Information Processing Systems. Montréal Canada:[s.n.], 2015:649-657.
[31] Voorhees E M, Tice D M. The TREC-8 question answering track evaluation[C]//TREC. Citeseer. Gaithersburg:[s.n.], 1999:82.
[32] Ba J L, Kiros J R, Hinton G E. Layer normalization[J]. arXiv preprint arXiv:1607. 06450. 2016.
[33] Mccann B, Bradbury J, Xiong C, et al. Learned in translation:contextualized word vectors[C]//Advances in Neural Information Processing Systems. Long Beach, USA:[s.n.], 2017:6294-6305.
[34] Zhang Y, Wallace B. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification[C]//Proceedings of the Eighth International Joint Conference on Natural Language Processing. Taipei:[s.n.], 2017:253-263.
[35] Johnson R, Zhang Tong. Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55^th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Stroudsburg, PA, USA:Association for Computational Linguistics, 2017:562-570.
[36] Tai Kaisheng, Socher R, Manning C D. Improved semantic representations from tree-structured long short-term memory networks[C]//Proceedings of the 53^rd Annual Meeting of the Association for Computational Linguistics and the 7^th International Joint Conference on Natural Language Processing (Volume 1:Long Papers). Stroudsburg, PA, USA:Association for Computational Linguistics, 2015:1556-1566.

用于文本分类的多探测任务语言模型微调

Based on Multiple Probing Tasks Fine-Tuning of Language Models for Text Classification

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 12

编辑推荐

Metrics

本文评价

[1]	张天骐汪锐安泽亮方竹王雪怡. 基于多任务学习的MIMO-OFDM信号联合信噪比估计与调制识别[J]. 北京邮电大学学报, 2022, 45(6): 98-104.
[2]	谢永红, 陶浒, 贾麒, 杨石兵, 韩辛亮. 基于预训练语言模型的中医症状标准化方法[J]. 北京邮电大学学报, 2022, 45(4): 13-18,57.
[3]	谢永红陶浒贾麒杨石兵韩辛亮. 基于预训练语言模型的中医症状标准化方法[J]. 北京邮电大学学报, 2022, 45(4): 14-20.
[4]	田亮, 潘鹏, 刘玉敏. 基于神经网络的纳米光子结构逆设计[J]. 北京邮电大学学报, 2022, 45(3): 112-116.
[5]	李建桂, 梁越, 高鹏飞, 刘绍华, 马应龙. 基于层次化类别嵌入的电力ICT系统故障分类[J]. 北京邮电大学学报, 2021, 44(4): 34-40.
[6]	王玉峰, 肖灿彬, 陈焱, 金群. 一种利用多任务学习的短期住宅负荷预测方案[J]. 北京邮电大学学报, 2021, 44(3): 47-52.
[7]	刘逸琛, 孙华志, 马春梅, 姜丽芬, 钟长鸿. 一种基于高层特征融合的网络商品分类[J]. 北京邮电大学学报, 2020, 43(5): 98-104,117.
[8]	莫歧, 王小捷. 联合分类与匹配的FAQ问答模型[J]. 北京邮电大学学报, 2019, 42(4): 76-81.
[9]	黄文良1,2, 李石坚1, 刘菊新1, 徐从富1. 一个大规模垃圾短信实时过滤系统[J]. 北京邮电大学学报, 2008, 31(3): 33-37.
[10]	张锦1,2, 李光3, 曹伍4, 胡瑞芬1. 基于主成分分析的自动文本分类模型研究[J]. 北京邮电大学学报, 2006, 29(s2): 136-138.
[11]	吕琳，刘玉树，刘妍. 运用语义集索引法实现英文文本分类[J]. 北京邮电大学学报, 2006, 29(2): 18-21.
[12]	孙健, 王伟, 钟义信. 基于K-最近距离的自动文本分类的研究[J]. 北京邮电大学学报, 2001, 24(1): 42-46.