全卷积神经结构的段落式图像描述算法

doi:10.13190/j.jbupt.2019-057

北京邮电大学学报 ›› 2019, Vol. 42 ›› Issue (6): 155-161.doi: 10.13190/j.jbupt.2019-057

全卷积神经结构的段落式图像描述算法

李睿凡^1,2, 梁昊雨¹, 冯方向¹, 张光卫^2,3, 王小捷^1,2

1. 北京邮电大学计算机学院, 北京 100876;
2. 教育部信息网络工程研究中心, 北京 100876;
3. 北京邮电大学网络技术研究院, 北京 100876

收稿日期:2019-04-14 出版日期:2019-12-28 发布日期:2019-11-15
作者简介:李睿凡(1975-),男,副教授,E-mail:rfli@bupt.edu.cn.
基金资助:
国家重点研发计划项目（2019YFF0303302）；国家自然科学基金项目（61906018）；国家电网公司总部科技项目（5200-201918255A-0-0-00）

Paragraph Image Captioning with Deep Fully Convolutional Neural Networks

LI Rui-fan^1,2, LIANG Hao-yu¹, FENG Fang-xiang¹, ZHANG Guang-wei^2,3, WANG Xiao-jie^1,2

1. School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China;
2. Engineering Research Center of Information Networks, Ministry of Education, Beijing 100876, China;
3. Institute of Network Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China

Received:2019-04-14 Online:2019-12-28 Published:2019-11-15

摘要/Abstract

摘要： 针对段落式图像描述生成研究中提升描述语句之间的连贯性问题，提出了一种基于全卷积结构的图像段落描述算法.采用基于卷积网络的区域检测器获取图像表示，结合段落在语言学角度的层次性，构建一种层次性的深度卷积解码器对图像表示解码，自动生成段落式文本描述.同时将门控机制嵌入卷积解码器网络中，以提升模型的记忆能力.实验结果表明，相比于基于循环神经网络等传统段落图像的描述方法，新算法能够为图像生成更为连贯的段落式文本描述，在评测指标上取得较好的结果.

关键词: 卷积网络, 深度学习, 图像描述, 连贯性

Abstract: How to improve the coherence among descriptive sentences for the paragraph image captioning is paid attention currently. A fully convolutional neural architecture for paragraph image captioning was proposed. An image representation is first obtained using a region detector based on a convolutional network. Then a hierarchical deep convolutional decoder is constructed to translate the image representation, automatically generating a paragraph text description. In addition, the gating mechanism is embedded in the convolutional decoder network to improve memory capacity of the model. Experiments demonstrate that compared with those traditional methods based on recurrent neural networks, the proposed algorithm can generate more coherent paragraph text descriptions for images, achieving better results on evaluation metrics.

Key words: convolutional networks, deep learning, image captioning, coherence

中图分类号:

TN309.2

李睿凡, 梁昊雨, 冯方向, 张光卫, 王小捷. 全卷积神经结构的段落式图像描述算法[J]. 北京邮电大学学报, 2019, 42(6): 155-161.

LI Rui-fan, LIANG Hao-yu, FENG Fang-xiang, ZHANG Guang-wei, WANG Xiao-jie. Paragraph Image Captioning with Deep Fully Convolutional Neural Networks[J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2019, 42(6): 155-161.

参考文献

[1] Vinyals O, Toshev A, Bengio S, et al. Show and tell:a neural image caption generator[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE Press, 2015:3156-3164.
[2] Lu Jiasen, Xiong Caiming, Parikh D, et al. Knowing when to look:adaptive attention via a visual sentinel for image captioning[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE Press, 2017:375-383.
[3] Mao Yuzhao, Zhou Chang, Wang Xiaojie, et al. Show and tell more:topic-oriented multi-sentence image captioning[C]//Proceedings of the 27^th International Joint Conference on Artificial Intelligence. California:International Joint Conferences on Artificial Intelligence Organization, 2018:4258-4264.
[4] Xu K, Ba J, Kiros R, et al. Show, attend and tell:neural image caption generation with visual attention[C]//International Conference on Machine Learning. Lille, France:ACM, 2015:2048-2057.
[5] You Quanzeng, Jin Hailin, Wang Zhaowen, et al. Image captioning with semantic attention[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE Press, 2016:4651-4659.
[6] Karpathy A, Li Feifei. Deep visual-semantic alignments for generating image descriptions[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE Press, 2015:3128-3137.
[7] Anderson P, He Xiaodong, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York:IEEE Press, 2018:6077-6086.
[8] Krause J, Johnson J, Krishna R, et al. A hierarchical approach for generating descriptive image paragraphs[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE Press, 2017:317-325.
[9] Liang Xiaodan, Hu Zhiting, Zhang Hao, et al. Recurrent topic-transition GAN for visual paragraph generation[C]//2017 IEEE International Conference on Computer Vision (ICCV). New York:IEEE Press, 2017:3362-3371.
[10] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems. Cambridge:MA, MIT Press, 2014:2672-2680.
[11] Chatterjee M, Schwing A G. Diverse and coherent paragraph generation from images[M]//Computer Vision-ECCV 2018. Cham:Springer International Publishing, 2018:747-763.
[12] Wang Z, Luo Y, Li Y, et al. Look deeper see richer:depth-aware image paragraph captioning[C]//2018 ACM Multimedia Conference. Association for Computing Machinery. New York:ACM Press, 2018:672-680.
[13] Che Wenbin, Fan Xiaopeng, Xiong Ruiqin, et al. Paragraph generation network with visual relationship detection[C]//2018 ACM Multimedia Conference on Multimedia-MM'18. New York:ACM Press, 2018:1435-1443.
[14] Dauphin Y N, Fan A, Auli M, et al. Language modeling with gated convolutional networks[C]//The 34^th International Conference on Machine Learning-Volume 70. Sydney, Australia:ACM Press, 2017:933-941.
[15] Krishna R, Zhu Yuke, Groth O, et al. Visual genome:connecting language and vision using crowdsourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123(1):32-73.
[16] Chen X, Fang H, Lin T Y, et al. Microsoft COCO captions:data collection and evaluation server[J]. arXiv preprint arXiv:1504. 00325, 2015.
[17] Papineni K, Roukos S, Ward T, et al. BLEU:a method for automatic evaluation of machine translation[C]//The 40^th Annual Meeting on Association for Computational Linguistics (ACL). PA, USA:ACL, 2002:311-318.
[18] Vedantam R, Zitnick C L, Parikh D. CIDEr:consensus-based image description evaluation[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York:IEEE Press, 2015:4566-4575.

全卷积神经结构的段落式图像描述算法

Paragraph Image Captioning with Deep Fully Convolutional Neural Networks

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	刘阳, 滕颖蕾, 牛涛, 郅佳琳. 基于深度强化学习的滤波器剪枝方案[J]. 北京邮电大学学报, 2023, 46(3): 31-36.
[2]	靳梦凡, 黄智濒, 储志强. 复杂环境下多模态指导的点云补全方法[J]. 北京邮电大学学报, 2023, 46(3): 103-108.
[3]	黄跃珍戴晶帼张承业魏东. 基于关联图关键边发现的人脸图像聚类算法[J]. 北京邮电大学学报, 2023, 46(1): 97-102.
[4]	荣震宇刘建毅. 基于Transformer和MLP的眼底血管分割算法[J]. 北京邮电大学学报, 2023, 46(1): 26-31.
[5]	杨杰张书杰. 基于密集YOLOv3的印刷电路板缺陷识别[J]. 北京邮电大学学报, 2022, 45(5): 42-48.
[6]	刘金桐, 杨国兴, 刘晓鸿, 王光宇. 针对心电图分类模型的平滑攻击算法[J]. 北京邮电大学学报, 2022, 45(4): 44-50.
[7]	黄柳婷, 刘可欣, 牛凯, 常春, 贺志强. 基于深度学习的哮喘患者CT影像黏液栓自动识别[J]. 北京邮电大学学报, 2022, 45(4): 58-63.
[8]	柳长源, 虎浩媛, 毕晓君. 双线性融合网络的驾驶员分心行为识别[J]. 北京邮电大学学报, 2022, 45(2): 79-84.
[9]	赵海英, 朱会, 侯小刚. 基于改进EMA单元的传统服饰图像语义分割[J]. 北京邮电大学学报, 2022, 45(1): 69-74.
[10]	张滨宇, 赵衍运, 杜昀昊, 万俊峰, 佟知航. 一种基于深度学习的PCB图像字符检测方法[J]. 北京邮电大学学报, 2022, 45(1): 108-114.
[11]	贾珺, 冯春燕, 夏海轮, 张天魁, 李成钢. 基于样本均衡与特征交互的通信网络故障预测方法[J]. 北京邮电大学学报, 2021, 44(6): 59-66.
[12]	普运伟, 郭江, 刘涛涛, 吴海潇. 基于多学习单元卷积神经网络的雷达辐射源信号识别[J]. 北京邮电大学学报, 2021, 44(6): 74-82.
[13]	王文竹, 肖波, 陈柯宏. 一种面向目标的情感极性分析方法[J]. 北京邮电大学学报, 2021, 44(5): 21-27.
[14]	王艺霏, 莫爽, 吴文睿, 范少华, 肖丁. 基于内外卷积网络的网络入侵检测[J]. 北京邮电大学学报, 2021, 44(5): 94-100.
[15]	张栋, 陈鸿龙. 融合文本信息的轻量级图卷积网络推荐模型[J]. 北京邮电大学学报, 2021, 44(5): 88-93.