一种基于滑动窗口分割的中国手语识别系统

doi:10.13190/j.jbupt.2021-001

北京邮电大学学报 ›› 2021, Vol. 44 ›› Issue (5): 48-54.doi: 10.13190/j.jbupt.2021-001

一种基于滑动窗口分割的中国手语识别系统

王鑫炎¹, 王青山¹, 马晓迪¹, 刘鹏², 戴海鹏³

1. 合肥工业大学数学学院, 合肥 230601;
2. 杭州电子科技大学计算机学院, 杭州 310018;
3. 南京大学计算机科学与技术学院, 南京 210023

收稿日期:2021-03-26 出版日期:2021-10-28 发布日期:2021-09-06
通讯作者: 王青山(1975-),男,教授,E-mail:qswang@hfut.edu.cn. E-mail:qswang@hfut.edu.cn
作者简介:王鑫炎(1998-),男,本科生.

A Split Sliding Window-Based Continuous Chinese Sign Language Recognition System

WANG Xin-yan¹, WANG Qing-shan¹, MA Xiao-di¹, LIU Peng², DAI Hai-peng³

1. Institute of Mathematics, Hefei University of Technology, Hefei 230601, China;
2. School of Computer Science, Hangzhou Dianzi University, Hangzhou 310018, China;
3. School of Computer Science and Technology, Nanjing University, Nanjing 210023, China

Received:2021-03-26 Online:2021-10-28 Published:2021-09-06

摘要/Abstract

摘要： 听力障碍者在全世界残疾人群体中占有较大的比重.他们能通过手语与健全人交流，但因手语不被大众所掌握，导致彼此交流存在较大障碍.为此提出了一种基于滑动窗口分割（SSW）的连续中国手语识别系统来实现手语自动识别.SSW系统将通过滑动窗口选取出来的手语信号平均分割，依次删去其中一组数据，从而得到新的数据，输入手语识别神经网络进行训练，得出单个手语单词手势预测值，最后运用基于阈值的多投票策略对识别出的预测值进行判断，得出识别结果.SSW系统在对20名志愿者采集的30条手语语句上进行训练，结果显示，所提SSW系统自动识别手语的平均准确率在测试集上达到83.9%，较长短期记忆网络模型提高了16.7%.

关键词: 滑动窗口, 双向长短期记忆网络, 阈值, 数据分割, 手语识别

Abstract: A large proportion of the world's disabled population is accounted for the individuals with hearing impairment which can communicate with people through the sign language. However, sign language is not mastered by the public, and there are still big obstacles between the individuals with hearing impairment and the normal people. A continuous Chinese sign language recognition system based on split sli-ding window (SSW) to realize automatic sign language recognition is proposed. The SSW system divides the sign language signal selected through the sliding window, and deletes one group of data to get new data in the original order, which is inputted to the sign language recognition neural network for training to obtain the gesture prediction value of a single sign language word. Finally, the majority voting strategy based on threshold is used to judge the identified prediction values. The SSW system is trained on 30 sign language sentences collected by 20 volunteers. The results show that the average accuracy of the SSW system reachs 83.9% on the test dataset, which is 16.7% higher than the long short-term memory model.

Key words: sliding window, bi-directional long short-term memory network, threshold, data segmentation, sign language recognition

中图分类号:

TP183

王鑫炎, 王青山, 马晓迪, 刘鹏, 戴海鹏. 一种基于滑动窗口分割的中国手语识别系统[J]. 北京邮电大学学报, 2021, 44(5): 48-54.

WANG Xin-yan, WANG Qing-shan, MA Xiao-di, LIU Peng, DAI Hai-peng. A Split Sliding Window-Based Continuous Chinese Sign Language Recognition System[J]. Journal of Beijing University of Posts and Telecommunications, 2021, 44(5): 48-54.

参考文献

[1] Mellon N, Niparko J K, Rathmann C, et al. Should all deaf children learn sign language[J]. Pediatrics, 2015, 136(1):170-176.
[2] Koller O, Zargaran S, Ney H, et al. Deep sign:hybrid CNN-HMM for continuous sign language recognition[C]//BMVC 2016. York:BMVC, 2016:1-12.
[3] Koller O, Zargaran S, Ney H, et al. Re-sign:re-aligned end-to-end sequence modelling with deep recurrent CNN-HMMs[C]//CVPR 2017. Hawaii:IEEE Press, 2017:3416-3424.
[4] Cui Runpeng, Liu Hu, Zhang Changshui, et al. Recu-rrent convolutional neural networks for continuous sign language recognition by staged optimization[C]//CVPR 2017. Hawaii:IEEE Press, 2017:1610-1618.
[5] Yang Wenwen, Tao Jinxu, Ye Zhongfu, et al. Conti-nuous sign language recognition using level building based on fast hidden Markov model[J]. Pattern Recognition Letters, 2016, 78(15):28-35.
[6] Huang Jie, Zhou Wenggang, Zhang Qilin, et al. Video-based sign language recognition without temporal segmentation[C]//AAAI-18. Louisiana:AAAI Press, 2018:2257-2264.
[7] Guo Dan, Zhou Wengang, Wang Meng, et al. Hierarchical LSTM for sign language translation[C]//AAAI-18. Louisiana:AAAI Press, 2018:6845-6852.
[8] Zhang Jihai, Zhou Wengang, Li Houqiang, et al. A threshold-based HMM-DTW approach for continuous sign language recognition[C]//ICIMCS'14. New York:ACM Press, 2014:237-240.
[9] Li Kehuang, Zhou Zhengyu, Lee C H. Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications[J]. ACM Transactions on Accessible Computing, 2016, 8(2):1-23.
[10] Bukhari J, Rehman M, Malik S I, et al. American sign language translation through sensory glove; signspeak[J]. International Journal of u- and e- Service, Science and Technology, 2015, 8(1):131-142.
[11] Kamal S M, Chen Y, Li S, et al. Technical approaches to Chinese sign language processing:a review[J]. IEEE Access, 2019, 7:96926-96935.
[12] Gao Wen, Fang Gaolin, Zhao Debin, et al. A Chinese sign language recognition system based on SOFM/SRN/HMM[J]. Pattern Recognition, 2004, 37(12):2389-2402.
[13] Benalcazar M E, Jaramillo A G, Zea J A, et al. Hand gesture recognition using machine learning and the Myo armband[C]//25th European Signal Processing Confe-rence. Kos island:Institute of Electrical and Electronics Engineers Incorporated, 2017:1040-1044.
[14] Yang Xidong, Chen Xiang, Cao Xiang, et al. Chinese sign language recognition based on an optimized tree-structure framework[J]. IEEE Journal of Biomedical and Health Informatics, 2017, 21(4):994-1004.
[15] Kaya E, Kumbasar T. Hand gesture recognition systems with the wearable Myo armband[C]//CEIT 2018. Istanbul:IEEE Press, 2018:1-6.
[16] Zhuang Yongjie, Lv Bo, Sheng Xinjun, et al. Towards Chinese sign language recognition using surface electromyography and accelerometers[C]//201724th International Conference on Mechatronics and Machine Vision in Practice. Albany:IEEE Press, 2017:1-5.
[17] Yu Yi, Chen Xiang, Cao Shuai, et al. Exploration of Chinese sign language recognition using wearable sensors based on deep belief net[J]. IEEE Journal of Biome-dical and Health Informatics, 2020, 24(5):1310-1320.
[18] Mittal A, Kumar P, Roy P P, et al. A modified LSTM model for continuous sign language recognition using leap motion[J]. IEEE Sensors Journal, 2019, 19(16):7056-7063.
[19] Gupta R, Jha N. Real-time continuous sign language classification using ensemble of windows[C]//ICACCS 2020. Copenhagen:IEEE Press, 2020:73-78.
[20] Wahid F, Tafreshi R, Langari R, et al. A multi-window majority voting strategy to improve hand gesture recognition accuracies using electromyography signal[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(2):427-436.
[21] Hou Jiahui, Li Xiangyang, Zhu Peide, et al. Signspeaker:a real-time, high-precision smartwatch-based sign language translator[C]//MobiCom'19. New York:ACM Press, 2019:1-15.

一种基于滑动窗口分割的中国手语识别系统

A Split Sliding Window-Based Continuous Chinese Sign Language Recognition System

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 10

编辑推荐

Metrics

本文评价

[1]	叶康张淑军郭淇李辉崔雪红. 基于CM-Transformer的连续手语识别[J]. 北京邮电大学学报, 2022, 45(5): 49-53,78.
[2]	张金玲, 崔彤, 刘为斌. 基于自适应时间窗函数优化的心电特征波形识别[J]. 北京邮电大学学报, 2019, 42(4): 82-88.
[3]	邱雪松, 张珣, 宋彦斌, 赵兵, 徐思雅. 基于熵和线性关系的两级流量异常检测方法[J]. 北京邮电大学学报, 2018, 41(4): 56-62.
[4]	张金玲, 郑鑫. 平稳小波变换和新阈值函数用于心电信号去噪[J]. 北京邮电大学学报, 2017, 40(1): 28-31.
[5]	王海瑞. 应用GPD模型评估实时通信系统WCET[J]. 北京邮电大学学报, 2016, 39(s1): 6-9.
[6]	田立勤, 李君建, 毋泽南. 权重均衡优化的Web用户行为的信任评估[J]. 北京邮电大学学报, 2016, 39(6): 99-103,130.
[7]	赵生妹, 唐文娟, 郑宝玉. 广义半迭代硬阈值追踪算法及在“鬼”成像中的应用[J]. 北京邮电大学学报, 2016, 39(1): 117-121.
[8]	秦晓芳, 隋东, 王翯, 张欣, 杨大成. 联合HARQ与H.264的无线视频传输[J]. 北京邮电大学学报, 2008, 31(1): 79-83.
[9]	王海婴1 ，吕静1 ，袁璐1 ，石明军2. 基于3G无线网络的H.264视频包抗误码传输方案[J]. 北京邮电大学学报, 2006, 29(3): 126-130.
[10]	崔毅东，林宇，徐雅静，徐惠民. 新的基于逐跳时间标签的链路利用率测量方法[J]. 北京邮电大学学报, 2006, 29(2): 5-9.