Journal of Beijing University of Posts and Telecommunications

  • EI核心期刊

Journal of Beijing University of Posts and Telecommunications ›› 2023, Vol. 46 ›› Issue (4): 58-63.

Previous Articles     Next Articles

Lightweight Chinese Lipreading Model and Dataset Construction

SUN Baosheng,  XIE Dongliang   

  • Received:2022-05-31 Revised:2022-08-30 Online:2023-08-28 Published:2023-08-24
  • Contact: Dongliang Xie E-mail:xiedl@bupt.edu.cn

Abstract: In order to promote the rapid development and practical application of Chinese lipreading, a lightweight lipreading model is proposed based on the combination of interleaved group convolution and dilated convolution. In the proposed model, the interleaved group convolution is taken to learn the correlation between different features and the dilated convolution is taken to expand the model receptive field, which greatly reduces the amount and complexity of model parameter and improves the accuracy of model recognition. In addition, the largest sentence-level Chinese lipreading dataset is recorded in a controlled environment to enrich the Chinese lipreading dataset. The applicability of the lightweight lipreading model is verified on the recorded datasets and public datasets. The learning ability of the model to the video frame and text mapping relationship is analyzed visually through the heatmap.

Key words: Chinese lipreading , lightweight , interleaved group convolution, dilated convolution

CLC Number: