Abstract: To capture the global and local features of sign language actions and preserve the original structure and context in the image, an improved convolution multilayer perceptron Transformer ( CM-Transformer) model is proposed for continuous sign language recognition. The structural consistency advantage of convolution layer and the global modeling performance of self attention model encoder are combined by CM-Transformer to capture long-term sequence dependence. Meanwhile, the feedforward layer of self attention model is replaced by multilayer perceptron to perform translation invariance and locality. In addition, random frame discarding and random gradient stopping techniques are used to reduce the training computation in time and space, and prevent over fitting. Thus, an efficient and lightweight network has been constructed. Finally the connectionist temporal classification decoder is used to align the input and output sequences to obtain the final recognition result. Experimental results on two large benchmark data sets show the effectiveness of the proposed method.

Key words: continuous sign language recognition,  convolutional neural network, self-attention model,  multilayer perceptron