Abstract: The syntax elements extract from encoding video data streaming, such as motion vectors and residuals, can be used to characterize the motion of action recognition and obtain the better precision than optical-flow. But its inherent pixel noise and feature sparsity may also lead to some error when fine movements recognized. To address these issues, a dynamic gesture recognition framework was designed to get higher-precision and lower-complexity, by using the data optimization of syntax elements in coding video. Specifically, a key P-frame selection strategy is introduced to cope with the feature sparsity by selecting encoding frames which cover higher information content. Moreover, a joint residual feature representation method is proposed to remove the noisy motion vectors outside the hand by using finer gesture contour maps obtained from residuals. It is demonstrated that the presented model achieves the similar computation effects as optical flow. Experiments on the baseline dataset, VIVA dataset, SKIG dataset, NvGesture and EgoGesture dataset, the results show that the scheme achieves an average recognition accuracy of 82.94%, 99.72%, 81.12% and 90.48% using only RGB data, reducing storage overhead by 89% and achieving similar results to SOTA methods at 4.7 times the operating speed.

Key words: dynamic gesture recognition, encoded video, motion vector, residual, data optimization