To deal with the problem of low robustness of SiamFC in complex scenarios in which the object is moving fast, the background is similar to the foreground, and the illumination is strong, a new tracking method called SA-Siam++ was proposed based on two-branch siamese network, including semantic branch which is used to extract semantic information through the hourglass-channel attention mechanism and the appearance branch which is used to extract appearance information through SiamFC. In addition, replacing the AlexNet network with an improved VGG-16 network can significantly increase the feature extraction capabilities. Finally, experiments were carried out on OTB-2013, OTB-2015, UAV123 and VOT2018 which are standard object tracking datasets. It is shown show that the obtained with the proposed algorithm are greatly improved compared with the existing mainstream algorithms, and the average frame rate reaches 49 FPS, that can meet the real-time requirements.