Log Template Extraction Algorithm Based on Normalized Feature Discrimination

doi:10.13190/j.jbupt.2019-033

Journal of Beijing University of Posts and Telecommunications ›› 2020, Vol. 43 ›› Issue (1): 68-73.doi: 10.13190/j.jbupt.2019-033

• Papers • Previous Articles Next Articles

Log Template Extraction Algorithm Based on Normalized Feature Discrimination

SHUANG Kai^1,2, LI Yi-wen¹, Lü Zhi-heng¹, HAN Jing³, LIU Jian-wei³

1. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China;
2. Science and Technology of Information Transmission and Dissemination in Communication Networks Laboratory, Shijiazhuang 050081, China;
3. ZTE Corporation, Shenzhen 518057, China

Received:2019-03-22 Online:2020-02-28 Published:2020-03-27
Supported by:

Abstract

Abstract: A log template extraction algorithm based on normalized feature discrimination is proposed, aiming at the problem that the number of clusters needs to be provided as a priori information in traditional log template extraction. First, log data is initially compressed to reduce data redundancy. Then, a log clustering process is implemented, and the normalized feature is used to discriminate whether the clustering result meets requirement:if so, the clustering process is successfully completed; if not, the number of log clusters is adjusted by using binary search and redo clustering. Finally, the log template is extracted via clustering results. In addition, an evaluation metric that measures the effectiveness of template extraction is designed. Experiments on real data indicated that the algorithm can achieve more stable and accurate template extraction performance than the benchmark method, and it had good generalization performance.

Key words: template extraction, log analysis, text clustering, normalized feature

CLC Number:

TP391

SHUANG Kai, LI Yi-wen, Lü Zhi-heng, HAN Jing, LIU Jian-wei. Log Template Extraction Algorithm Based on Normalized Feature Discrimination[J]. Journal of Beijing University of Posts and Telecommunications, 2020, 43(1): 68-73.

References

[1] 董蜻灵, 李芳, 何婷婷. 基于LDA模型的文本聚类研究[C]//孙茂松, 陈群秀. 中国计算语言学研究前沿进展(2009r 2011). 北京:清华大学出版社, 2011:455-461.
[2] Vaarandi R. A data clustering algorithm for mining patterns from event logs[C]//Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003). Kansas City:IEEE, 2003:119-126.
[3] Fu Q, Lou J G, Wang Y, et al. Execution anomaly detection in distributed systems through unstructured log analysis[C]//2009 Ninth IEEE International Conference on Data Mining. Miami:IEEE, 2009:149-158.
[4] Makanju A, Zincirheywood A N N, Milios E E. Clustering event logs using iterative partitioning[C]//Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM, 2009:1255-1264.
[5] Makanju A, Zincirheywood A N N, Milios E E. Investigating event log analysis with minimum apriori information[C]//Ifip/IEEE International Symposium on Integrated Network Management. Ghent:IEEE, 2013:962-968.
[6] Tang L, Li T, Perng C S. LogSig:generating system events from raw textual logs[C]//Proceedings of the 20th ACM International Conference on Information and Knowledge Management. New York:ACM, 2011:785-794.
[7] Nandi A, Mandal A, Atreja S, et al. Anomaly detection using program control flow graph mining from execution logs[C]//ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM, 2016:215-224.
[8] Vaarandi R, Pihelgas M. LogCluster-A data clustering and pattern mining algorithm for event logs[C]//2015 11th International Conference on Network and Service Management (CNSM). Barcelona:IEEE, 2015:1-7.
[9] Aharon M, Barash G, Cohen I, et al. One graph is worth a thousand logs:uncovering hidden structures in massive system event logs[C]//Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Berlin:Springer, 2009:227-243.
[10] Vaarandi R. A breadth-first algorithm for mining frequent patterns from event logs[C]//International Conference on Intelligence in Communication Systems. Berlin:Springer, 2004:293-308.

[1]	BAO Rongzhen, ZHU Zhiyu, YANG Yang, FENG Chunyan. Visible Light Communication Assisted Perspective Circle and Lines Positioning Algorithm Based on A Single Light Source [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(4): 32-39.
[2]	ZHANG Xiaoqian, WANG Xiao, XUE Xuqian, TAN Zhen, PU Lei. Image Segmentation Algorithm Based on Weighted Multi-Kernel Subspace Clustering [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(3): 78-83.
[3]	WANG Ming, LIN Beibei, ZHANG Shunli. Gait Recognition Based on Frame-Level and Spatio-Temporal Double Branch Network [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(3): 73-77.
[4]	CHANG Xiao, HUANG Zhibin, YU Min, YANG Wubing. A Deep Decision Tree Model for Aerospace Big Data [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(3): 1-6.
[5]	. Key Recognition Technology Based on Vibration Perception [J]. Journal of Beijing University of Posts and Telecommunications, 0, (): 0-0.
[6]	. Research On Pedestrian Detection Algorithm Based on Multi-camera Feature Fusion [J]. Journal of Beijing University of Posts and Telecommunications, 0, (): 0-0.
[7]	. Fine-grained emotion analysis of online comments based on the fusion of ontology and deep learning [J]. Journal of Beijing University of Posts and Telecommunications, 0, (): 0-0.
[8]	Qi-Wei GONG. PERC Roberta：Emotion Recognition in Conversation using ERC Roberta with Learning [J]. Journal of Beijing University of Posts and Telecommunications, 0, (): 0-0.
[9]	. 3D Segmentation of Brain Tumor MRI Image based on RAPNet [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(2): 91-97.
[10]	. Traditional clothing image classification algorithm based on multi-layer discriminant dictionary learning [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(2): 104-108.
[11]	. Simulation Reconstruction of Non-Newtonian Fluids for Monocular Video [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(2): 109-115.
[12]	. Ultrasonic Phased Array Focusing Scanning Imaging Method Based on Phase Shift Migration [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(2): 98-103.
[13]	. Stagewise Weakly Supervised Satellite Imagery Road Extraction Based on Road Centerline Scribbles [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(2): 84-90.
[14]	. Image Quality Evaluation Method Based on Human Visual System [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(2): 129-136.
[15]	RONG Zhenyu, LIU Jianyi. Retinal Blood Vessel Segmentation Based on Transformer and MLP [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(1): 26-31.

Log Template Extraction Algorithm Based on Normalized Feature Discrimination

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments