Analytics Towards Big Data

doi:10.13190/j.jbupt.2015.03.001

JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM ›› 2015, Vol. 38 ›› Issue (3): 1-12.doi: 10.13190/j.jbupt.2015.03.001

• Review • Next Articles

Analytics Towards Big Data

GAO Zhi-peng¹, NIU Kun², LIU Jie³

1. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China;
2. School of Software Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China;
3. School of Electronic Engineering, Beijing University of Posts and Telecommunications, Beijing 100876, China

Received:2015-03-27 Online:2015-06-28 Published:2015-06-28

Abstract

Abstract:

Big data analysis aiming at knowledge discovery from the analysis of big data, which involves visualization analysis, data mining, prediction analysis, semantic analysis and data quality management, plays an essential part in the procedures of big data processing. In this paper, it introduces the basic theories, cutting-edge technologies and processing tools for big data analysis, and summarizes encountered opportunities and challenges, along with discussions about the development trend and future prospects of big data analysis.

Key words: big data, visual analysis, data mining, predictive analysis, semantic analysis, data quality management

CLC Number:

TN911.22

GAO Zhi-peng, NIU Kun, LIU Jie. Analytics Towards Big Data[J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2015, 38(3): 1-12.

References

[1] 任磊, 杜一, 马帅, 等. 大数据可视分析综述[J]. 软件学报, 2014, 25(9): 1909-1936. Ren Lei, Du Yi, Ma Shuai, et al. Visual analytics towards big data[J]. Journal of Software, 2014, 25(9): 1909-1936.

[2] 维克托·迈尔·舍恩伯格, 肯尼斯·库克耶. 大数据时代[M]. 杭州: 浙江人民出版社, 2012. Viktor M S, Kenneth C. Big data: a revolution that will transform how we live, work and think[M]. Hangzhou: Zhejiang People's Publishing House, 2012.

[3] Gantz J, Reinsel D. 2011 digital universe study: extracting vaLue from chaos[M]. IDC Go-to-Market Services, 2011: 1-12.

[4] 李学龙, 龚海刚. 大数据系统综述[J]. 中国科学: 信息科学, 2015, 45(1): 1-44. Li Xuelong, Gong Haigang. A survey on big data systems[J]. Science China Information Sciences, 2015, 45(1): 1-44.

[5] 陶雪娇, 胡晓峰, 刘洋. 大数据研究综述[J]. 系统仿真学报, 2013, 8(25): 144-145. Tao Xuejiao, Hu Xiaofeng, Liu Yang. Overview of big data research[J]. Journal of System Simulation, 2013, 8(25): 144-145.

[6] 孟小峰, 慈祥. 大数据管理: 概念、技术与挑战[J]. 计算机研究与发展, 2013, 50(1): 146-169. Meng Xiaofeng, Ci Xiang. Big data management: concepts, techniques and challenges[J]. Journal of Computer Research and Development, 2013, 50(1): 146-169.

[7] 宫夏屹, 李伯虎, 柴旭东, 等. 大数据平台技术综述[J]. 系统仿真学报, 2014, 26(3): 489-496. Gong Xiayi, Li Bohu, Chai Xudong, et al. Survey on big data platform technology[J]. Journal of System Simulation, 2014, 26(3): 489-496.

[8] 程学旗, 靳小龙, 王元卓, 等. 大数据系统和分析技术综述[J]. 软件学报, 2014, 25(9): 1889-1908. Cheng Xueqi, Jin Xiaolong, Wang Yuanzhuo, et al. Survey on big data system and analytic technology[J]. Journal of Software, 2014, 25(9): 1889-1908.

[9] 覃雄派, 王会举, 杜小勇, 等. 大数据分析——RDBMS与MapReduce的竞争与共生[J]. 软件学报, 2012, 23(1): 32-45. Qin Xiongpai, Wang Huiju, Du Xiaoyong, et al. Big data analysis—competition and symbiosis of RDBMS and MapReduce[J]. Journal of Software, 2012, 23(1): 32-45.

[10] 顾君忠. 大数据与大数据分析[J]. 软件产业与工程, 2013(4): 17-21. Gu Junzhong. Big data and big data analysis[J]. Software Industry and Engineering, 2013(4): 17-21.

[11] Pirolli P, Card S K. The sensemaking process and leverage points for analyst technology as identified though cognitive task analysis[C]//Maybury M. Proc. of the Int'l Conf. on Intelligence Analysis. MacLean: MITRE, 2005: 1-6.

[12] Pirolli P, Card S K. Information foraging in information access environments[C]//Katz I R. Proc. of the CHI. New York: ACM Press, 1995: 51-58.

[13] Pirolli P. Information foraging theory: adaptive interaction with information[M]. New York: Oxford University Press, 2007: 31-35.

[14] Card S K, Mackinlay J D, Shneiderman B. Readings in information visualization: vsing vision to think[M]. San Francisco: Morgan-Kaufmann Publishers, 1999: 1-712.

[15] Green T M, William R, Brian F. Visual analytics for complex concepts using a human cognition model[C]//Grinsten G. Proc. of the VAST. Columbus: IEEE Press, 2008: 91-98.

[16] Wunsche B. A survey, classification and analysis of perceptual concepts and their application for the effective visualisation of complex information[C]//Chrucher N, Churcher C. Proc. of the APVIS. Darlinghurst: Australian Computer Society, 2004: 17-24.

[17] North C, Chang R, Endert A, et al. Analytic provenance: process+interaction+insight[C]//Tan D. Proc. of the CHI. New York: ACM Press, 2011: 33-36.

[18] Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations[C]//Gershon N. Proc. of the INFOVIS. San Francisco: IEEE Press, 1996: 336-343.

[19] Eades P, Huang Maolin. Navigating clustered graphs using force-directed methods[J]. Journal of Graph Algorithms and Applications, 2000, 4(3): 157-181.

[20] Brehmer M, Munzner T. A multi-level typology of abstract visualization tasks[J]. IEEE Trans. on Visualization and Computer Graphics, 2013, 19(12): 2376-2385.

[21] Keim D, Andrienko G, Fekete J, et al. Visual analytics: definition, process, and challenges[C]//Kerren A. Proc. of the Information Visualization. LNCS 4950. Berlin: Springer-Verlag, 2008: 154-175.

[22] Pike W A, Stasko S J, Chang R, et al. The science of interaction[J]. Information Visualization, 2009, 8(4): 263-274.

[23] Puerta A, Eisenstein J. Towards a general computational framework for model-based interface development systems[J]. Knowledge-Based Systems, 1999, 12(8): 433-442.

[24] Zhao Jian, Chevalier F, Collins C, et al. Facilitating discourse analysis with interactive visualization[J]. IEEE Trans. on Visualization and Computer Graphics, 2012, 18(12): 2639-2648.

[25] Collins C, Carpendale S, Penn G. Docuburst: visualizing document content using language structure[J]. Computer Graphics Forum, 2009, 28(3): 1039-1046.

[26] Paulovich F V, Minghim R. Hipp: a novel hierarchical point placement strategy and its application to the exploration of document collections[J]. IEEE Trans. on Visualization and Computer Graphics, 2008, 14(6): 1229-1236.

[27] Herman I, Melancon G, Marshall M S. Graph visualization and navigation in information visualization: a survey[J]. IEEE Trans. on Visualization and Computer Graphics, 2000, 6(1): 24-43.

[28] Shneiderman B. Tree visualization with tree-maps: 2-d spacing-filling approach[J]. ACM Trans. on Graphics, 1992, 11(1): 92-99.

[29] Zhang Xiu, Yuan Xiaoru. Treemap visualization[J]. Journal of Computer-Aided Design and Computer Graphics, 2012, 24(9): 1113-1124.

[30] Gou Liang, Zhang Xiaolong. Treenetviz: revealing patterns of networks over tree structures[J]. IEEE Trans. on Visualization and Computer Graphics, 2011, 17(12): 2449-2458.

[31] Phan D, Lin Xiao, Yeh R, et al. Flow map layout[C]//Andrews K. Proc. of the INFOVIS. Los Alamitos: IEEE Press, 2005: 219-224.

[32] Buchin K, Speckmann B, Verbeek K. Flow map layout via spiral trees[J]. IEEE Trans. on Visualization and Computer Graphics, 2011, 17(12): 2536-2544.

[33] Scheepens R, Willems N, Wetering V D H, et al. Composite density maps for multivariate trajectories[J]. IEEE Trans. on Visualization and Computer Graphics, 2011, 17(12): 2518-2527.

[34] Tobler W. Experiments in migration mapping by computer[J]. The American Cartographer, 1987, 14(2): 155-163.

[35] Peuquet D J, Kraak M J. Geobrowsing: creative thinking and knowledge discovery using geographic visualization[J]. Information Visualization, 2002, 1(1): 80-91.

[36] Ahlberg C, Shneiderman B. Visual information seeking: tight coupling of dynamic query filters with starfield displays[C]//Beth A, Susan D, Judith O. Proc. of the CHI. New York: ACM Press, 1994: 313-317.

[37] Jing Yang, Hubball D, Ward M S, et al. Value and relation display: interactive visual exploration of large data sets with hundreds of dimensions[J]. IEEE Trans. on Visualization and Computer Graphics, 2007, 13(3): 494-507.

[38] Inselberg A, Dimsdale B. Parallel coordinates: a tool for visualizing multi-dimensional geometry[C]//Kaufman A. Proc. of the Visualization. San Francisco: IEEE Press, 1990: 361-378.

[39] 马斌, 周平, 张建业, 等. 大数据时代的数据挖掘[J]. 中国科技信息, 2014(23): 117-118. Ma Bin, Zhou Ping, Zhang Jianye, et al. Data mining in times of big data[J]. China Science and Technology Information, 2014(23): 117-118.

[40] 冯宏亮. 数据挖掘中若干关键算法的研究[D]. 西安: 西安科技大学, 2010. Feng Hongliang. The research on several key algorithms of data mining[D]. Xi'an: Xi'an University of Science and Technology, 2010.

[41] 李平荣. 大数据时代的数据挖掘技术与应用[J]. 重庆三峡学院学报, 2014, 30(151): 45-47. Li Pingrong. Data mining technology and its applications in big data era[J]. Journal of Chongqing Three Gorges University, 2014, 30(151): 45-47.

[42] Yuan Soetsyr, Chen Yenchuan. Semantic ideation learning for agent-based e-brainstorming[J]. IEEE Transactions on Knowledge and Data Engineering, 2008, 20(2): 261-275.

[43] Kimmo L, Anita R, Hannu L. Delphi method analysis: the role of regulation in the mobile operator business in Finland[C]//Technology Management for Global Economic Growth (PICMET), 2010 Proceedings of PICMET'10: IEEE Press, 2010: 1-7.

[44] Edi Ka. Unknowable states and choice-based definitions of subjective probabilities[J]. Economics Letters, 2008, 99: 534-536.

[45] Zhi Jin, Chen Xiaohong, Didar Z. Performing projection in problem frames using scenarios[C]//Proc. 16^th Asia-Pacific Software Engineering Conference: IEEE Press, 2009: 249-256.

[46] Togn H. On a threshold model pattern recognition and signal processing[C]//Amsterdam: Sijthoff and Noordhoff Press, 1978: 18-29.

[47] Chen Guanglei, Wang Zhaojun. The multivariate partially linear model with b-spline[J]. Chinese Journal of Applied Probability, 2010, 26(2): 138-150.

[48] Zhang G P. Time series forecasting using a hybrid ARIMA and neural network mode[J]. Neurocomputing, 2003, 50(5): 159-175.

[49] 秦春秀, 祝婷, 赵捧未, 等. 自然语言语义分析研究进展[J]. 图书情报工作, 2014, 58(22): 130-137. Qin Xiuchun, Zhu Ting, Zhao Pengmo, et al. Research review on semantics analysis of natural language[J]. Library and Information Service, 2014, 58(22): 130-137.

[50] 赵妍妍, 秦兵, 刘挺. 文本情感分析综述[J]. 软件学报, 2010, 21(8): 1834-1848. Zhao Yanyan, Qin Bing, Liu Ting, et al. Sentiment analysis[J]. Journal of Software, 2010, 21(8): 1834-1848.

[51] 唐杰, 梅俏竹. 数据挖掘学科发展研究[A]. 2012—2013控制科学与工程学科发展报. 北京: 中国科学出版社, 2014. Tang Jie, Mei Qiaozhu. Recent advances of data mining in China[A]. 2012—2013 Report on Advances in Control Science and Engineering. Beijing: China Science and Technology Press, 2014.

[52] 韩京宇, 徐立臻, 董逸生. 数据质量研究综述[J]. 计算机科学, 2008, 35(2): 1-5, 12. Han Jingyu, Xu Lizhen, Dong Yisheng. An overview of data quality research[J]. Computer Science, 2008, 35(2): 1-5, 12.

[53] 王宏志. 大数据质量管理: 问题与研究进展[J]. 科技导报, 2014, 32(34): 78-84. Wang Hongzhi. Big data quality management: problems and progress[J]. Science and Technology Review, 2014, 32(34): 78-84.

[54] 韩京宇, 徐立臻, 董逸生. 一种大数据量的相似记录检测方法[J]. 计算机研究与发展, 2005, 42(12): 2206-2212. Han Jingyu, Xu Lizhen, Dong Yisheng. An approach for detecting similar duplicate records of massive data[J]. Journal of Computer Research and Development, 2005, 42(12): 2206-2212.

[55] 邱越峰, 田增平. 一种高效的识别相似重复记录的方法[J]. 计算机学报, 2001, 24(1): 69-77. Qiu Yuefeng, Tian Zengping. An efficient approach for detecting approximately duplicate database records[J]. Chinese Journal of Computers, 2001, 24(1): 69-77.

[56] Chaud H S, Ganjam K, Ganti V, et al. Robust and efficient fuzzy match for on line data cleaning[C]//Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data. San Diego: ACM Press, 2003: 313-324.

[57] Grzymala B J, Hu Ming. A comparison of several approaches to missing attribute values in data mining[C]//Proceedings of the Second International Conference on Rough Sets and Current Trends in Computing. Banff: Springer Berlin Heidalberg, 2001: 378-385.

[58] Bhattacharya I, Getoor L. Iterative record linkage for cleaning and integration[C]//Proceedings of the 9^th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery. Maison de la Chimie: ACM Press, 2004: 11-18.

[59] Rohrs C E, Berry R A. A linear control approach to explicit rate feedback in ATM networks[C]//Proc. of IEEE in Focom'97. Kode: IEEE Press, 1997: 277-282.

[60] Pluempitiwiriyawej C, Hammer J. Element matching across data-oriented XML sources using a multi-strategy clustering model[J]. Data and Knowledge Engineering, 2004, 48(3): 297-333.

[61] Zhao Li, Yuan Sungsam, Yang Qixiao, et al. Dynamic similarity for fields with null values[C]//Proceedings of the 4^th International Conference on Data Warehousing and Knowledge Discovery. France: Springer Berlin Heidelberg, 2002: 161-169.

[62] Wu Xintao, Barbara D. Modeling and imputation of large incomplete multidimensional data sets[C]//Proceedings of the International Conference on Data Warehousing and Knowledge Discovery. Springer Berlin Heidelberg, 2002: 286-295.

[63] Neal R M. Probabilistic inference using Markov chain Monte Carlo methods[M]. CRG TR-93-1. Department of Computer Science, University of Toronto, 1993.

[64] Hipp J, Guntzer U, Grimmer U. Data quality mining: making a virtue of necessity[C]//In Proc. of the 6^th ACM SIGMOD Workshop on Research Issues in Data Mining and Knowledge Discovery. New York: ACM Press, 2001: 52-57.

[65] Lueebber D, Grimmer U. Systematic development of data miningbased data quality tools[C]//29^th VLDB. Berlin: Morgan Kaufmann, 2003: 548-559.

[66] Dasu T, Johnson T. Exploratory data mining and data cleaning[M]. Hoboken: John Wiley and Sons, Inc, 2003.

[67] Fellegi I P, Holt D. A systematic approach to automatice dit and imputation[J]. Americ Statistics of Association, 1976, 71(353): 17-35.

[68] Fan Weigui, Lu Hongjun, Madnick S E, et al. Discovering and reconciling value conflicts for numerical data integration[J]. Information Systems, 2001(26): 635-656.

[69] Motro A, Anokhin P, Acar A C. Utility-based resolution of datainconsistencies[C]// IQIS 2004: 35-43.

[70] Kaldewey T, Shekita E J, Tata S. Clydesdale: structured data processing on MapReduce[C]//Rundensteiner E A, Markl V, Manolescu I, et al. Proc. of the EDBT 2012. Berlin: ACM Press, 2012: 15.

[71] Jindal A, Quiane-Ruiz J A, Dittrich J. Trojan data layouts: right shoes for a running elephant[C]//Chase J S, Abbadi A E, Babu S, et al. Proc. of the SOCC. Cascais: ACM Press, 2011.

[72] Neumeyer L, Robbins B, Nair A, et al. S4: distributed stream computing platform[C]//Fan W, Hsu W, Webb G I, et al. Proc. of the ICDM Workshops 2010. Sydney: IEEE Computer Society, 2010: 170-177.

[73] Afrati F N, Ullman J D. Optimizing joins in a Map Reduce environment[C]//Manolescu I, Spaccapietra S, Teubner J, et al. Proc. of the EDBT 2010. Lausanne: ACM Internatinoal Conf. Proc. Series, 2010: 99-110.

[74] Okcan A, Riedewald M. Processing theta-joins using MapReduce[C]//Sellis T K, Miller R J, Kementsietsidis A, et al. Proc. of the SIGMOD 2011. Athens: ACM Press, 2011: 949-960.

[75] Nykiel T, Potamias M, Mishra C, et al. MRShare: sharing across multiple queries in MapReduce[J]. Proc. of the VLDB Endowment, 2010, 3(1-2): 494-505.

[76] Roy I, Ramadan H E, Setty S T V, et al. Airavat: security and privacy for MapReduce[C]//Castro M, Snoeren A C. Proc. of the NSDI 2010. San Jose: USENIX Association, 2010: 297-312.

[77] Willis L, Jignesh M P. Energy management for MapReduce clusters[J]. Proc. of the VLDB Endowment, 2010, 3(1-2): 129-139.

[78] 2015年度大数据发展的十大趋势[J]. 中国有线电视, 2015(1): 100-101. Trends of big data development in 2015[J]. China Digital Cable TV, 2015(1): 100-101.

Analytics Towards Big Data

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	CHANG Xiao, HUANG Zhibin, YU Min, YANG Wubing. A Deep Decision Tree Model for Aerospace Big Data [J]. Journal of Beijing University of Posts and Telecommunications, 2023, 46(3): 1-6.
[2]	YAO Jun-liang, XUE Hai-tao, LIU Qing. Daily Water Volume Prediction Algorithm of Urban Smart Water Based on Big Data [J]. Journal of Beijing University of Posts and Telecommunications, 2021, 44(4): 82-88.
[3]	XIANG Ying-zhuo, WEI Qiang, YOU Ling. An Information Diffusion Pattern Mining Method Based on Communication Actions [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2019, 42(3): 83-90.
[4]	YI Zhi-ling, CUI Chun-feng, HAN Shuang-feng, PAN Cheng-kang, CHEN Ya-mi. Analysis of Key Technologies of 5G-Oriented Cellular Internet of Things [J]. Journal of Beijing University of Posts and Telecommunications, 2018, 41(5): 20-25.
[5]	ZHANG Si-hai, ZHANG Jian-hua, CHEN Ying, ZHU Jin-kang. Wireless Big Data Enabled Emerging Technologies for Beyond 5G System [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2018, 41(5): 52-61.
[6]	LI Fang-wei, NIE Yi-fang, LIU Fan, ZHU Jiang. The Mechanism of Situation Data Fusion Based on Trust in Wireless Sensor Network [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2015, 38(5): 71-76.
[7]	MENG Xiang-wu, JI Wei-yu, ZHANG Yu-jie. A Survey of Recommendation Systems in Big Data [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2015, 38(2): 1-15.
[8]	WANG Xiao-jun. Employing Item Attribute and Preference to Enhance the Collaborative Filtering Recommendation [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2014, 37(6): 68-71,76.
[9]	FU Yu, YU Yan-hua, SONG Mei-na, ZHAN Xiao-su. A Closed Sequential Pattern Mining Algorithm in Time Order [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2013, 36(4): 19-22.
[10]	XIONG Wen, JIN Yao-hong. Hybrid Clustering Using ACO and AHC [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2013, 36(3): 60-63,78.
[11]	WANG Zhong-Feng, WANG Zhi-Hai. Equivalent Classed of TAN Classifier Structure and Their Application on Learning Algorithm [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2012, 35(1): 72-76.
[12]	. An Approach on Association Patterns Mining in Telecommunication Alarm Database [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2011, 34(2): 85-89.
[13]	. 〗Design and Performance Analysis of a Parallel Decision Tree Algorithm on Data Mining Grid [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2009, 32(s1): 49-52.
[14]	XU Qianfang1 ， XIAO Bo2 ， GUO Jun1. An Mining Algorithm with Alarm Association Rules Based on Statistical Correlation [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2007, 30(1): 66-70.
[15]	GUO Yan-hui1,2, WANG Cong1, ZHONG Yi-xin1 . Language Cognition and Topic Identification [J]. JOURNAL OF BEIJING UNIVERSITY OF POSTS AND TELECOM, 2006, 29(s2): 88-92.