IDEAS home Printed from https://ideas.repec.org/a/apb/jaterr/2017p108-116.html

Analysis of real-time data with spark streaming

Author

Listed:
  • Nikitha Johnsirani Venkatesan

    (School of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea)

  • Choon Sung Nam

    (School of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea)

  • Earl Kim

    (School of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea)

  • Dong Ryeol Shin

    (School of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, South Korea)

Abstract

Data analysis in real-world application domains is a very challenging issue. For example, Thousand Gigabytes of multimedia data gets poured into Social media each and every minute. Since social media and most of the organizations are dealing with Big Data, tools like Hadoop and Spark system is more appropriate for dealing with those data. Hadoop and Map Reduce analyze the data only in batch mode. This makes it difficult for the real-time analysis because it increases latency. In order to solve the above problem, we used Spark streaming to do real-time data analysis. Spark streaming helps to iterate through the data much faster due to its in-memory processing. This paper presents an online machine learning system for real-time data. Using Spark streaming, data from online messaging system is streamed into the local system. Streaming K-means algorithm is applied to cluster the different languages of the people from various countries. Results show that predictions of the incoming data is accurate and fast the when Apache spark is used. Our results and methods are compared with other articles which have used spark streaming for real-time data processing. Queries like total word count and segregation based on keywords are done and the results are presented. The data are then stored in the local disk for future querying process.

Suggested Citation

  • Nikitha Johnsirani Venkatesan & Choon Sung Nam & Earl Kim & Dong Ryeol Shin, 2017. "Analysis of real-time data with spark streaming," Journal of Advances in Technology and Engineering Research, A/Professor Akbar A. Khatibi, vol. 3(4), pages 108-116.
    • Handle: RePEc:apb:jaterr:2017:p:108-116
    DOI: 10.20474/jater-3.4.1
    as

    Download full text from publisher

    File URL: https://tafpublications.com/platform/Articles/full-jater3.4.1.php
    Download Restriction: no
    File URL: https://tafpublications.com/gip_content/paper/jater-3.4.1.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.20474/jater-3.4.1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Feng-Jen Yang, 2016. "The User Interface Design of an Intelligent Tutoring System for Relational Database Schema Normalization," International Journal of Technology and Engineering Studies, PROF.IR.DR.Mohid Jailani Mohd Nor, vol. 2(3), pages 70-75.
    2. J. A. Hartigan & M. A. Wong, 1979. "A K‐Means Clustering Algorithm," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 28(1), pages 100-108, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yuan-Hsun Liao & Hsiao-Hui Li & Zong-Han Xie, 2019. "Study of Virtual Reality and IoT for Exploring the Deep Sea," Journal of ICT, Design, Engineering and Technological Science, Juhriyansyah Dalle, vol. 3(1), pages 11-14.
    2. Mohd Hilmi Othman & Arif Izzudin Muhammad & Sulaiman Hassan, 2018. "Investigation of Clothes Recycling as Colouring Agent for Polypropylene-Nanoclay Nanocomposites," International Journal of Technology and Engineering Studies, PROF.IR.DR.Mohid Jailani Mohd Nor, vol. 4(1), pages 1-6.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. HaiYun Wang & JianPing Zhao & ChunHou Zheng & YanSen Su, 2022. "scDSSC: Deep Sparse Subspace Clustering for scRNA-seq Data," PLOS Computational Biology, Public Library of Science, vol. 18(12), pages 1-18, December.
    2. Custodio João, Igor & Lucas, André & Schaumburg, Julia & Schwaab, Bernd, 2023. "Dynamic clustering of multivariate panel data," Journal of Econometrics, Elsevier, vol. 237(2).
    3. Xu, Jing & Wang, Xiaoying & Gu, Yujiong & Ma, Suxia, 2023. "A data-based day-ahead scheduling optimization approach for regional integrated energy systems with varying operating conditions," Energy, Elsevier, vol. 283(C).
    4. Carlos Carrasco-Farré, 2022. "The fingerprints of misinformation: how deceptive content differs from reliable sources in terms of cognitive effort and appeal to emotions," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 9(1), pages 1-18, December.
    5. Roberto Benocci & Giovanni Brambilla & Alessandro Bisceglie & Giovanni Zambon, 2020. "Eco-Acoustic Indices to Evaluate Soundscape Degradation Due to Human Intrusion," Sustainability, MDPI, vol. 12(24), pages 1-19, December.
    6. Felix Mbuga & Cristina Tortora, 2021. "Spectral Clustering of Mixed-Type Data," Stats, MDPI, vol. 5(1), pages 1-11, December.
    7. Emma L. Schultz & David T. Tan & Kathleen D. Walsh, 2010. "Endogeneity and the corporate governance - performance relation," Australian Journal of Management, Australian School of Business, vol. 35(2), pages 145-163, August.
    8. Muxuan Pan & Hao Wang & Jinquan Huang, 2019. "T–S Fuzzy Modeling for Aircraft Engines: The Clustering and Identification Approach," Energies, MDPI, vol. 12(17), pages 1-15, August.
    9. Kolos Cs. Ágoston & Marianna E.-Nagy, 2024. "Mixed integer linear programming formulation for K-means clustering problem," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 32(1), pages 11-27, March.
    10. Zhang, Weibin & Zha, Huazhu & Zhang, Shuai & Ma, Lei, 2023. "Road section traffic flow prediction method based on the traffic factor state network," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 618(C).
    11. Ting Liu & Nick Shryane & Mark Elliot, 2022. "Attitudes to climate change risk: classification of and transitions in the UK population between 2012 and 2020," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 9(1), pages 1-15, December.
    12. Renata De Paris & Christian V Quevedo & Duncan D A Ruiz & Osmar Norberto de Souza, 2015. "An Effective Approach for Clustering InhA Molecular Dynamics Trajectory Using Substrate-Binding Cavity Features," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-25, July.
    13. Michal Bernardelli & Zbigniew Korzeb & Pawel Niedziolka, 2021. "The banking sector as the absorber of the COVID-19 crisis’ economic consequences: perception of WSE investors," Oeconomia Copernicana, Institute of Economic Research, vol. 12(2), pages 335-374, June.
    14. Jelle R Dalenberg & Luca Nanetti & Remco J Renken & René A de Wijk & Gert J ter Horst, 2014. "Dealing with Consumer Differences in Liking during Repeated Exposure to Food; Typical Dynamics in Rating Behavior," PLOS ONE, Public Library of Science, vol. 9(3), pages 1-11, March.
    15. Abdur Rahman M. A. Basher & Caleb Hallinan & Kwonmoo Lee, 2025. "Heterogeneity-preserving discriminative feature selection for disease-specific subtype discovery," Nature Communications, Nature, vol. 16(1), pages 1-25, December.
    16. Muhannad Mohammed Alfehaid, 2025. "Integrating Hard and Green Infrastructure for Sustainable Tourism: A Spatial Analysis of Saudi Regions," Sustainability, MDPI, vol. 17(20), pages 1-15, October.
    17. Carlos Fernández-Hernández & Carmelo J. León & Jorge E. Araña & Flora Díaz-Pére, 2016. "Market segmentation, activities and environmental behaviour in rural tourism," Tourism Economics, , vol. 22(5), pages 1033-1054, October.
    18. Wubishet A. Bekele & Raz Avni & Clayton L. Birkett & Asuka Itaya & Charlene P. Wight & Justin Bellavance & Sophie Brodführer & Francisco J. Canales & Craig H. Carlson & Anne Fiebig & Yongle Li & Steve, 2025. "Global genomic population structure of wild and cultivated oat reveals signatures of chromosome rearrangements," Nature Communications, Nature, vol. 16(1), pages 1-14, December.
    19. Haiyang Xia & Song Zha & Jijun Huang & Jibin Liu, 2020. "Radio environment map construction by adaptive ordinary Kriging algorithm based on affinity propagation clustering," International Journal of Distributed Sensor Networks, , vol. 16(5), pages 15501477209, May.
    20. Juvie Pauline L. Relacion, 2017. "Patient Management Information System for the University of the Immaculate Conception College Department Clinic," International Journal of Technology and Engineering Studies, PROF.IR.DR.Mohid Jailani Mohd Nor, vol. 3(5), pages 213-223.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:apb:jaterr:2017:p:108-116. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: A/Professor Akbar A. Khatibi (email available below). General contact details of provider: https://tafpublications.com/platform/published_papers/10 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.