Vol. 3 No. 2 (2023): Journal of AI-Assisted Scientific Discovery
Articles

Real-time Analytics: Implementing ML algorithms to analyze data streams in real-time

Muneer Ahmed Salamkar
Senior Associate at JP Morgan Chase, USA
Cover

Published 05-09-2023

Keywords

  • machine learning,
  • data streams

How to Cite

[1]
Muneer Ahmed Salamkar, “Real-time Analytics: Implementing ML algorithms to analyze data streams in real-time”, Journal of AI-Assisted Scientific Discovery, vol. 3, no. 2, pp. 587–612, Sep. 2023, Accessed: Dec. 24, 2024. [Online]. Available: https://scienceacadpress.com/index.php/jaasd/article/view/223

Abstract

Real-time analytics has become a cornerstone of modern data-driven decision-making, enabling businesses to extract actionable insights from data as it flows. Implementing machine learning (ML) algorithms for analyzing data streams in real time transforms how organizations respond to critical events, offering unparalleled speed and accuracy. This approach involves leveraging advanced ML models that can process, analyze, and derive insights from continuous data streams, such as customer interactions, financial transactions, or IoT sensor data, without latency. Key challenges include: Handling high-velocity data, Ensuring system scalability & Addressing issues like data noise and missing values in real time. Solutions like distributed computing frameworks, event-driven architectures, and specialized ML algorithms, like online learning and incremental models, have emerged to meet these demands. By integrating real-time analytics with ML, businesses can unlock opportunities like fraud detection, personalized recommendations, and operational efficiency improvements. This shift enhances responsiveness and helps organizations predict and prevent potential issues before they escalate. The implementation process involves deploying ML pipelines capable of handling dynamic data inputs, optimizing algorithms for streaming data, and ensuring robust system reliability. With use cases spanning e-commerce, healthcare, finance, and beyond, real-time ML analytics reshapes industries by bridging the gap between data collection and decision-making. As organizations continue to prioritize real-time capabilities, the convergence of ML and stream processing offers transformative potential for businesses striving to maintain a competitive edge in today’s fast-paced landscape.

Downloads

Download data is not yet available.

References

  1. Boppiniti, S. T. (2021). Real-time data analytics with ai: Leveraging stream processing for dynamic decision support. International Journal of Management Education for Sustainable Development, 4(4).
  2. Gayam, S. R., Yellu, R. R., & Thuniki, P. (2021). Artificial Intelligence for Real-Time Predictive Analytics: Advanced Algorithms and Applications in Dynamic Data Environments. Distributed Learning and Broad Applications in Scientific Research, 7, 18-37.
  3. Pattyam, S. P. (2019). Advanced AI Algorithms for Predictive Analytics: Techniques and Applications in Real-Time Data Processing and Decision Making. Distributed Learning and Broad Applications in Scientific Research, 5, 359-384.
  4. Mohammadi, M., Al-Fuqaha, A., Sorour, S., & Guizani, M. (2018). Deep learning for IoT big data and streaming analytics: A survey. IEEE Communications Surveys & Tutorials, 20(4), 2923-2960.
  5. Deekshith, A. (2019). Integrating AI and Data Engineering: Building Robust Pipelines for Real-Time Data Analytics. International Journal of Sustainable Development in Computing Science, 1(3), 1-35.
  6. Verma, S., Kawamoto, Y., Fadlullah, Z. M., Nishiyama, H., & Kato, N. (2017). A survey on network methodologies for real-time analytics of massive IoT data and open research issues. IEEE Communications Surveys & Tutorials, 19(3), 1457-1477.
  7. Syafrudin, M., Alfian, G., Fitriyani, N. L., & Rhee, J. (2018). Performance analysis of IoT-based sensor, big data processing, and machine learning model for real-time monitoring system in automotive manufacturing. Sensors, 18(9), 2946.
  8. D'Andrea, E., Ducange, P., Lazzerini, B., & Marcelloni, F. (2015). Real-time detection of traffic from twitter stream analysis. IEEE transactions on intelligent transportation systems, 16(4), 2269-2283.
  9. Ranjan, R. (2014). Streaming big data processing in datacenter clouds. IEEE cloud computing, 1(01), 78-83.
  10. Rathore, M. M., Paul, A., Hong, W. H., Seo, H., Awan, I., & Saeed, S. (2018). Exploiting IoT and big data analytics: Defining smart digital city using real-time urban data. Sustainable cities and society, 40, 600-610.
  11. Bańbura, M., Giannone, D., Modugno, M., & Reichlin, L. (2013). Now-casting and the real-time data flow. In Handbook of economic forecasting (Vol. 2, pp. 195-237). Elsevier.
  12. Gaber, M. M., Zaslavsky, A., & Krishnaswamy, S. (2005). Mining data streams: a review. ACM Sigmod Record, 34(2), 18-26.
  13. Sahal, R., Breslin, J. G., & Ali, M. I. (2020). Big data and stream processing platforms for Industry 4.0 requirements mapping for a predictive maintenance use case. Journal of manufacturing systems, 54, 138-151.
  14. KATRAGADDA, V. (2022). Dynamic Customer Segmentation: Using Machine Learning to Identify and Address Diverse Customer Needs in Real-Time. IRE Journals, 5(10), 278-279.
  15. Poria, S., Cambria, E., Gelbukh, A., Bisio, F., & Hussain, A. (2015). Sentiment data flow analysis by means of dynamic linguistic patterns. IEEE Computational Intelligence Magazine, 10(4), 26-36.
  16. Thumburu, S. K. R. (2022). A Framework for Seamless EDI Migrations to the Cloud: Best Practices and Challenges. Innovative Engineering Sciences Journal, 2(1).
  17. Thumburu, S. K. R. (2022). Real-Time Data Transformation in EDI Architectures. Innovative Engineering Sciences Journal, 2(1).
  18. Gade, K. R. (2021). Cost Optimization Strategies for Cloud Migrations. MZ Computing Journal, 2(2).
  19. Gade, K. R. (2020). Data Analytics: Data Privacy, Data Ethics, Data Monetization. MZ Computing Journal, 1(1).
  20. Katari, A., Ankam, M., & Shankar, R. Data Versioning and Time Travel In Delta Lake for Financial Services: Use Cases and Implementation.
  21. Katari, A., Muthsyala, A., & Allam, H. HYBRID CLOUD ARCHITECTURES FOR FINANCIAL DATA LAKES: DESIGN PATTERNS AND USE CASES.
  22. Gade, K. R. (2018). Real-Time Analytics: Challenges and Opportunities. Innovative Computer Sciences Journal, 4(1).
  23. Gade, K. R. (2017). Integrations: ETL vs. ELT: Comparative analysis and best practices. Innovative Computer Sciences Journal, 3(1).
  24. Thumburu, S. K. R. (2021). Optimizing Data Transformation in EDI Workflows. Innovative Computer Sciences Journal, 7(1).
  25. Thumburu, S. K. R. (2020). Interfacing Legacy Systems with Modern EDI Solutions: Strategies and Techniques. MZ Computing Journal, 1(1).