Combining AI with Data Engineering Pipelines: Improving Real-Time Decision-Making Systems
International Journal of Emerging Trends in Science and Technology,
,
14 April 2022
,
Page 01-24
https://doi.org/10.18535/ijetst/v9i04.04
AI with data engineering pipelines as an innovative way has become relevant in the context of optimizing the real-time decision-making systems. The objective of this research is to explore the potential of AI models in making decision support more efficient when integrated into today’s data science platforms. The overarching goal is to provide an ability to fill the gap between data collection, analysis and decision-making through use of intelligent systems, which adapt to environments and continually learn.
To this end, the study uses a mix of architectural assessment, a prototypical implementation, and case studies. State-of-art technologies like machine learning and deep learning are combined with data engineering technologies like Apache Kafka, Spark, Kubernetes for real time data processing and management. It involves emulating high velocity data streams and applying AI’s analytics to it what would be accomplished in real time.
The results show increased adaptability, speed and effectiveness in decision-making systems. By directly integrating AI into an organization’s data streams, it is possible to decrease the time taken to process the data by half, and get a more accurate analysis by 30%. Real-world applications of the technique involve fraud detection in financial services and improved supply chain management; performance of the method in larger organizational contexts is also demonstrated.
The implication from this research is that integration of AI with data engineering pipelines is not only possible but imperative especially to organizations that are willing to step up their game and competitiveness in the current era characterised by an influx in data generation. Subsequent work will investigate how new technologies like federated learning and edge computing can be applied progressively to boost the effectiveness of real-time decision making frameworks.