Metadata-Driven Self-Optimizing ETL Framework with Machine Learning-Based Workload Prediction

In contemporary data-driven systems, data processing pipelines need to be efficient and adaptive and capable of supporting dynamically changing workloads. Traditional Extract Transform Load (ETL) systems are normally configured to use fixed parameters, which results in underutilization of resources and longer execution time. The paper suggests implementing a metadata-based self-optimizing ETL pipeline, which incorporates the machine learning-based workload forecasting to maximize the performance and scalability. The proposed system will use historical metadata and anticipate the future workloads to optimize the scheduling and resource allocation processes dynamically. A feedback-based system is added to enable constant enhancement of prediction quality and efficiency of the system. Through testing on synthetic data, it is shown that the proposed model outperforms the current models with respect to execution time, resource usage, throughput, latency and prediction accuracy. The findings reveal the usefulness of metadata-based decision-making as a predictive modeling approach in next-generation ETL optimization.