E-ETL framework: ETL process reparation algorithms using case-based reasoning

Abstract

External data sources (EDSs) being integrated in a data warehouse (DW) frequently change their structures/schemas. As a consequence, in many cases, an already deployed ETL workflow stops its execution, yielding errors. Since structural changes of EDSs are frequent, an automatic reparation of an ETL workflow after such a change is of high importance. In this paper we present a framework, called E-ETL, for handling the evolution of an ETL layer. In the framework, an ETL workflow is semi-automatically or automatically (depending on a case) repaired as the result of structural changes in data sources, so that it works with the changed data sources. E-ETL supports three different reparation methods, but in this paper we discuss the one that is based on case-based reasoning. The proposed framework is being developed as a module external to an ETL engine, so that it can work with any engine that supports API for manipulating ETL workloads.