Intelligent Decision Making Based on Data Mining using Differential Evolution Algorithms and Framework for ETL Workflow Management

by Jungh K
For this week’s blog assignment, I chose an article, titled “Intelligence Decision Making Based on Data Mining using Differential Evolution Algorithms and Framework for ETL Workflow Management”.   The authors propose an integrated DSS, which utilizes a data mining technique and a framework for effective ETL workflows.  The specific data mining technique proposed the authors is to add a specialized component, known as the Artificial Intelligence Component (AIC), to business intelligence system.  The AIC utilizes Differential Evolution Algorithms, which replace an option for the current situation to an optimized option, if one exists.  Through this procedure, the authors argue that the DEA will adapt itself to improve the intelligence decision making process with the passages of time.  On top of the data mining discussed in the article, the authors propose to add two layers, application and workflow scheduling, to workflow management.  The application layer receives ETL jobs directly from the data generator.  The authors state that they are numbers of considerations, which must be taken into, for ETL processes.  The considerations include source availability, target availability, priority, job duration, upper bound, required resources, and prerequisite jobs.  The workflow management layer is divided into two parts:  workflow scheduling and workflow execution.  By incorporating the aforementioned considerations for ETL processes, workflow scheduling layer utilizes various algorithms to optimize scheduling.  The work execution layer tracks different ETL jobs and distributes throughout available servers.

This class lecture topics include data quality and integration, which involve ETL processes.  As data are gathered from different sources to handle the big data, a lot of factors can affect the data quality.  The methods proposed by the authors can help reduce incomplete data by utilizing artificial intelligence.  However, the authors do not provide examples on how artificial intelligence would be specifically used in a real life situation.  Nevertheless, if artificial intelligence can correctly complete any incomplete, it would save DBAs and analysts a lot of headaches.  One noticeable trend I have seen is that many of corporations starts to utilize big data to verify one’s personal information.  For instance, when I tried to join UPS or Square website, I just had to provide basic information and the rest of them were automatically pulled from their server once my identity was authenticated.

Shaikh, M.U.; Malik, S.U.R.; Qureshi, M.A.; Yaqoob, S.; , “Intelligent Decision Making Based on Data Mining Using Differential Evolution Algorithms and Framework for ETL Workflow Management,” Computer Engineering and Applications (ICCEA), 2010 Second International Conference on , vol.1, no., pp.22-26, 19-21 March 2010
doi: 10.1109/ICCEA.2010.12