Novelty framework for knowledge discovery in databases

Abstract

Knowledge Discovery in Databases (KDD) is an iterative process that aims at extracting interesting, previously unknown and hidden patterns from huge databases. Use of objective measures of interestingness in popular data mining algorithms often leads to another data mining problem, although of reduced complexity. The reduction in the volume of the discovered rules is desirable in order to improve the efficiency of the overall KDD process. Subjective measures of interestingness are required to achieve this. In this paper we study novelty of the discovered rules as a subjective measure of interestingness. We propose a framework to quantify novelty of the discovered rules in terms of their deviations from the known rules. The computations are carried out using the importance that the user gives to different deviations. The computed degree of novelty is then compared with the user given threshold to report novel rules to the user. We implement the proposed framework and experiment with some public datasets. The experimental results are quite promising. © Springer-Verlag Berlin Heidelberg 2004.