Effectively classifying short texts via improved lexical category and semantic features

Abstract

Classification of short text is challenging due to its severe sparseness and high dimension, which are typical characteristics of short text. In this paper, we propose a novel approach to classify short texts based on both lexical and semantic features. Firstly, the term dictionary is constructed by selecting lexical features that are most representative words of a certain category, and then the optimal topic distribution from the background knowledge repository is extracted via Latent Dirichlet Allocation. The new feature for short text is thereafter constructed. The experimental results show that our method achieved significant quality enhancement in terms of short text classification.