A New Method to Measure Similarity of Words in Japanese Twitter Based on Related Images

Abstract

Twitter, as a popular form of social media in Japan, has emerged as a valuable data resource for various important social network analysis tasks. However, Japanese tweets often contain nonstandard words and variant notations, owing to which several words with the same meaning may be written differently. The use of such words will generate the sparsity problem and decrease the accuracy of similarity measures between users. Furthermore, the performance of user or tweet recommendations may be deteriorated. Therefore, words with the same meaning must be unified in the preprocessing step. In this research, assuming that words with the same meaning have similar or common related images, we propose a method to use word-related images to measure the similarity between words. A manually annotated Japanese data set is created to evaluate the proposed method. Experimental results indicate that the proposed method outperforms the existing methods in most cases.