Semi-Supervised Self-Training Approach for Web Robots Activity Detection in Weblog

Abstract

Due to the significant added value of web servers, they are vulnerable to attacks, so web security has received a lot of attention. Web server logging systems that record each user request performed by users have become an important data analysis object in web security. Traditionally, system experts’ analyses log data manually using keyword searches and regular expressions. However, the amount of log data and attack types makes routine detection ineffective. Machine learning-based supervised and unsupervised detection approaches have been employed extensively during the last decade to improve traditional detection methods. The proposed semi-supervised STBOOST web robot detection system uses self-training with XGBoost as its base classifier. Experimental data are taken from the open-source data repository, the NASA 95 dataset, and e-commerce site access logs. In both datasets, self-training XGBoost outperforms XGBoost and can detect anonymous web robots using unlabeled data.