Explicit Use of Term Occurrence Probabilities for Term Weighting in Text Categorization

Abstract

In this paper, the behaviors of leading symmetric and asymmetric term weighting schemes are analyzed in the context of text categorization. This analysis includes their weighting patterns in the two dimensional term occurrence probability space and the dynamic ranges of the generated weights. Additionally, one of the newly proposed term selection schemes, multi-class odds ratio, is considered as a potential symmetric weighting scheme. Based on the findings of this study, a novel symmetric weighting scheme derived as a function of term occurrence probabilities is proposed. The experiments conducted on Reuters-21578 ModApte Top10, WebKB, 7-Sectors and CSTR2009 datasets indicate that the proposed scheme outperforms other leading schemes in terms of macro-averaged and micro-averaged F-1 scores.