For bounded datasets such as the TRECWeb Track (WT10g) the computation of term frequency (TF) and inverse document frequency (IDF) is not difficult. However, when the corpus is the entire web, direct IDF calculation is impossible and values must instead be estimated. Most available datasets provide values for term count (TC) meaning the number of times a certain term occurs in the entire corpus. Intuitively this value is different from document frequency (DF), the number of documents (e.g., web pages) a certain term occurs in. We investigate the relationship between TC and DF values of terms occurring in theWeb as Corpus (WaC) and also the similarity between TC values obtained from theWaC and the Google N-gram dataset. A strong correlation between the two would gives us confidence in using the Google N-grams to estimate accurate IDF values which for example is the foundation to generate well performing lexical signatures based on the TF-IDF scheme. Our results show a very strong correlation between TC and DF within theWaC with Spearman's ρ = 0.8 (p ≤ 2.2×10 -16) and a high similarity between TC values from theWaC and the Google N-grams. © Springer-Verlag Berlin Heidelberg 2009.
CITATION STYLE
Klein, M., & Nelson, M. L. (2009). Correlation of term count and document frequency for google n-grams. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 5478 LNCS, pp. 620–627). https://doi.org/10.1007/978-3-642-00958-7_58
Mendeley helps you to discover research relevant for your work.