Fast co-occurrence thesaurus construction for Chinese news

This paper reports an approach to automatic thesaurus construction for Chinese news articles. An effective Chinese word segmentation and keyword extraction algorithm is first presented. For each document, an average of 33% keywords unknown to a lexicon of 123,226 terms can be identified. The extraction error rate is 3.6%. Keywords extracted from each document are then further filtered for term association analysis by a modified Dice coefficient formula. Association weights larger than a threshold are then accumulated over all the documents to yield the final term pair similarities. Compared to previous studies, this method not only speeds up the thesaurus generation process drastically, but also achieves a similar percentage level of term relatedness.