Use this URL to cite or link to this record in EThOS: http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.485828
Title: Adaptive Models of Chinese Text
Author: Wu, Peiliang
Awarding Body: University of Wales, Bangor
Current Institution: Bangor University
Date of Award: 2007
Availability of Full Text:
Access through EThOS:
Abstract:
This thesis is concerned with how to build adaptive language models of Chinese text, which can be used in different Chinese natural language processing (NLP) applications. The Prediction by Partial Matching (PPM) language model has been widely used in many NLP areas. To apply this model for Chinese, many problems arise that originate from the language's large alphabet. In this thesis, the PPM-ch model is introduced first to improve the traditional PPM model, by first using preprocessing techniques, then a frequency sorting technique and a variation of PPM that performs no exclusions. PPMO, a novel variant of the PPM model is then proposed. Unlike traditional PPM models, which output an escape symbol when a novel symbol occurs in a context model, PPMO separates the coding process into two streams, named the orders stream and the symbols stream. This algorithm is the first PPM variant that does not use the escape mechanism and it achieves the best compression results. We have also investigated Chinese Word segmentation by using our PPM-ch and PPMO model. Although our PPMO models havenot been carefully crafted for segmentation usage, we still achieve satisfactory results.
Supervisor: Not available Sponsor: Not available
Qualification Name: University of Wales, Bangor, 2007 Qualification Level: Doctoral
EThOS ID: uk.bl.ethos.485828  DOI: Not available
Share: