Study of various methods for tokenization

Abstract

Tokenization is the mechanism of splitting or fragmenting the sentences and words to its possible smallest morpheme called as token. Morpheme is smallest possible word after which it cannot be broken further. As the tokenization is initial phase and as well very crucial phase of Part-Of-Speech (POS) tagging in Natural Language Processing (NLP). Tokenization could be sentence level and word level. This paper analyzes the possible tokenization methods that can be applied to tokenize the word efficiently.