This technology converts running text into tokens according to language-specific rules. The output is in the format of one token per line. Punctuation is removed from words and appears on new lines.