<< Click to Display Table of Contents >> Navigation:  General Functions > Configuring the MIE (Programmer's Guide) > Alternate Language Support > Tokenizer > Token

Tokens are created for extracted text and then inserted back in place of the text for which they were created.  The token itself still contains the extracted text, however that text will be replaced at run time from the text that is contained in the Alt Language Lines table (including English).  Therefore, the text remains with the token just to maintain readability of the original text.  However, the text that will display will not come from the token itself, but rather from the table that contains the various translations of the token's text (again, English included).

A token looks like this:

~|<TextLangLineNo>-<CRC32>|<text>|~

 

The start delimiter of the token is ~| and the end delimiter is |~

<TextLangLineNo> is the primary key value, in the Alt Language Lines table of the English version of the extracted text.

<CRC32> is a Cyclical Redundancy Code (CRC) value that is calculated by the Tokenizer when the token is created.  A CRC is unique based on the text and the order of the characters that form the text.

<text> is the actual text that was extracted for tokenization.  This text will be in English.