Tokenizer

Methods for handling token stream.

Method	Description	Available from version
createFromString	Converts text into tokens, creating a new token stream object.
createFromTokens	Creates new token stream objects and sets new tokens.
endOfInput	Checks if token list end has been reached.
getCurrentToken	Returns current token without changing current position.
getTokens	Returns all tokens.
nextToken	Returns next token in the list.
putBack	Moves current position to a step back.
resetState	Resets internal state.
setBookmark	Registers and bookmarks current position.
restoreBookmark	Restores previously bookmarked position.
skipWhiteSpace	Skips all space characters and comments.
testText	Compares text in the parameter `$text` with text contained in the token.
testUpperText	Compares text in the parameter `$text` with text contained in the token.