reorganize module

_archive/rhai_engine/rhaibook/engine/token-mapper.md

@@ -0,0 +1,79 @@
+Remap Tokens During Parsing
+===========================
+
+{{#include ../links.md}}
+
+[`Engine::on_parse_token`]: https://docs.rs/rhai/{{version}}/rhai/struct.Engine.html#method.on_parse_token
+[`Token`]: https://docs.rs/rhai/{{version}}/rhai/enum.Token.html
+
+
+The Rhai [`Engine`] first parses a script into a stream of _tokens_.
+
+Tokens have the type [`Token`] which is only exported under [`internals`].
+
+The function [`Engine::on_parse_token`], available only under [`internals`], allows registration of a
+_mapper function_ that converts (remaps) a [`Token`] into another.
+
+
+```admonish tip.small "Hot Tips: Use as safety checks"
+
+Since it is called for _every_ token parsed from the script, this token mapper function
+can also be used to implement _safety checks_ against, say, stack-overflow or out-of-memory
+situations during parsing.
+
+See [here][memory] for more details.
+```
+
+
+Function Signature
+------------------
+
+```admonish tip.side "Tip: Raising errors"
+
+Raise a parse error by returning [`Token::LexError`](https://docs.rs/rhai/{{version}}/rhai/enum.Token.html#variant.LexError)
+as the mapped token.
+```
+
+The function signature passed to [`Engine::on_parse_token`] takes the following form.
+
+> ```rust
+> Fn(token: Token, pos: Position, state: &TokenizeState) -> Token
+> ```
+
+where:
+
+| Parameter |                                        Type                                         | Description                      |
+| --------- | :---------------------------------------------------------------------------------: | -------------------------------- |
+| `token`   |                                      [`Token`]                                      | the next symbol parsed           |
+| `pos`     |                                     `Position`                                      | location of the [token][`Token`] |
+| `state`   | [`&TokenizeState`](https://docs.rs/rhai/{{version}}/rhai/struct.TokenizeState.html) | current state of the tokenizer   |
+
+
+Example
+-------
+
+```rust
+use rhai::{Engine, FLOAT, Token};
+
+let mut engine = Engine::new();
+
+// Register a token mapper function.
+engine.on_parse_token(|token, pos, state| {
+    match token {
+        // Change 'begin' ... 'end' to '{' ... '}'
+        Token::Identifier(s) if &s == "begin" => Token::LeftBrace,
+        Token::Identifier(s) if &s == "end" => Token::RightBrace,
+
+        // Change all integer literals to floating-point
+        Token::IntegerConstant(n) => Token::FloatConstant((n as FLOAT).into()),
+        
+        // Disallow '()'
+        Token::Unit => Token::LexError(
+            LexError::ImproperSymbol("()".to_string(), "".to_string()).into()
+        ),
+
+        // Pass through all other tokens unchanged
+        _ => token
+    }
+});
+```