diff --git a/Sophia-FQA.md b/Sophia-FQA.md index d86e83d..f26b655 100644 --- a/Sophia-FQA.md +++ b/Sophia-FQA.md @@ -120,8 +120,8 @@ Most compilers have some variation of the following architecture: This is where compiler engineering gets interesting, and factors like artistic choice and taste start to dominate. Different optimizations occur at different levels of intermediate - represntation. The structure of this meta-step depends heavily on - the source and target languages, problem domains, goals of the + representation. The structure of this meta-step depends heavily + on the source and target languages, problem domains, goals of the specific compiler, etc. This is the step in which we think of phrases in the language in @@ -147,20 +147,20 @@ comments and whitespace, and "signal" is everything else. Most compilers discard "noise" tokens (comments and whitespace). GSC retains them for two reasons: -1. sanity-checking to make sure information isn't lost on accident; - e.g. one of gsc's tests -2. future-proofing in case we want to add Python/Lisp -style doc comments as a language feature down the line. +1. sanity-checking at various stages to make sure non-noise + information isn't lost on accident; +2. future-proofing in case we want to add Python/Lisp style doc + comments as a language feature down the line. -```python -def foo(): - "this is a doc comment for foo" - print("hi from foo") -``` + ```python + def foo(): + "this is a doc comment for foo" + print("hi from foo") + ``` -![](./uploads/python-doc1.png) + ![](./uploads/python-doc1.png) -![](./uploads/python-doc2.png) + ![](./uploads/python-doc2.png) However for non-bikeshed compiler tasks (figuring out what the code is supposed to *do* and then expressing that in the target language), @@ -317,6 +317,63 @@ contract Hello = {tk,ws,{10,23},"\n"} ``` +# How token parsing works + +The basic approach is very simple: + +1. Each token shape has a parser; e.g. + ```erlang + slurp_token_of_shape(lcom, Pos, SrcStr) -> + case SrcStr of + "//" ++ _ -> + {Line, Rest} = takeline("", SrcStr), + Token = #tk{shape = lcom, + pos = Pos, + str = Line}, + {tokmatch, Token, Rest}; + _ -> + no_tokmatch + end; + ``` + +2. There is a pre-defined parse order + + ```erlang + token_shapes_parse_order() -> + [% comments and whitespace + lcom, bcom, ws, sep, + % literals + char, string, int16, int10, bytes, ak, ct, sg, + % qualified names need to go ahead of unqualifieds + qid, qcon, tvar, + % keywords need to be parsed ahead of ids + kwd, id, con, + % ops [=, =>, >>] + op]. + ``` + +3. We look at the head of the source string and try to parse it + against each token shape + + ```erlang + -spec slurp_token(Pos, SrcStr) -> Result + when Pos :: tk_pos(), + SrcStr :: string(), + Result :: {tokmatch, Token, Rest} + | no_tokmatch + | {error, gsc_err()} + | {ierr, unterminated_block_comment}, + Token :: tk(), + Rest :: string(). + % @doc + % grab a single token off the front of the string according to + % `token_shapes_parse_order/0' + + slurp_token(Pos, SrcStr) -> + % this is the easiest format if i need to fuck with it + slurp_token_shapes(token_shapes_parse_order(), Pos, SrcStr). + ``` + # Defining Events in interfaces