wip

Sophia-FQA.md

@@ -120,8 +120,8 @@ Most compilers have some variation of the following architecture:
     This is where compiler engineering gets interesting, and factors
     like artistic choice and taste start to dominate. Different
     optimizations occur at different levels of intermediate
-    represntation. The structure of this meta-step depends heavily on
-    the source and target languages, problem domains, goals of the
+    representation. The structure of this meta-step depends heavily
+    on the source and target languages, problem domains, goals of the
     specific compiler, etc.
 
     This is the step in which we think of phrases in the language in
@@ -147,20 +147,20 @@ comments and whitespace, and "signal" is everything else.
 Most compilers discard "noise" tokens (comments and whitespace). GSC
 retains them for two reasons:
 
-1. sanity-checking to make sure information isn't lost on accident;
-   e.g. one of gsc's tests
-2. future-proofing in case we want to add Python/Lisp
-style doc comments as a language feature down the line.
+1. sanity-checking at various stages to make sure non-noise
+   information isn't lost on accident;
+2. future-proofing in case we want to add Python/Lisp style doc
+   comments as a language feature down the line.
 
-```python
-def foo():
+   ```python
+   def foo():
        "this is a doc comment for foo"
        print("hi from foo")
-```
+   ```
 
-![](./uploads/python-doc1.png)
+   ![](./uploads/python-doc1.png)
 
-![](./uploads/python-doc2.png)
+   ![](./uploads/python-doc2.png)
 
 However for non-bikeshed compiler tasks (figuring out what the code
 is supposed to *do* and then expressing that in the target language),
@@ -317,6 +317,63 @@ contract Hello =
 {tk,ws,{10,23},"\n"}
 ```
 
+# How token parsing works
+
+The basic approach is very simple:
+
+1.  Each token shape has a parser; e.g.
+    ```erlang
+    slurp_token_of_shape(lcom, Pos, SrcStr) ->
+        case SrcStr of
+            "//" ++ _ ->
+                {Line, Rest} = takeline("", SrcStr),
+                Token = #tk{shape = lcom,
+                            pos   = Pos,
+                            str   = Line},
+                {tokmatch, Token, Rest};
+            _ ->
+                no_tokmatch
+        end;
+    ```
+
+2.  There is a pre-defined parse order
+
+    ```erlang
+    token_shapes_parse_order() ->
+        [% comments and whitespace
+         lcom, bcom, ws, sep,
+         % literals
+         char, string, int16, int10, bytes, ak, ct, sg,
+         % qualified names need to go ahead of unqualifieds
+         qid, qcon, tvar,
+         % keywords need to be parsed ahead of ids
+         kwd, id, con,
+         % ops [=, =>, >>]
+         op].
+    ```
+
+3.  We look at the head of the source string and try to parse it
+    against each token shape
+
+    ```erlang
+    -spec slurp_token(Pos, SrcStr) -> Result
+        when Pos    :: tk_pos(),
+             SrcStr :: string(),
+             Result :: {tokmatch, Token, Rest}
+                     | no_tokmatch
+                     | {error, gsc_err()}
+                     | {ierr, unterminated_block_comment},
+             Token  :: tk(),
+             Rest   :: string().
+    % @doc
+    % grab a single token off the front of the string according to
+    % `token_shapes_parse_order/0'
+
+    slurp_token(Pos, SrcStr) ->
+        % this is the easiest format if i need to fuck with it
+        slurp_token_shapes(token_shapes_parse_order(), Pos, SrcStr).
+    ```
+
 
 
 # Defining Events in interfaces