1. Parsing

Why tree-sitter, and why strict

The grammar is a hand-written tree-sitter grammar (packages/tree-sitter-clay/grammar.js). Tree-sitter always produces a tree, even for broken input, by inserting ERROR/MISSING nodes. That is perfect for an editor (highlight while typing) and dangerous for a hash. So there are two modes over the same grammar:

• tolerant (editor highlighting): use whatever tree you get;
• strict (compiler path): if any node is ERROR or MISSING, throw. Broken code is never hashable.

function parseStrict(src: string): Node {
  const tree = parser.parse(src);
  const bad = firstNode(tree.rootNode, n => n.type === "ERROR" || n.isMissing);
  if (bad) throw new ParseError(bad.startPosition, src);
  return tree.rootNode;
}

This site does the same: ClayCode refuses to build a page containing a sample that does not parse.

The concrete syntax tree

For let add = (a: int, b: int): int => a + b the grammar yields (abridged):

(source_file
  (let_definition name: (identifier)
    value: (function
      parameters: (parameters (parameter name: (identifier) type: (type_annotation ...))
                              (parameter name: (identifier) type: (type_annotation ...)))
      return_type: (type_annotation type: (type_identifier))
      body: (binary_expression left: (identifier) right: (identifier)))))

Names like name: and body: are grammar fields; the AST builder reads children by field, not position.

Two design points worth copying:

• Disjoint lexical classes. Console, Circle are ability_name / constructor_name tokens (capitalised); add is an identifier. A capitalised name can never be confused with a variable, so Console.print(x) needs no lookahead.
• One declared conflict. (x) => ... versus (x) is genuinely ambiguous until the =>, so the grammar declares a GLR conflict for exactly that and nothing else.

The AST

packages/compiler/src/ast.ts is a plain tagged union that still looks like the source. For add:

{ "name": "add",
  "value": { "kind": "lambda",
             "params": [{ "name": "a", "type": "int" }, { "name": "b", "type": "int" }],
             "returnType": "int",
             "body": { "kind": "binary", "op": "+",
                       "left":  { "kind": "var", "name": "a" },
                       "right": { "kind": "var", "name": "b" } } },
  "line": 1,
  "source": "let add = (a: int, b: int): int => a + b" }

Note source: the original text is kept beside the tree, unhashed, because desugaring is lossy and users want their own text (and comments) back. Only what lowering produces is hashed.

If you build this yourself: write the AST as a union of tagged objects, one kind per syntax form, and keep every conversion in one file (to-ast.ts). It stays boring, which is the point.

Next: Core IR.