Milestones 3–4: AST, transformers, and the tree-shape gate

CLAUDE.md

@@ -29,8 +29,18 @@ in `internal/grammar/README.md`.
 - `cmd/regenerate/` — rebuild `parser/testdata/` from a DuckDB source tree
   + the pinned binary. The only way corpus expectations change.
 - `cmd/next-test/` — print the next todo case from the corpus metadata.
-- `parser/` — corpus conformance harness (`parser_test.go`); the public
-  Parse API and AST arrive in milestone 3.
+- `ast/` — the public AST: statements, query nodes, table refs,
+  expressions, DDL infos. Ports upstream's parser/ node classes.
+- `parser/` — the public `Parse`/`ParseStatement`/`ParseExpr` API, the
+  hand-written transformer (`transform_*.go`, the port of upstream's
+  `transformer/`), the corpus conformance harness (`parser_test.go`) and
+  the serialize goldens gate (`serialize_test.go`).
+- `internal/serialize/` — renders the AST in `json_serialize_sql`-
+  compatible JSON; `serialize.Equal` is the structural comparator (its
+  doc comment lists the normalizations).
+- `cmd/serialize-diff/` — diff darkwing's serialization against a live
+  oracle binary for ad-hoc statements or the whole corpus SELECT subset;
+  writes `parser/testdata/serialize/goldens.jsonl`.
 
 ## Rules of the port
 
@@ -56,12 +66,20 @@ Debugging a parse:
 ```
 go run ./cmd/debug-parse 'SELECT 1'            # ParseResult tree
 go run ./cmd/debug-parse -tokens 'SELECT 1'    # token dump
+go run ./cmd/debug-parse -ast 'SELECT 1'       # transformed AST as JSON
 ```
 
 ## Conformance loop
 
-The milestone-2 gate: darkwing accepts a statement iff the pinned DuckDB
-binary parses it, over the whole corpus (`go test ./parser`).
+Two corpus gates run under `go test ./parser`:
+
+- **Accept/reject** (`TestCorpus`): darkwing accepts a statement iff the
+  pinned DuckDB binary parses it. Since milestone 3 the classification
+  runs the full Parse pipeline, so transformer-raised Parser Errors count
+  alongside the matcher's syntax errors.
+- **Tree shape** (`TestSerializeGoldens`): `internal/serialize` output
+  must match vendored `json_serialize_sql` goldens for a corpus sample
+  (see `parser/testdata/serialize/README.md`).
 
 ```
 go run ./cmd/next-test                                  # pick the next todo case
@@ -70,10 +88,17 @@ go test ./parser -run TestCorpus -check-parse 'FRAGMENT' # dump detail for it
 go test ./parser                                        # gate
 ```
 
-Some oracle rejects come from upstream's *transformer* (Parser Errors whose
-message is not "syntax error at or near ..."): the matcher alone cannot
-reject those, so they stay in todo metadata until the matching transformer
-lands (milestones 3-5). Todo entries carry a note saying why.
+Oracle rejects raised by upstream's *transformer* (Parser Errors whose
+message is not "syntax error at or near ...") that darkwing's transformer
+does not reproduce yet stay in todo metadata with a note saying why; the
+harness flags todo entries that start agreeing so they get removed.
+
+Sweeping tree shapes against a live oracle (milestones 3-4 exit
+criteria: zero mismatches over the corpus SELECT subset):
+
+```
+DARKWING_DUCKDB=/path/to/duckdb-cli go run ./cmd/serialize-diff -corpus
+```
 
 Regenerating the corpus (needs a DuckDB checkout at the pinned commit and
 the matching nightly CLI; see `internal/grammar/README.md` for the pin):

ast/ast.go

@@ -0,0 +1,107 @@
+// Package ast defines darkwing's typed syntax tree for the DuckDB SQL
+// dialect. Node names follow DuckDB's parsed statement and expression
+// classes (SelectStatement, ColumnRefExpression, ...) so the transformer
+// and serializer read side-by-side with upstream's transformer/*.cpp;
+// sqlc's future convert.go owns the mapping to sqlc's own sql/ast.
+package ast
+
+// Span is a half-open byte range [Start, End) into the original SQL text.
+//
+// Statement spans tile the input: sqlc slices the source by statement span
+// to find `-- name:` comments. Expression spans mirror DuckDB's
+// query_location bookkeeping (the position upstream reports in errors and
+// json_serialize_sql), which for some nodes is the operator token rather
+// than the full extent of the expression.
+type Span struct {
+	Start int `json:"start"`
+	End   int `json:"end"`
+}
+
+// Node is implemented by every syntax tree node.
+type Node interface {
+	// Pos returns the byte offset where the node begins.
+	Pos() int
+	// End returns the byte offset just past the node.
+	End() int
+	// Children returns the node's direct children, for generic walks.
+	Children() []Node
+}
+
+// spanned supplies the Span field plus the Pos/End half of Node; every
+// concrete node embeds it.
+type spanned struct {
+	Span Span `json:"span"`
+}
+
+func (s *spanned) Pos() int { return s.Span.Start }
+func (s *spanned) End() int { return s.Span.End }
+
+// SetSpan overwrites the node's span; used by the transformer.
+func (s *spanned) SetSpan(sp Span) { s.Span = sp }
+
+// Stmt is implemented by all statement nodes.
+type Stmt interface {
+	Node
+	stmtNode()
+}
+
+// Expr is implemented by all expression nodes (DuckDB's ParsedExpression
+// hierarchy).
+type Expr interface {
+	Node
+	exprNode()
+	// GetAlias and SetAlias expose the alias every DuckDB expression
+	// carries (e.g. `expr AS name` in a select list).
+	GetAlias() string
+	SetAlias(string)
+}
+
+// TableRef is implemented by all table reference nodes.
+type TableRef interface {
+	Node
+	tableRefNode()
+}
+
+// QueryNode is a node of the query tree wrapped by SelectStatement:
+// SelectNode, SetOperationNode, RecursiveCTENode or CTENode — DuckDB's
+// shape.
+type QueryNode interface {
+	Node
+	queryNode()
+	// ModifiersRef exposes the node's result modifier list (ORDER BY,
+	// LIMIT, ...) for the transformer to append to.
+	ModifiersRef() *[]ResultModifier
+	// CTEMapRef exposes the node's CTE map for the transformer.
+	CTEMapRef() *CTEMap
+}
+
+// ResultModifier is a modifier applied to a query node's result: ORDER BY,
+// LIMIT, DISTINCT, LIMIT PERCENT.
+type ResultModifier interface {
+	Node
+	resultModifierNode()
+}
+
+// exprs converts a []Expr to []Node, skipping nils. The transformer
+// stores untyped nils in absent interface fields, so a plain nil check
+// suffices.
+func exprs(list []Expr) []Node {
+	nodes := make([]Node, 0, len(list))
+	for _, e := range list {
+		if e != nil {
+			nodes = append(nodes, e)
+		}
+	}
+	return nodes
+}
+
+// add appends non-nil nodes. Callers pass interface fields directly; the
+// transformer never stores typed nils, so a plain nil check suffices.
+func add(nodes []Node, extra ...Node) []Node {
+	for _, n := range extra {
+		if n != nil {
+			nodes = append(nodes, n)
+		}
+	}
+	return nodes
+}