Compilation Pipeline

Pipeline Overview

flowchart TD
  A["Source (.skein)"] --> B["Lexer"]
  B -->|"Token stream"| C["Parser"]
  C -->|"AST"| D["Analyzer"]
  D -->|"Annotated AST"| E["CodeGen"]
  E -->|"Core Erlang AST"| F[":compile.forms"]
  F -->|"BEAM bytecode"| G[":code.load_binary"]
  G --> H["Loaded module"]

  B -.- B1["Source text → Token stream"]
  C -.- C1["Token stream → AST"]
  D -.- D1["Type checking, capability checking"]
  E -.- E1["AST → Core Erlang (via :cerl)"]
  F -.- F1["Core Erlang → BEAM bytecode"]
  G -.- G1["BEAM bytecode → Loaded module"]

Entry Points

The compiler has two entry points in Skein.Compiler:

compile_string/1

Compiles a Skein source string to a loaded BEAM module:

{:module, mod} = Skein.Compiler.compile_string(~S"""
module Math {
  fn add(a: Int, b: Int) -> Int {
    a + b
  }
}
""")
mod.add(3, 4) #=> 7

compile_file/1

Reads a .skein file and compiles it:

{:module, mod} = Skein.Compiler.compile_file("examples/hello.skein")

Both return {:module, module()} on success or {:error, [Skein.Error.t()]} on failure.

Pipeline Stages

Stage 1: Lexer

Input: UTF-8 source string Output: {:ok, [token()]} or {:error, [Error.t()]}

Converts source text to a flat list of tokens. Each token is a tuple:

{:keyword, {line, col}}           # e.g. {:module, {1, 1}}
{:token_type, {line, col}, value} # e.g. {:ident, {1, 8}, "hello"}

See Lexer for details.

Stage 2: Parser

Input: Token list Output: {:ok, %AST.Module{}} or {:error, [Error.t()]}

Builds a structured AST from the token stream using recursive descent with Pratt-style operator precedence.

See Parser for details.

Stage 3: Analyzer

Input: AST Output: {:ok, annotated_ast} or {:error, [Error.t()]}

The analyzer runs a suite of passes over the AST (Passes 0a–10 in the module pass driver, plus a dedicated agent pass suite for agent bodies). Broadly, the passes group into five areas:

1. Name resolution — builds a symbol table of all declarations (functions, types, enums), resolves identifier references, and reports unknown identifiers (E0010) and unknown types (E0024)
2. Type checking — validates function return types, operator types, match arm consistency, and function call arity, reporting type mismatches (E0020) and non-exhaustive match errors (E0021; E0024 for enums — both are hard errors, not warnings)
3. Capability checking — walks function bodies to find effect calls (http.*, store.*, memory.*, llm.*, tool.*, queue.*, topic.*, event.*, process.*, timer.*, uuid.*, instant.*), verifies each has a covering capability declaration, and reports missing capabilities (E0012) with fix_code for agent auto-fix
4. Transition checking — validates agent Phase enum transitions, ensuring transition(Phase) calls reference valid target phases reachable from the current phase (E0030)
5. Lint warnings — detects unused bindings (W0001), unused capabilities (W0002), and unreachable code after stop() (W0003)

Stage 4: Code Generator

Input: AST Output: {:ok, [{module_atom, beam_binary}]} or {:error, [Error.t()]}

Translates the AST to Core Erlang using the :cerl module, then compiles to BEAM bytecode via :compile.forms/2. The result is a list of named binaries because one source file can produce several BEAM modules: the primary module comes first, followed by one entry per nested agent.

See Code Generator for details.

Stage 5: Module Loading

The final step uses :code.load_binary/3 to load each generated binary into the running VM:

Enum.each(modules, fn {module_name, beam_binary} ->
  :code.load_binary(module_name, ~c"nofile", beam_binary)
end)
# The compiler returns {:module, primary_module_atom}

Every module (the primary first, then nested agents) is loaded and immediately available for function calls.

Error Handling

Each stage returns {:ok, result} or {:error, errors}. The with construct chains them:

def compile_string(source) do
  with {:ok, tokens} <- Lexer.tokenize(source),
       {:ok, ast} <- Parser.parse(tokens),
       {:ok, annotated_ast} <- Analyzer.analyze(ast),
       {:ok, modules} <- CoreErlang.generate(annotated_ast) do
    load_modules(modules, ~c"nofile")
  end
end

If any stage fails, the error propagates immediately. Errors are %Skein.Error{} structs with structured fields for both human and machine consumption.