Language and conformance contract

This is the stable narrative contract for Shoal’s language implementation. It replaces the old monolithic docs/TDD.md as a link target while deliberately refusing to freeze stale Rust details in prose. Syntax, semantics, registries, and tests each have a more precise executable authority.

The guiding rule is:

Narrative states intent and invariants; source defines accepted structure; the conformance corpus pins observable language behavior.

Authority order🔗

When two artifacts disagree, resolve the dispute using this order and then repair the lower layers:

1. A deliberately reviewed conformance case is the normative observable behavior for the case it covers.
2. Parser/evaluator/value source plus focused tests describes current behavior where the corpus is silent.
3. Executable registries and schemas define enumerated names and shapes: builtin names, method metadata, AST variants, adapter schema, protocol types, and error enums.
4. This chapter and focused architecture chapters define cross-cutting intent, invariants, and ownership.
5. Historical root docs, comments, examples, and roadmap prose are migration inputs, not competing sources of truth.

This does not mean an accidental test can permanently override design. A behavior change begins by deciding the intended result, updating or adding the corpus case, then changing implementation. It does mean prose alone cannot declare an unimplemented feature “done.”

Stable ownership map🔗

Lexical contract🔗

Shoal source is UTF-8 and lexing is modal. The parser requests tokens in expression or command mode according to grammar position; the lexer does not inspect runtime bindings.

Common lexical invariants🔗

• Newline and semicolon terminate statements unless the parser recognizes continuation.
• A comment begins with # only where # starts a token; command word ver#2 remains one word.
• Double-quoted strings support interpolation and escapes; single-quoted strings are raw; triple forms span lines.
• Numeric maximal munch includes size and duration suffixes.
• Tagged regex and datetime literals are lexically distinct values.
• Illegal shell sigils and lone pipe syntax receive curated diagnostics rather than generic token errors.
• Every successful token span must advance; lexer fuzz/property tests defend against non-progress.

The exact token enum and spelling rules live in shoal-syntax::lexer. A new token is incomplete until both modes, parser use, formatter round-trip, highlighter/completer behavior, and diagnostics have been audited.

Command-mode contract🔗

Command mode recognizes words, path-shaped words, glob-shaped words, flags, leading environment assignments, redirects, the end-of-flags marker, background suffix, and expression arguments. It is word-oriented: punctuation that would be an operator in expression mode can be literal command text.

Shape classification carries semantic consequences:

Unknown external commands retain raw string behavior where no adapter or closure signature exists. Known calls can bind named flags and coerce word values according to their signature.

Expression-mode contract🔗

Expression mode recognizes identifiers, literals, delimiters, operators, calls, field/method/index postfixes, collections, lambdas, and control expressions. Bare unquoted filesystem paths and globs are not general expression literals; explicit constructors or strings are required outside command positions.

Statement dispatch contract🔗

Statement dispatch is the language’s central ambiguity rule:

The parser accepts a ParseCtx classification because lexical scope changes whether an identifier head is a value expression or command. This is the one intentional parse-time environment seam. Script/module parsing without a populated context cannot reproduce every REPL binding-dependent decision; hosts must choose and test their context deliberately.

^head forces command interpretation. Dynamic execution uses the explicit runtime call surface. An identifier immediately followed by a method/field chain can use invoke-then-chain behavior where implemented. The highlighter and completer approximate this rule; the parser remains authoritative.

Grammar and precedence contract🔗

The recursive-descent/Pratt parser owns the accepted grammar. The stable conceptual precedence from tight to loose is:

postfix field/method/index/call
unary
multiplicative
additive
range
comparison/membership
logical and
logical or
null coalescing
postfix catch
assignment

Comparison chaining is rejected rather than interpreted Python-style. Logical operators short-circuit. Assignments require a mutable binding/lvalue. Control flow is represented explicitly in the AST rather than implemented through textual rewriting.

Do not copy a full EBNF into another narrative page. The parser modules, AST serde types, formatter, and conformance corpus evolve together; a second grammar inevitably drifts. The parser/formatter chapter supplies the detailed ambiguity map and change procedure.

Desugaring contract🔗

Surface sugar must lower to a stable AST meaning without text splicing:

The exact desugaring representation is owned by shoal-ast and shoal-syntax; semantic equivalence belongs in corpus cases. If formatting a parsed form emits a canonical spelling, its parse-format-parse AST must remain equivalent.

Value and condition contract🔗

The live Value enum, not an old prose list, is authoritative. Values include scalar quantities, paths/globs/regex, structured collections, errors/outcomes, streams/tasks/plans, closures/partials, secrets, lazy CAS-backed bytes, namespaces, and host snapshots.

Key invariants:

• conditions accept booleans and command outcomes; arbitrary list/string/number truthiness is a type error;
• data equality is structural for variants that define it, with special/opaque variants following explicit implementation rules;
• integer and quantity arithmetic detects overflow and division by zero;
• paths preserve OS-native bytes; strict string conversion can fail while display is lossy;
• secrets render by name and do not become ordinary interpolated strings;
• a stream is single-consumption and unbounded streams reject inherently terminal collection without an explicit bound;
• an outcome preserves status versus signal, stdout/stderr, duration, pid, command, and success.

Detailed operator and render matrices live in Value algebra.

Calls, flags, and type annotations🔗

Evaluation is strict and left-to-right. Positional and named arguments bind in signature order; defaults are evaluated according to the closure/call implementation; variadic tails collect remaining arguments. Adapter signatures can translate long and short flags before argv assembly.

Current implementation does not provide complete runtime type soundness for user annotations:

• expression-call paths do not uniformly enforce parameter annotations;
• command word coercion converts recognized word forms, but already-typed non-string values can pass through;
• declared return annotations are not enforced.

Annotations are therefore useful call metadata/coercion hints, not a proof of static or dynamic type safety. The canonical call chapter documents exact behavior. A future enforcement change is a language behavior change and needs corpus coverage for expression calls, command calls, defaults, variadics, and return values.

Scope, modules, cwd, and environment🔗

• let creates immutable bindings; var creates mutable bindings.
• Blocks and closures use lexical environments.
• Modules load source files, evaluate them in a module environment, and expose exports through a module value.
• Session cwd and environment are evaluator state; scoped with behavior must restore state on all exits.
• Host-injected ports, configuration, Reef, policy, event bus, and journal are not ordinary lexical bindings and require deliberate inheritance in child evaluators.

The last point is a current correctness/security gap: several child-evaluator construction paths copy only part of host state. Narrative claims about structured concurrency or scoped authority do not override that source fact. See the evaluator and security chapters for the audited map.

Process-position contract🔗

An external command can be executed in captured mode or PTY mode. The high-level intent is:

• an ordinary interactive statement can preserve terminal behavior through PTY tee;
• value-producing, redirected, scripted, or non-TTY execution uses capture;
• interact/TUI adapter classification can request terminal treatment;
• cancellation targets the child’s process group and records signal death distinctly;
• captured output is bounded in memory, optionally spilling to the journal CAS;
• adapter ok_codes defines success; the default is exit code zero.

Source position alone is not the complete implementation key: host interactivity, redirection, adapter class, and explicit overrides participate. Consult the process and PTY chapters before changing this decision.

Stream contract🔗

Streams are pull-driven values with single-consumption state. Finite and unbounded sources are distinguished so terminal operations can reject unsafe collection. Operators compose lazily where implemented; bounded channels/tee paths must express backpressure or drop behavior explicitly.

Do not use the word “bounded” as a blanket claim. Some bridges use bounded synchronous channels, while the evaluator’s in-language event bus has unbounded live subscriber channels behind a bounded replay ring. The separate kernel EventBus uses bounded 256-event subscriber queues and coalesced gap summaries; the two buses must not be described as one identical backpressure implementation. The stream/channel chapter is the source-derived matrix.

Builtin and method contract🔗

Names are never pinned by prose lists. The canonical builtin command-head registry is shoal_syntax::commands::builtin_names(), consumed by parser/host tooling and evaluator dispatch. The discoverable value-method metadata lives in shoal-value/src/methods/suggest.rs; actual method behavior lives in the dispatch modules. They are intended to agree but currently have known drift, so neither may be treated as a generated perfect registry yet.

Adding a builtin or method requires:

1. executable registry/metadata update;
2. actual dispatch implementation;
3. argument/error/effect behavior tests;
4. a conformance case for stable user-visible behavior;
5. completion/highlighter/LSP audit;
6. external reference and focused internal ledger update.

Current registry/dispatch drift is documented rather than hidden. Do not “fix” prose to claim they match until tests prove it.

Canonical render contract🔗

The conformance harness compares the final value using render_inline, so its output is a protocol within the repository. Stable principles include:

• null and booleans use lowercase tokens;
• integers are decimal; floats follow the Rust display path used by the renderer;
• inline strings are quoted/escaped; top-level block rendering may print raw string contents;
• paths display lossily and quote when required by the renderer;
• regex, size, duration, datetime, time, list, record, table, error, outcome, secret, stream, task, plan, and lazy-byte values have variant-specific stable forms;
• record order follows the value representation rather than a prose promise of arbitrary map order.

Any render change can alter corpus expectations, shell output, prompt/picker display, journal value blobs, and test snapshots. Treat it as a compatibility change and audit every consumer.

Stable language error codes🔗

ErrorVal.code is program-visible through caught errors and is asserted by the corpus. Current source-emitted families include:

Some codes appear only in focused/live paths and not yet in corpus expectations. Conversely, historical lists named codes such as lang_block_unbalanced that current source may not emit as a distinct ErrorVal. The executable construction sites plus ReefCode enum are current authority.

Rules for codes:

• code spelling is stable API; messages and hints may improve without forcing callers to parse text;
• choose a specific existing code before falling back to custom;
• caught error fields must preserve code, message, hint, stderr, status, and span where available;
• new cross-subsystem codes need an executable enum/registry if possible, corpus coverage, and both internal/external documentation;
• JSON-RPC integer error codes are a different layer and must not be conflated with ErrorVal.code.

Normative conformance corpus🔗

spec/cases/*.toml is the behavioral specification. At the 2026-07-16 audit it contains 77 suite files and 1,310 globally named cases. A case has this conceptual shape:

[[case]]
name = "globally-unique-behavior-name"
src = """
let x = 2 + 3
x * 2
"""

# exactly one expectation family:
value = "10"
# error = "type_error"
# error_contains = "optional teaching substring"
# parse_error = true
# parse_error_contains = "optional parse teaching substring"

fixture = ["a.txt", "sub/b.log"]
stdin = "reserved / harness-dependent"
skip = "specific host-dependent reason"

Harness lifecycle🔗

sequenceDiagram
accTitle: Harness lifecycle
accDescr: Shows the components and relationships described in Harness lifecycle.
  participant H as conformance harness
  participant T as fresh temp directory
  participant P as shoal-syntax
  participant E as fresh Evaluator
  participant R as render_inline

  H->>T: create fixture paths
  H->>P: parse source in script context
  alt parse expectation
    P-->>H: diagnostic and span
  else evaluation expectation
    P-->>E: Program
    E-->>H: final Value or ErrorVal
    H->>R: render final Value
    R-->>H: canonical text
  end
  H->>H: compare and collect all failures

Each case uses a fresh evaluator and cwd, with no shared journal. Multi-statement source compares the last value. Fixtures create empty files and parent directories. Expectations should avoid ambient PATH, locale, current time, network, user config, or OS-specific rendering unless the case is explicitly skipped with a reason.

Current corpus state🔗

The live audit result was 1,306 passed, 0 failed, and 4 skipped. The skips cover a native-thread recursion-stack condition, a Node block, a jq feed composition, and full-chain Reef which. Counts are evidence from that run, not a permanently hardcoded health claim; release notes must run the corpus again.

Exhaustive suite ledger🔗

Every suite is named below so a language area cannot disappear behind an aggregate count. Counts come from [[case]] records in the current tree and sum to 1,310. This table should eventually be generated and checked in CI; until then, adding, renaming, or splitting a suite requires updating it.

Core syntax, control flow, and diagnostics🔗

Values, operators, methods, and rendering🔗

Filesystem, command binding, adapters, and namespaces🔗

External I/O, shell blocks, and outcomes🔗

Reef🔗

Streams and sinks🔗

Two-runner risk🔗

There are parallel harness entrypoints in shoal and shoal-eval. They should execute the same schema and semantics, but duplicated parsing/fixture/error-comparison logic can drift. The long-term contract should live in one shared test-support implementation invoked from both packages.

Behavior-change protocol🔗

For any observable language change:

1. state whether the change is bug fix, specification correction, or compatibility break;
2. add or revise the smallest corpus cases that distinguish old and new behavior;
3. add focused unit/property/live tests for the implementation invariant;
4. change source and registries;
5. run formatter idempotence and conformance through both harnesses;
6. audit completion, highlighting, LSP, adapters, wire rendering, and docs as applicable;
7. record migration advice when existing scripts can break;
8. update implementation status—do not leave completed roadmap prose as the only evidence.

Historical TDD reconciliation🔗

The old TDD mixed durable choices, stale lists, implemented behavior, and aspirations. Its valuable content is absorbed as follows:

This page is deletion-ready with respect to docs/TDD.md: no source comment should continue to use an ignored or root Markdown document as its normative semantic link.

Invariants reviewers should defend🔗

• Parser mode is driven by grammar position, not runtime token guessing.
• Binding-aware statement dispatch enters through an explicit parse context.
• Aliases and other sugar manipulate AST/value structures, never splice shell text.
• Only boolean/outcome conditions are accepted.
• Error codes and canonical render forms are program-visible compatibility surfaces.
• Streams cannot be silently consumed twice or infinitely collected.
• OS effects are represented at a host/port boundary even where current code has known leaks.
• Enumerated names come from executable registries.
• A feature with renderer/schema support but no host producer is labeled scaffolded.
• The corpus decides covered observable behavior, and a behavior change updates the corpus first.