On this page ↓

Runtime book

Calls, closures, modules, and namespaces

Callable dispatch, strict argument binding, closure capture, module evaluation and caching, constructors, and synthetic namespace behavior.

Status
Callable and module semantics
For
Language and runtime contributors
On this page
  1. Call surface map
  2. Runtime callable values
  3. Recursion guard
  4. Strict closure binding
  5. Closure environment lifecycle
  6. Command references
  7. Constructor table
  8. Module resolution
  9. Module evaluation lifecycle
  10. Export model
  11. Circular-use diagnostics
  12. Synthetic namespaces
  13. Data namespace conversions
  14. HTTP namespace limits
  15. Configuration namespace
  16. Known sharp edges
  17. Change protocol

Shoal has several callable surfaces that converge only after parsing: user closures, command references, evaluator-recognized functions, type-like constructors, namespace methods, value methods, and ordinary commands. Their precedence and argument conversions are language semantics.

Sources: call.rs, modules.rs, namespaces.rs, and env.rs.

Call surface map🔗

flowchart TD
accTitle: Call surface map
accDescr: Shows the components and relationships described in Call surface map.
  Syntax["name(args) / recv.method(args)"] --> EvalArgs
  EvalArgs --> Special["parallel / retry / on / now / today / assert / run / save / open"]
  EvalArgs --> Constructor["path / glob / regex / channel / every / watch / tail"]
  EvalArgs --> EnvLookup["environment callable"]
  EnvLookup --> Closure
  EnvLookup --> CmdRef
  EvalArgs --> Namespace["json/yaml/toml/csv/math/http/os/config"]
  EvalArgs --> ValueMethod
  EvalArgs --> CommandFallback

eval_args evaluates positional expressions left-to-right, then named argument expressions in their source order, producing CallArgs { pos, named }. Named arguments remain an ordered vector, not a map. Duplicate-name rejection, if desired, must happen at syntax or binding time; map-like overwrite behavior should never be assumed.

Runtime callable values🔗

Only two ordinary Value variants are accepted by call_value_inner:

CallableStored stateInvocation behavior
Closureoptional name, params, rest, return annotation, body, captured Env, docscreates child of captured env and binds arguments
CmdRefstructured CmdCallappends call-site arguments/flags, then executes command in value position

All other values produce type_error: <type> is not callable. Constructors and namespace methods are evaluator dispatch cases, not first-class callable values. Consequently let f = json.parse does not yield a function; namespace functions must be called through their syntactic receiver.

Recursion guard🔗

call_value increments call_depth, rejects depth above the named 128-call ceiling with recursion_limit, calls the inner dispatcher, and decrements afterward. The decrement happens for success and returned VResult errors. Function-call expressions dispatch without retaining the large non-call expression frame, and closure blocks enter block evaluation directly; exact-boundary and mutual-recursion regressions reach the typed error on an explicit 32 MiB native thread stack. The conformance corpus still skips its unbounded-recursion case because its default Rust test worker has a smaller native stack than that explicitly tested runtime boundary.

Strict closure binding🔗

Closure calls reject excess positional arguments unless a rest parameter exists. They reject any named argument whose name is absent from the declared parameter list. For each declared parameter, binding priority is:

  1. a matching named argument;
  2. the positional argument at the parameter’s declaration index;
  3. the parameter’s default expression;
  4. otherwise arg_error for a missing argument.

This means named arguments do not compact the positional list. A positional value at index i targets parameter i even if another parameter was supplied by name.

Defaults execute after the evaluator has swapped to captured_env.child() and after earlier parameters have been declared. They can therefore reference earlier parameters and captured bindings. On binding/coercion failure, the previous evaluator environment is restored before the error returns.

call_value_inner is the authoritative annotation boundary for both expression and command calls. It applies shell-word parsing to string inputs and strict UTF-8 path to str conversion for path-shaped command words; every other already-tagged value must match its declared runtime type. list<T> is checked/coerced recursively, table<T> is restricted to record rows, optional T? accepts null, defaults pass through the same path, and invalid annotation names/shapes fail closed. Declared returns reuse the validator in exact-check mode: no string parsing or numeric widening is performed after the body returns.

The rest parameter receives a List of positional values after the fixed parameter count. Named arguments never flow into rest. Both ...xs: T and ...xs: list<T> apply T to every collected item, including expression-valued arguments.

Closure environment lifecycle🔗

sequenceDiagram
accTitle: Closure environment lifecycle
accDescr: Shows the components and relationships described in Closure environment lifecycle.
  participant D as Declaration
  participant E as Env handle
  participant C as Closure
  participant I as Invocation
  D->>E: capture current Env.clone()
  D->>C: store body + params + Env
  I->>E: captured.child()
  I->>I: bind params/rest
  I->>I: in_fn_body += 1
  I->>I: eval body in value position
  I->>I: in_fn_body -= 1
  I->>I: restore caller Env

Env is an Arc<Mutex<EnvInner>>. Cloning an environment shares its bindings; it is not a snapshot. Each EnvInner has a variable map and optional parent. get checks the current map, releases the lock, then walks the parent. assign walks to the binding that owns the name and rejects immutable bindings. declare always writes the current scope and replaces any same-scope binding.

The mutex makes environment handles shareable, but it does not make arbitrary evaluator execution parallel-safe: the Evaluator itself remains mutable and closures also rely on its cwd, ports, command catalog, and other session fields.

Command references🔗

An alias is stored as a CmdRef containing a structured AST command. Calling it clones that command, converts each positional runtime value to a command argument, and appends named arguments as long flags. Bool(true) becomes a presence flag; every other named value becomes the flag’s value.

alias gs = git status
gs(--short: true)       # structured equivalent of git status --short

The evaluator then calls ordinary command dispatch in value position. The clone is important: one alias invocation cannot mutate the command used by the next.

Constructor table🔗

Constructors are recognized by name before ordinary command fallback.

NameAccepted inputResult / notes
pathexactly one str or pathPath, with path input cloned
globone pattern string, optional hidden/follow nameslazy GlobVal rooted at current cwd; current implementation stores hidden but not follow
regexexactly one stringcompiled RegexVal or validation error
channelexactly one stringlanguage channel handle record
everyone nonnegative durationlazy interval stream
watchone path or glob; recursive defaults truefilesystem event stream
tailone file path; from_start defaults falseappended-line stream

Constructor dispatch returns Option<Value> internally: None means the name was not a constructor and resolution continues. An error means the name was recognized but arguments were invalid; it must not fall through to an external command of the same name.

Module resolution🔗

use ./lib/deploy is resolved relative to the evaluator cwd. If the supplied path has no extension, the loader tries <path>.shl first and the exact path second. It canonicalizes the first existing candidate and uses that canonical path as both cache key and cycle identity.

Canonicalization means symlink and spelling aliases generally converge on one module identity. It also means module resolution requires the file to exist and can fail on filesystem permission or canonicalization errors before evaluation begins.

Module evaluation lifecycle🔗

flowchart LR
accTitle: Module evaluation lifecycle
accDescr: A canonical module path is cycle-checked, evaluated in a fresh environment, reduced to explicit exports, cached, and bound into the caller as a record.
  Use["use path"] --> Canon["canonical path"]
  Canon --> Cached{"cached?"}
  Cached -->|yes| Bind["bind cloned export record"]
  Cached -->|no| Cycle{"already loading?"}
  Cycle -->|yes| Error["module cycle error"]
  Cycle -->|no| Parse["read + parse"]
  Parse --> Fresh["fresh root env + module cwd"]
  Fresh --> Eval["evaluate top-level statements"]
  Eval --> Exports["collect exported fn / let names"]
  Exports --> Record["ordered export record"]
  Record --> Cache["cache by canonical path"]
  Cache --> Bind

The loader resets in_fn_body to zero during module top-level evaluation, allowing module setup to use top-level-only operations. It restores the prior environment, cwd, and function nesting after success or failure. The module-stack entry is also popped before propagating an evaluation error.

An uncached module reserves against the bounded module identity budget before I/O, then reads at most the parser’s 4 MiB source wall plus one sentinel byte through the inherited Fs port. Invalid UTF-8 or oversized source never enters the cache or active module stack; correcting the file permits a later load in the same healthy evaluator.

A module is evaluated at most once per evaluator. Its exports value is cloned from the cache on subsequent use statements, and then declared under the canonical file’s stem. Caching means changes to the file are not observed until a new evaluator/session.

Export model🔗

Only top-level export fn and export let contribute fields to the exports record. Exported let patterns are traversed recursively, including list-pattern rest names. Aliases and other statement forms are not exported by collect_exports.

Private declarations remain in the module environment. An exported closure captures that environment and can continue reading private helpers after the loader restores the caller environment. Exported plain values are cloned into the record.

There is no module manifest, package identity, explicit import list, re-export mechanism, or cache invalidation protocol in this layer. Module naming is the file stem, so two files with the same stem cannot be bound simultaneously in one scope without replacement/shadowing behavior.

Circular-use diagnostics🔗

Before loading, the canonical path is checked against module_stack. A match constructs an error chain from every stack path plus the repeated path. This is dynamic cycle detection: only the actual evaluation path matters. Cached modules return before the stack check because completed loads cannot form an active cycle.

Synthetic namespaces🔗

Namespaces do not have a runtime Value variant. The evaluator recognizes an unbound receiver name and dispatches directly:

NamespaceFieldsMethods
jsonnoneparse, stringify
yamlnoneparse, stringify
tomlnoneparse, stringify
csvnoneparse, stringify
mathpi, e, tau, inf, nan, sqrt2numeric functions such as sqrt, trig, logs, rounding, pow, min, max, hypot, clamp
httpnoneget, delete, post, put
osnonesystem/session queries and environment operations implemented in namespaces.rs
configprojected keysget, all

An environment binding shadows a namespace. This check is explicit: namespace dispatch occurs only when env.get(ns).is_none(). Field access on a function-only namespace returns a diagnostic telling the caller to invoke it.

Names and argument shapes live in one evaluator registry shared by runtime admission, the REPL completer, and the LSP vocabulary. Every method rejects extra positional arguments and unknown named arguments before codec allocation, network setup, configuration access, or other dispatch. Optional forms remain explicit: json.stringify(value, pretty)/pretty:, HTTP’s body/header positions, and the headers: HTTP named argument. This prevents math.sqrt(4, 999)-style typos from silently changing meaning as implementations evolve.

Data namespace conversions🔗

JSON, YAML, and TOML parsing first deserialize into serde_json::Value, then use the shared fallible json_to_value projection. Stringification performs the reverse value_to_json projection. Integer values outside Shoal’s signed 64-bit range fail with number_range; the JSON source path additionally checks exact number tokens before serde’s generic value representation can round them. That common bridge gives all three formats the same non-lossy numeric boundary.

All four data namespaces route through data_codecs.rs. Parsing admits no more than 16 MiB of input, 131,072 structural nodes, depth 128, or 16 MiB of retained Shoal values. JSON, YAML, and CSV encode through a writer that cannot cross 16 MiB. TOML preflights a conservative escape and repeated table path bound before its internally buffered serializer runs, then renders through that same writer. Limit failures are typed data_materialization_limit errors and leave the evaluator reusable.

json.stringify accepts pretty either as a named boolean or second positional true. TOML stringification can reject values which are legal Shoal/JSON but invalid at TOML’s top level.

CSV parse treats the first row as headers and returns a Table; all fields are strings. CSV stringify accepts a table, a list of records, or one record. It chooses header order from the first record and renders non-string cells inline. Missing fields become empty cells, and later-record keys that were absent from the first record are omitted. CSV also stops at 16,384 rows and 131,072 cells.

HTTP namespace limits🔗

HTTP calls are synchronous evaluator effects. The implementation applies a 30-second request timeout and caps response bodies at 64 MiB before projecting the response. Its convenience JSON projection uses the data-codec input/tree walls and becomes null when the body is invalid or outside them; the raw bounded body remains available. get and delete do not take a body; post and put do. Changes here belong in the effect/security audit because namespace syntax can otherwise make network access look like a pure method.

Configuration namespace🔗

The evaluator never walks configuration files on a config access. The host injects an already resolved ConfigPort; default construction uses an empty ConfigSnapshot. This keeps language reads consistent with host-applied configuration and makes tests deterministic.

The direct config.member form projects one key. config.get/config.all are method forms. Adding a host setting requires deciding whether it is intentionally public to language programs and how its type projects into Value; do not expose arbitrary internal configuration serialization by accident.

Known sharp edges🔗

  • Namespace functions are not first-class values.
  • glob(..., follow: ...) is accepted by constructor validation, but the stored GlobVal shown in call.rs only records hidden; contributors should verify the intended follow-symlink contract.
  • Module evaluation has a fresh lexical root, not a pure/effect-isolated sandbox.
  • Cache lifetime is the evaluator lifetime; no file watcher or content hash invalidates modules.
  • Synthetic namespace precedence is hand-coded in expression evaluation. A new access path can omit the shadowing rule and create inconsistent behavior.

Change protocol🔗

For a new call form or namespace member:

  1. decide whether it must be first-class; if yes, a synthetic namespace may be the wrong model;
  2. place it deliberately in the dispatch precedence and specify user-binding shadow behavior;
  3. validate unknown named arguments and excess positionals consistently;
  4. use existing coercion functions instead of parsing rendered values ad hoc;
  5. route filesystem, process, clock, network, opener, secret, and config work through the correct capability boundary;
  6. preserve the call-site span and narrower argument spans;
  7. test direct use, captured use, module use, and command-name collision;
  8. for modules, test cache identity, relative resolution, private capture, and a multi-file cycle;
  9. update external language documentation if syntax-visible;
  10. update kernel value projection when a newly returned runtime type can reach RPC.
Type to search every guide↑↓ navigate ↵ open esc close
Diagram