On this page ↓

Process I/O

External commands and data exchange

Run raw and adapted programs, understand PTY versus capture, serialize stdin with feed, redirect output, and embed interpreter source safely.

Status
Current process runner and adapter pack
For
Shell and integration authors
On this page
  1. The three command tiers
  2. Statement execution and capture
  3. Argument conversion
  4. Adapter behavior
  5. Bundled and custom adapters
  6. Feed a finite value to stdin
  7. Redirect command output
  8. Interpreter blocks
  9. Effects and plans
  10. Secrets at process boundaries

Shoal treats process invocation as a typed boundary: command arguments become argv, stdin is supplied explicitly, and completion produces an outcome containing status, bytes, timing, and optionally parsed data.

The three command tiers🔗

flowchart TD
accTitle: The three command tiers
accDescr: Shows the components and relationships described in The three command tiers.
    C["command head"] --> B{"Shoal builtin?"}
    B -- yes --> N["native implementation"]
    B -- no --> F{"forced with ^ / dynamic run?"}
    F -- yes --> X["raw external"]
    F -- no --> A{"active adapter?"}
    A -- yes --> D["declarative invocation + parser"]
    A -- no --> X
    N --> O["typed value / outcome"]
    D --> O
    X --> O
  • Builtins provide native typed behavior.
  • Adapters validate a known command surface, may pin a machine-readable output mode, declare effects, and parse output.
  • Raw externals receive argv without an adapter schema.

Skip adapter dispatch with ^ (provided no callable binding or builtin owns the same head):

git status               # adapter when active
^git status --porcelain  # raw git; bypasses adapter validation/parsing

^ also bypasses a non-callable value shadow, but it does not bypass functions, aliases, other callables, or builtins. Those resolve first. Use run("name", ...) when an executable shares one of those names.

Dynamic raw names use run:

let tool = "printf"
run(tool, "%s\n", "hello")

Statement execution and capture🔗

In an interactive terminal, a foreground external command in statement position can run through a real PTY and tee bytes directly to the terminal. This preserves colors, progress UI, prompts, signals, and terminal sizing. Shoal marks the outcome as streamed so its renderer does not duplicate the bytes afterward.

In value position, scripts, -c, and other non-interactive hosts, output is captured:

let result = (^git rev-parse HEAD)
result.ok
result.stdout
result.out

Interactive programs that truly need a terminal can also be reached through the interact surface or MCP PTY tools. Do not expect a captured value invocation to satisfy arbitrary TUI terminal assumptions.

Argument conversion🔗

Raw external argv accepts values with a clear argument representation. Paths remain path arguments; scalar values render as one argument. Collections, records, tasks, streams, errors, closures, and secrets are not silently flattened into arbitrary argv.

^printf "%s\n" (42)
^cat (path("./README.md"))

Use a typed Shoal function or adapter when a command needs schema-driven conversion and validation. Use -- to stop flag interpretation where the command grammar supports it.

Adapter behavior🔗

An adapter can describe:

  • executable name and class (cli, tui, daemon, interpreter);
  • accepted status codes;
  • top-level and subcommand parameters;
  • positional binding and short-flag aliases;
  • a fixed invocation prefix;
  • flags consumed by Shoal to protect a pinned output format;
  • output parser and optional type shape;
  • effects used by planning/policy.

For example, the bundled Git status adapter invokes porcelain v2 and parses rows with status, semantic state, path, and optional original path. The bundled ps and df adapters pin portable column layouts and return typed tables.

(git status).where(.state == "modified").map(.path)
(ps).sort_by(.cpu).reverse().first(10)
(df).where(.avail < 5gb)

An active adapter rejects unknown flags rather than passing them through and hoping its parser still matches. Prefix with ^ when using an unsupported real CLI flag intentionally.

If parsing fails, raw stdout remains available on the outcome; Shoal does not fabricate structured rows.

Bundled and custom adapters🔗

The repository currently ships 49 adapter definition files, catalogued in Reference inventory. They cover common version-control, container, cloud, package, system-inspection, archive, search, language-runtime, and data tools.

Add directories in configuration:

[adapters]
dirs = ["~/.config/shoal/adapters"]

A minimal custom file:

[cmd.example]
bin = "example"
class = "cli"
ok_codes = [0]
params = { format = "str?" }
output = { parse = "json", type = "record" }
effects = ["fs.read(cwd)"]

[cmd.example.sub.list]
invoke = ["list", "--format", "json"]
params = { all = "bool" }
output = { parse = "json", type = "table<{name: str}>" }
effects = ["fs.read(cwd)"]

Supported output parsers are json, ndjson, csv, tsv, z-records, porcelain-v2, cols, cols2, tsv-headerless, lines, kv, and none.

Malformed adapter files produce warnings while valid siblings continue loading. The host overlays the bundled directory followed by configured directories into one evaluator catalog: later commands replace same-named earlier commands, while disjoint commands remain. It emits an override warning for each replaced command. Completion retains the per-directory catalogs with the same later-wins lookup order, so execution and completion agree.

Feed a finite value to stdin🔗

feed is explicit byte plumbing:

"alpha\nbeta\n".feed(^wc -l)
["alpha", "beta"].feed(^sort -r)
{ project: "shoal", ready: true }.feed(^jq .project)

Both orientations are accepted when useful:

value.feed(command args)
(command args).feed(value)

The serialization contract is:

Valuestdin bytes
strUTF-8 bytes, no added newline
bytesraw bytes
int, float, bool, size, duration, datetime, timeinline rendered text, no added newline
list<str>newline-joined, with a trailing newline
other listcompact JSON array
record, tablecompact JSON
successful outcomesemantic out when structured, otherwise stdout

A path is a name, not its content, and cannot be fed directly:

path("input.txt").read.feed(^consumer)
path("input.bin").read_bytes.feed(^consumer)

Secrets, tasks, closures, errors, globs, and regexes are not feedable. Streams feed a process incrementally through a bounded stdin queue; each item uses the same serialization rules and is line-framed. This works for finite and live streams, and stops the upstream pump when the command exits or fails to spawn:

tail(path("input.log"), from_start: true)
  .take(100)
  .feed(^consumer)

An endless stream keeps the command alive until the command exits, the stream is bounded, or the operation is cancelled.

Redirect command output🔗

^tool > ./result.txt
^tool >> ./result.log
^tool < ./input.txt

Output redirection applies to builtins and externals. Redirect targets are evaluated and type-checked once before dispatch, so an invalid target cannot be discovered only after a command has already mutated state or spawned. Process-backed externals and adapters accept <; structured builtins reject redirected stdin before performing any effect because they do not expose a byte-stdin contract.

If captured stdout spilled to content-addressed storage, output redirection copies the full content incrementally from its verified reader rather than materializing the blob or writing only its resident preview.

Overwriting an existing file can participate in journal undo when the prior bytes fit the journal cap. Creating a new file by redirection currently records no delete-on-undo inverse. Append/overwrite behavior and limitations are in Filesystem, jobs, history, and undo.

Interpreter blocks🔗

Interpreter blocks preserve source verbatim instead of forcing it through Shoal string and command parsing:

python3 {
import json
print(json.dumps({"answer": 42}))
}.out.answer

sh '''
printf '%s\n' "$HOME"
'''

Known inline conventions are:

ToolsProgram delivery
sh, bash, zsh, fish, python, python3-c PROGRAM
node, ruby, perl, lua, Rscript, osascript-e PROGRAM
php-r PROGRAM
denoeval PROGRAM
jqprogram as positional argument
yq-o=json PROGRAM
configured interpreter adapterits declared invoke plus invoke_payload

The fallback conventions apply when an AST is evaluated without a matching adapter. In a normal configured host, an interpreter-class adapter is authoritative for the executable, invocation prefix, argument-vs-stdin delivery, output parser, and accepted status codes.

Source and data are separate channels when the interpreter supports an inline-program argument:

{"name":"shoal"}.feed(python3 {
import json, sys
print(json.load(sys.stdin)["name"].upper())
})

An interpreter whose program itself is delivered on stdin cannot simultaneously receive fed data on that same stdin; the evaluator reports an error instead of concatenating two protocols.

sh { ... } is the escape hatch for a small, intentional POSIX pipeline. Shoal cannot infer detailed filesystem/network effects inside opaque shell source, and undo cannot reverse those mutations automatically.

Effects and plans🔗

Every process spawn has at least a spawn effect. Adapters can add precise filesystem/network effects derived from typed parameters. Raw commands and interpreter blocks are necessarily more opaque.

plan { git push origin main }
explain("git status")

Planning does not make an adapter truthful by itself—the adapter declaration is part of the trusted integration surface. Review custom effect declarations before using them for agent authorization.

Secrets at process boundaries🔗

Secret values deliberately fail ordinary argv, interpolation, feed, and serialization. Policy-aware integrations can inject them in explicit contexts such as an HTTP header. This prevents accidental log/render leaks but does not protect a secret after an authorized child receives it.

For remote agent execution, combine secret handling with principal identity, capability approval, and leash policy; see Agents, kernel, and MCP.

Type to search every guide↑↓ navigate ↵ open esc close
Diagram