Repository navigation
feat(graph): implement fan-in joins and graph correctness - #15
Merged
Merged
Conversation
Every built-in capability runtime receives the same two inputs: the params bound to the capability invocation and the capability's own resolved configuration. Six built-ins each merged them differently - set let the configuration win while http, command, and delay let the params win - and each coerced values in its own way, so a number sent where a string was expected silently became text. This adds one package that owns the merge rule and the value accessors: - params override configuration, because a param is an explicit argument at the call site and a configuration value is the ambient default declared alongside the capability; - typed accessors report a precise error instead of coercing, so a duration can no longer arrive as a nanosecond count and a URL can no longer be sent with a missing scheme; - one output-format convention (json | text) shared by the built-ins that produce a payload. The engine-owned RuntimeConfig is deliberately not part of this input. It instructs N.O.R.E. how to drive the capability and is never sent to the capability runtime, so a built-in must not read its options from it.
The command built-in reported a cancelled execution as a program that had finished successfully. exec.CommandContext kills the process when the context is cancelled, so the error it returns is the kill signal rather than the cancellation. The built-in only understood exec.ExitError, so a killed process produced neither an exit code nor an error: it reported exit code zero and success. The same gap hid a program that never started at all - one that could not be found, or was not executable - which was reported as a clean run. It now distinguishes three outcomes: - a cancelled execution reports the cancellation, because the program was stopped by a decision made elsewhere and produced no result; - a program that could not be started reports the failure to start; - a program that ran reports its exit code, whatever it was. The tests use the test binary itself as the program under test, re-executed with a marker argument, so they do not depend on a program being installed on the machine.
…lt-ins Which capability runtimes exist, what inputs they accept, what they produce and what they refuse was decided in six places at once: the runtime's registration table, the compiler's skip list, the CLI's help text, the TypeScript SDK's exports, and the documentation. Nothing kept them in step, so a built-in could be documented but not registered, or registered but not resolvable. The contract is now declared once, in the shared core package, and the runtime registers exactly what is declared. A declared built-in with no implementation is reported as a defect rather than leaving the built-in silently unavailable, and an implemented built-in that was never declared is reported too. Three built-ins are retired with it, because each was a mock rather than a capability: - ai returned a canned string instead of calling a provider; - delay only waited, which is a sequencing concern and not a capability; - log duplicated what the runtime already records. A reference to a retired built-in now fails with an explanation of what happened and what to use instead, rather than being sent to the package resolver where it would be reported as a missing package. The same error names the built-ins that do exist when a name in the core namespace is not one of them. The shared package also gains the security policy that bounds the built-ins. The ceilings are compiled in, so a project's configuration can narrow them but never widen them; the policy a project configures is resolved against those ceilings and a configuration that asks for more is refused. The registry also stops closing a runtime twice. A built-in is reachable under its canonical name and under its legacy bare name, so closing it once per name called its shutdown twice.
Each built-in is a capability with its own input contract, its own output shape and its own security boundary. Keeping them in one package made those boundaries invisible: a reader of the http runtime had to assume the file next to it did not also change how the runtime read its input. Each built-in now has its own package, and all of them read their input through the shared contract introduced with them. The set runtime changes behaviour as a result. It used to let the capability's configuration override the params bound to the invocation, which is the opposite of every other built-in and of the shared rule: a param is an explicit argument at the call site, and a configuration value is the ambient default declared alongside the capability. It now lets the param win. The command runtime stops coercing its arguments. An argument that was not a string was formatted with %v, so a number in an argument list became the text "42" and a boolean became "true" without anyone asking for it. An argument that is not a string is now an error.
The built-in capability runtimes reach outside the runtime, so what they may reach is configuration rather than convention. The new builtins section states the filesystem roots, the allowed command executables, the allowed HTTP hosts, and the size ceilings each built-in runs under. The section is resolved through the shared policy, so it can only narrow the limits compiled into the runtime: a project configuration file is part of the assembly's own input, and an input must not be able to widen what a capability can reach.
The built-in capability runtimes need to know what they may reach, and the policy that states it had no way to reach them. It now travels with the runtime: the process boundary decodes it, resolves it against the ceilings compiled into the binary, and hands it to the registry that constructs the built-ins. Resolving it at the boundary is what makes it trustworthy: a policy that arrived over the command line cannot widen what a capability may reach, and one that cannot be applied stops the runtime instead of being ignored. The http and command built-ins apply the limits the policy carries: the request timeout, the request and response body caps, and the bound on the output a program may produce. The command output is bounded while it is produced rather than after, so a program that writes without limit cannot exhaust the runtime's memory. A directly invoked capability runtime also receives a logger now. The direct-invocation path has no event bus, and a nil logger turned every diagnostic a capability runtime wrote into a panic.
The project configuration states what the built-in capability runtimes may reach, and the runtime is what enforces it. The policy therefore travels with the runtime, on the command line the project runtime is already started with. A direct invocation is project-independent, so it runs under the compiled-in ceilings.
The http built-in opened a socket to any address an assembly asked for. A capability whose url comes from an assembly's input could therefore reach the cloud metadata service, an internal API that trusts the network, or the runtime's own control surface. The built-in now refuses an address that is not publicly routable unless the policy names the host it may reach. The check happens when the connection is made, on the addresses the host actually resolves to, rather than on the name in the url: a host name is a claim about an address, and the claim is verified where it is used. The rest of the request is bounded as well. Redirects are not followed unless the capability asks for it, because a redirect is a second request to a host the caller never named. When they are followed, every hop is checked against the same rules. Retries are not made unless the capability asks for one, and only for methods that are idempotent: a retried POST may repeat an effect the server already performed. The response body is refused when it exceeds the limit rather than truncated, and the response headers are part of the answer, because a caller that cannot see them cannot read a rate-limit header. The default format is JSON, and a body that is not JSON is refused in that format rather than handed over as a string the caller cannot bind. A request may ask for a shorter timeout than the policy allows, but not for a longer one. The private-network restriction is lifted for a request the operator authored themselves. The allowance is not configurable from a project configuration file, because a project configuration is part of the assembly's own input.
The command built-in ran whatever program it was handed, in whatever directory it was handed, with the whole environment of the runtime, and without a deadline. The run is bounded now. The program is resolved against the program path it will run with, so the program that is checked is the program that runs, and the check is made on the resolved path rather than on the name. When the policy names the programs that may run, a program that is not named is refused. The working directory is confined to the permitted roots, and a symbolic link inside a root does not grant reach outside it. The program inherits an allowlisted environment rather than the environment of the runtime: a program that inherited everything would inherit the credentials of whoever started the runtime. The variables the capability asks for are added to it. The run has a deadline, taken from the policy. A capability may ask for a shorter one, never for a longer one. The program runs in its own process group and the whole group is stopped when the run ends, so a child the program started does not outlive it. A program that failed is a failed capability. The result of a failure is still available to a capability that asks to inspect it, and it says whether the program was stopped by its deadline. The reach of the built-ins follows the assembly: the runtime states the project root it was started for, and that root is where a command runs by default.
The http built-in declared a contract it did not implement: the query parameters, the content type, the credentials, and the retry policy were documented as accepted inputs, and the content type and the duration as produced outputs. An author who followed the documentation got a request that silently ignored half of it. The capability now accepts what it declares. The query parameters are appended to the URL. The content type is sent with the body. The credentials are applied to the request rather than to the URL, so they never end up in a log line or in an error that names the URL. The retry policy names how many attempts are made, how long the runtime waits between them, and which methods may be retried. The answer carries the content type of the response and how long the request took, which is what the contract says it carries. A nested object is read with the same rules as the outer values, so a built-in does not have to know how the fields of a nested object are typed.
An assembly could not read the files it was written to work with. It could call out to the network and it could run a program, but the files next to it were out of reach. The filesystem built-in is that reach: read a file, list a directory, and report what a path is. It reads only. Writing, deleting, and moving files are not part of it, and the descriptor says so. The reach is the project root plus the directories a project added to it. A path that resolves outside them is refused, and a symbolic link inside them does not grant reach beyond them. The content a single read returns is bounded, and a file above the bound is refused rather than shortened: a caller cannot tell a shortened file from a complete one. The entries a single listing returns are bounded too, and a listing that reaches the bound says it was shortened. A file that is not JSON is refused in the JSON format rather than handed over as a string the caller cannot bind, and the text format is how a caller asks for it as text. A path that is not there is an answer, not a failure: the caller asked what is there.
The set built-in returned its merged value set and ignored the format every other built-in honors. A capability that asked for the text format received a value set anyway, and the format it asked for was handed back to it as one of the values of the set. The set now reads the format and keeps it out of the result: the format is an instruction to the built-in rather than a value of the set. The text format keeps the names of the set and represents the values as text, so the shape of the result does not change between the formats and a downstream capability binds the same names either way.
An author who wanted to know what a built-in accepts and what it guarantees had to read the source or the documentation. The contract is compiled into the runtime, so the runtime can report it. `neuron builtins` lists the built-in capability runtimes, and `neuron builtins <name>` reports the contract of one of them: the inputs it accepts, the outputs it produces, and the boundaries it enforces. The verbose form explains each of them. The command needs neither a running runtime nor a project, because the contract does not depend on either. A name that names a built-in that was removed is reported as removed, with the replacement, rather than as a name that does not exist.
… SDK An assembly that used a built-in capability runtime had to declare the runtime by hand and had no types for it. The SDK exported one of them, and it no longer existed. The SDK now exports every built-in: set, http, command, and filesystem. Each export takes the name of the capability it runs, so two capabilities that use the same built-in do not collide, and each returns a definition with the contract of the built-in already applied. The http contract now matches what the runtime publishes: the query parameters, the content type, the credentials, and the retry policy are accepted, and the response headers, content type, and duration are produced. A built-in capability runtime runs in-process and is never resolved or installed, so it has no version and no registry. The manifest said "latest" from the "local" registry, which stated that a package that does not exist has a version. It now says the name alone. The delay export is gone: it only waited, which is a sequencing concern rather than a capability. The README states what replaced it.
BUILTINS.md is the authoritative reference for the four built-in capability runtimes: set, http, command, and filesystem. It lists the parameters, the results, and the boundaries the runtime enforces for each one. ARCHITECTURE.md updates the core built-in table to the current set. MIGRATION.md adds a section on the TypeScript SDK's built-in exports and the removal of delay. GETTING_STARTED.md already used only the surviving built-ins and needs no edit; MODULES.md expands the built-in section with the contract summary and the `neuron builtins` command.
Make the storage contract backend-neutral and explicit: add a driver registry and factory, move key construction out of SQLite, add sentinel errors and transactional storage with contract tests, and route the runtime through storage.Open. Execution persistence now writes before memory, distinguishes absence from backend/decode failures, and uses revision compare-and-swap with caller contexts. Instance metadata, API handlers, scheduler, engine, events, and assembly registration propagate persistence errors instead of reporting stale or partial state. Assembly resolution honors content hashes, and event deletion plus assembly registration are atomic.
Document the storage contract, transaction boundary, execution guarantees, durability modes, current metadata-only restore behavior, and same-version retention limitation. Update the storage provider migration note.
… state, scheduler
…an-in support - Remove the rejection of capabilities with multiple incoming bindings in the compiler - This allows the TypeScript SDK and YAML authoring surfaces to declare fan-in graphs - The join policy (JoinPolicyAll) is already stored per-binding and used by the scheduler - This change enables the core execution model change for fan-in as described in Phase 3
- When a capability completes, record the completion in the execution's join state - For targets with multiple predecessors (>1), only schedule when ALL predecessors have completed - For targets with a single predecessor, use the original immediate-scheduling behavior - The RecordCompletedBinding function is idempotent and handles the join state tracking - This implements the explicit join policy respectance described in R3-19 and R3-20
- R3-4: Add cross-binding target field conflict detection in compiler Reject two bindings writing to same target param path at compile time - R3-6: Add ExecutionBlueprint.Has() for deterministic graph identity Hash includes predecessors and join policy for durable execution - R3-21: Fix in-flight accounting in CompleteCurrentAndSchedule Only decrement inFlight when nextCapabilityCount > 0 - R3-16: Reject detached capability fan-in at build time Detached branches cannot participate in join - R3-10: Add duplicate target path validation in TypeScript SDK Reject two bindings writing to same param field
- nore/cmd/nore/main.go: align import tag whitespace - nore/internal/execution/execution.go: align done channel declaration - nore/internal/instance/manager.go: remove trailing whitespace after storage.Key() - nore/internal/types/blueprint.go: align struct tag whitespace - scripts/script.sh: add gofmt -l check to CI pipeline
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Implements graph-based capability composition and fan-in execution across the Assembly model, compiler, scheduler, and TypeScript SDK. Capability dependencies are derived from data references; shared predecessors can fan out to parallel branches, and join behavior is represented explicitly in the execution blueprint. The change also adds graph validation for conflicting target parameter paths and detached branches, deterministic graph identity that includes predecessors and join policy, and fixes scheduler in-flight accounting.
The branch also carries the supporting platform changes included in its 28-commit diff: built-in capability implementations and policy definitions, CLI/configuration updates, storage-provider and migration work, API/runtime persistence changes, SDK updates, tests, and documentation.
Why
The previous composition model relied on sequence/parallel composition structures and did not provide a complete, explicit model for multiple upstream capabilities converging on a dependent capability. A graph-based model makes dependencies derive from the actual expressions consumed by each capability, while compile-time validation prevents ambiguous or invalid joins from reaching execution.
How it was verified
Checklist
git commit -s)