add SIP - packaging components

[michelleN]

No description provided.

[itowlson]

Good stuff, thanks for getting this under way!

[itowlson]

I like this emphasis. The component cannot grant itself the permissions because we can't trust it, but we need to know what it wants from us.

(no action, just wanted to applaud)

[itowlson]

Recognises it by the manifest version declaration?

[michelleN]

by the file name and the manifest version declaration. I'll make that clearer.

[itowlson]

PEDANTRY CORNER: spin up -f bare.wasm does work. (but your point is still right of course)

[itowlson]

Correct to say a standalone component cannot have dependencies?

[fibonacci1729]

For now at least. Not forever.

[michelleN]

Noted in the future work section

[itowlson]

I'm getting a bit confused across OCI and wkg at this point, sorry. The draft uses OCI terminology and references in some places, and wkg in others (which is what is mentioned in the preamble).

[michelleN]

updated the wording to make it less confusing

[itowlson]

spin registry pull with applications puts them in the Spin cache I believe.

[thorstenhans]

Great stuff @michelleN

Although I am on the same page because it would streamline spin.toml and ensure the top-level application manifest remains somewhat "short", I went through it with some common middlewares and want to share an alternative approach:

What about keeping everything in spin.toml, because:

• dependencies are pre-built by component authors and consumed as .wasm binary (no-src) -> integration with spin build should be opt-in
• component source is available, but in different repository
• users may want to use variables defined on application scope (including interpolation)
• The component author is obviously in charge of documenting the needs of their components
  • IMO: Authoring dependency components at the same time / in the same context as implementing its consumer application is rare. Looking at other ecosystems that provide composition, it's way more common to consume these. If devs happen to identify good "sharable" component candidates while building an "app", it's pretty common to generalize the component, and implement/distribute it independently from the actual application lifecycle

I drafted a bit and ended up with something like this (not really different from the current approach, but adds more granular variable handling).

# ...

[variables]
app_name = { default = "My Spin App" }

[[trigger.http]]
route = "/api/..."
component = "api"

[component.api]
source = "api/target/wasm32-wasip2/release/api.wasm"
allowed_outbound_hosts = []
key_value_stores = ["default"]
dependencies = ["cors", "correlation-id", "header-injection"]

[component.cors]
source = { path = "../cors/target/wasm32-wasip2/release/cors.wasm" }

[component.cors.variables]
allowed_methods = "POST,GET,PUT,DELETE"
allowed_origings = "*"
# allowed_headers is unset (component might have smart defaults)

[component.correlation-id]
source = { path = "../cors/target/wasm32-wasip2/release/cid.wasm"} 

[component.correlation-id.variables]
header_name = "X-CORRELATION-ID"

[component.header-injection]
source = {path = "./components/hi/target/wasm32-wasip2/release/hi.wasm" }

[component.header-injection.build]
workdir = "./components/hi"
command = "cargo build --target wasm32-wasip2 --release"

[component.header-injection.variables]
header_name = "X-APP-NAME"
header_value = "{{ app_name }}"

[fibonacci1729]

Suggested change

	| `environments_variables` | Environment variables the component needs. Each entry is a bare name, or `{ name, default }`. |
	| `environment` | Environment variables the component needs. Each entry is a bare name, or `{ name, default }`. |

[fibonacci1729]

I maybe wouldn't say these mirror the capability fields in spin.toml -- it's likely, at least some of these, will diverge from the schema associated with the name used in spin.toml (e.g. environment is divergent).

[fibonacci1729]

We should probably not allow glob patterns here; globs are relative to the consuming app's directory and the dependency component has no idea what's there. What the component actually knows is the guest path it expects to read so i'd consider restricting this to { destination }.

[itowlson]

That's an interesting thought @fibonacci1729. I like the idea of being able to say "I expect to find whatever files you want me to use at /foo, but it's up to you where you source them."

There is another use case, but I am not sure about generality: static assets. Here the component / dep / middleware doesn't want to put it on the app to supply files, because it already has specific ones it wants to use. E.g. the HTML, JavaScript and CSS for the "authorisation failed" page, or a CSV file of internal data. However, while this (I think) makes sense for a local component (a TOML file), we can't currently publish it as a standalone Wasm binary.

[fibonacci1729]

I think for static assets that a lib dev would require (where they don't want to put the burden on the app to supply) we could, at least initially, document/encourage them to embed their assets into the binary.

[lann]

We would also have the option of embeddeding static assets in their own custom sections (e.g. named spin-file:path/to/file).

[fibonacci1729]

I think we'll need to enable push to embed metadata since build is optional.

[fibonacci1729]

@thorstenhans we can't keep everything in the spin.toml because this SIP explicitly covers the producer side, i.e.: How do I, as the author of cors build it, version it, publish it, and describes what it needs -- from it's own repo, where there is no application to hang a spin.toml off. What you're describing in your alternative proposal is the consuming side.

Authoring dependency components at the same time / in the same context as implementing its consumer application is rare... it's pretty common to generalize the component, and implement/distribute it independently from the actual application lifecycle

Strongly agree and that's the argument for a standalone manifest. The moment you "generalize and distribute independently," you need an independent build/version/publish story, which is what component.toml provides.

[itowlson]

@michelleN What's the relationship between this and #3664?

[michelleN]

@itowlson closed #3664 in favor of this PR

[michelleN]

Would appreciate another round of eyes on this @itowlson @fibonacci1729

[itowlson]

@michelleN Are we reviewing the implementation alongside the SIP, or is the implementation WIP that was just more convenient to do on the same branch?

[itowlson]

Basically looks good, although I remain a bit muddled about the wkg interop story. But really this will be very cool to have. I really like the requires approach and the way that propagates through the binary!

[itowlson]

super mega nit: promote this and other H3s to to H2 (and H4s to H3). These chunks are the top levels of the SIP, not mere subsections of Proposal.

[itowlson]

Possible thing, or maybe future idea: a site or url or docs field with the semantics of "go here to learn about this component." Not sure

[karthik2804]

I would imagine the repository would have the same semantics right?

[itowlson]

Not necessarily: the author might have a docs site or configuration guide. The semantics are certainly different: "go here for source code" vs "go here to learn." (Cf. docs.rs which calls this "homepage" vs "repository.") But yes, the first line of the repo README could point the docs/guide site, which is why it's definitely not a must-have on day one.

[itowlson]

For the source code, the licence is usually a licence file in the repo. For the binary distribution, I'm not sure if it makes sense? (e.g. what does it mean to GPL a binary) (But if this is a known and understood thing with binaries then no worries.)

[itowlson]

Should standalone components support profiles, like apps do? I am not sure whether it makes sense from a distribution POV, but maybe useful for "I want a debug build" nevertheless?

Maybe it should exist but be more limited than app profiles, e.g. only the build-command and source fields?

[itowlson]

Is the idea here for if I am building a component only for inclusion in an app manifest (and that will never have any independent existence)? I'm tempted to make version mandatory so that when someone comes to publish they don't get a nasty surprise, but I can see this use case.

(ETA: your note on watch fields suggests "for inclusion in an app manifest" is not an intended use case in this iteration)

[itowlson]

You talked earlier about republishing components from a registry etc. In such a case the component won't have a build command. So it seems like spin build needs to work in the absence of a build command, just to write the custom section?

[itowlson]

So a component published this way will not be usable via wkg, am I reading that right? We are defining separate conventions outside the standards? If so, I am wary.

Oh wait I see later on (in the "Alternatives Considered" section) you say that this is wkg. I am confused because I thought wkg used the package name rather than requiring a separate OCI reference. Perhaps worth expanding the SIP to talk more about how these components work with wkg?

[tschneidereit]

I've only looked at the SIP, not any of the implementation, but the SIP looks great! It largely subsumes #3073, too, which I'm very pleased by :)

I also really like the [requires] table! I had tried to come up with a good solution to that in #3073, but failed.

I think the feedback that follows is pretty much entirely about making this SIP fully subsume #3073, because I don't think we'll want multiple separate approaches to modularizing manifests. Same as individual crates from a cargo workspace can be published without publishing the entire workspace, I think we could, with only small tweaks, make this serve the same purpose as a cargo workspace and support publishing of individual components.

As such, I think would be good to put a bit more work into either moving integration with spin.toml and its [dependencies] support into this SIP itself, or to at least sketching it out sufficiently to make sure it'll hold up. What does it look like to refer to a local component by path? How can it be used as a dependency to another component?

#3073 had some small amount of that in the shape of supporting things like version.application = true as an equivalent to Cargo.toml's version.workspace = true. That I think transfers pretty cleanly. And perhaps using it in a spin.toml is as simple as this?

[component.my-packaged-component]
path = "relative/path/to/dir/containing/the/manifest"

I also think that we shouldn't postpone reasoning through how [requires] can be used as more than documentation: otherwise we run the risk of introducing slight impedance mismatches and having to bump manifest versions.

My thinking on that is, very roughly, that the [requires] table can serve both to declare what must be provided, and to optionally provide defaults. When using a component as a dependency, any requirement that doesn't have a default value must have the value provided as part of the dependency declaration. (And anything with defaults can have them overridden, of course.)

Speaking of dependencies, it might also be good to put a bit of thought into nested dependencies before moving that topic to future work. I initially thought "why not just say that it works just as in spin.toml, with the additions described above." Unfortunately I realized that we really don't want to bury the [requires] table and the provided values and make them invisible in the top-level manifest: otherwise it's trivial to hide allowed_outbound_hosts = ["*://*"] in a nested component.

Which in turn makes me realize that providing default values for [requires] runs that risk more generally, even with no nesting 🤔

One approach could be to not accept defaults ... by default. Instead, a default would have to be accepted explicitly with something like

[my-comp.dependencies.my-dep]
path = "my-dep"
requires = {
  allowed_outbound_hosts.default = true
}

(Yay multi-line nested tables or what those curly brace things are called!)

I still don't particularly like this though, because it still means that the default value is invisible, and might be changed behind the scenes. We might be able to handle that on the tooling side, though, by remembering the current default value ... somewhere ... and warning if it changes.

Alternatively, there is no .default = true. Instead, a default is never actually used directly, but serves as the input to a suggestion:

# spin build
Error: `my-comp`'s dependency `my-dep` requires `allowed_outbound_hosts` to be configured, 
but no configuration is provided.
Accept the default value of `["https://github.com", "https://api.github.com"]`?
[y/n]

Hitting y would add allowed_outbound_hosts = ["https://github.com", "https://api.github.com"] to the dependency's definition.

Problem with that approach: allowed outbound hosts are something the component typically really needs, so if the default changes, chances are the setting in spin.toml should also change. But we can't show a warning about that situation, because for all we know the disparity is deliberate because the default value wasn't acceptable.

So yeah, this would be good to think through a bit more 😬