feat(origin): hosted .kpar packages by download_url and checksum qualifiers

README.md

@@ -10,7 +10,8 @@ what package managers install.
 ## The contract, on one screen
 
 - **Identity** — an ECMA-427 Package-URL of type `generic` with the Annex B.3.5 `vcs_url`
-  qualifier: `pkg:generic/<name>?vcs_url=<vcs>+<transport>://<host>/<path>[@<version>][#<subpath>]`,
+  qualifier for a repository (or `download_url` for a hosted `.kpar`, under "Other sources"):
+  `pkg:generic/<name>?vcs_url=<vcs>+<transport>://<host>/<path>[@<version>][#<subpath>]`,
   the qualifier percent-encoded (`+` → `%2B`, `/` → `%2F`, `@` → `%40`, `#` → `%23`, `:` kept), e.g.
   `pkg:generic/OpenSysML-Extensions-Library?vcs_url=git%2Bhttps:%2F%2Fgithub.com%2FOpen-MBEE%2FOpenSysML-Extensions-Library%401.0.0`.
   `<vcs>` is `git`, `svn` or `hg`, `<transport>` one of Annex B.3.5's (`git`, `https`, `http`,
@@ -44,7 +45,15 @@ what package managers install.
   `.kpar` SHA-256, and path-traversal, symlink and size bounds on every archive and export.
 - **Other sources** — a local directory (editable, read in place), a local `.kpar`, a remote
   `.kpar` URL, OMG standard-library IRIs (`https://www.omg.org/spec/{KerML,SysML}/…/*.kpar`,
-  recorded but never installed: every reader bundles them).
+  recorded but never installed: every reader bundles them). A `.kpar` hosted at any http(s) URL
+  — a release asset, a repository manager's export — is also a Package-URL with the Annex B.3.5
+  `download_url` qualifier and an optional `checksum`:
+  `pkg:generic/<name>[@<version>]?download_url=<url>[&checksum=<algorithm>:<hex>[,…]]`, both
+  percent-encoded (`pkg:generic/tools?checksum=sha256:9f86…&download_url=https:%2F%2Fdist.example.org%2Ftools.kpar`).
+  The archive is fetched as a plain `.kpar` URL is (netrc, redirects, size bound), the declared
+  checksums are verified before it is opened (a mismatch installs nothing), `<version>` is the
+  archive's own and is held to `@<version>` / `versionConstraint`, and the lock's `sha256` is the
+  archive digest.
 
 Details — the grammar, the svn `tags/` convention, the tree-hash algorithm and the lock format —
 are in [docs/packages.md](docs/packages.md).

changes/unreleased/download-url-archives.added.md

@@ -0,0 +1 @@
+- A `.kpar` hosted at any http(s) URL can be named by an ECMA-427 Package-URL with the Annex B.3.5 `download_url` qualifier and an optional `checksum` (`pkg:generic/<name>[@<version>]?download_url=<url>[&checksum=<algorithm>:<hex>[,…]]`, percent-encoded; md5, sha1, sha224, sha256, sha384, sha512 and the sha3 family). The archive is fetched as a plain `.kpar` URL is, every declared checksum is verified before it is opened — a mismatch installs nothing — and it installs as `lib/<publisher>-<name>_<version>.kpar` with the archive's SHA-256 in the lock; `vcs_url` and `download_url` exclude each other, and `syspi add` writes the canonical form into `.project.json`.

cmd/syspi/main.go

@@ -45,6 +45,9 @@ ECMA-427 Package-URL (vcs: git, svn, hg; transports per Annex B.3.5, plus file):
 Accepted as input and normalised to it:
   pkg:git/<host>/<path>[@<version>][#<subpath>]
   <vcs>+<transport>://<host>/<path>[@<version>][#<subpath>]
+A .kpar hosted at an http(s) URL is its IRI, or a Package-URL with the Annex B.3.5 download_url
+qualifier and an optional checksum (<algorithm>:<hex>, verified before the archive is opened):
+  pkg:generic/<name>[@<version>]?download_url=https:%2F%2F<host>%2F<path>[&checksum=sha256:<hex>]
 Run "syspi <command> -h" for the flags of a command.
 `
 

docs/packages.md

@@ -50,6 +50,41 @@ host (Annex B.3.5), a transport outside the lists above, `bzr` or any other VCS,
 file — a `.kpar` — as KerML 1.1 §10.3 Table 12 describes `resource`; it is never cloned, and
 `user:password@` in it is refused (credentials belong in the netrc, never in `.project.json`).
 
+### Hosted archives
+
+A `.kpar` hosted at an http(s) URL is named either by that URL (a plain IRI, dereferenced as a
+project interchange file) or by a Package-URL of type `generic` carrying the Annex B.3.5
+`download_url` qualifier, with an optional `checksum`:
+
+```
+resource     = "pkg:generic/" name [ "@" version ] "?" qualifiers
+qualifiers   = [ "checksum=" encode(checksums) "&" ] "download_url=" encode(url)   (canonical order; any order accepted)
+url          = "http" [ "s" ] "://" host "/" path [ "?" query ]   (no user:password, no "#")
+checksums    = checksum *( "," checksum )                          ("," encoded as %2C)
+checksum     = algorithm ":" hex                                   (lower case; Annex B.3.5)
+algorithm    = "md5" | "sha1" | "sha224" | "sha256" | "sha384" | "sha512"
+             | "sha3-224" | "sha3-256" | "sha3-384" | "sha3-512"
+name         = the archive's .project.json "name"
+version      = semantic version the archive's .project.json must carry; an exact constraint
+```
+
+`vcs_url` and `download_url` exclude each other on one purl; `checksum` is only accepted with
+`download_url` (a repository is held by its tree hash). Any URL works — a release asset, a
+repository manager's export — and it need not end in `.kpar`; what it serves must be one. The
+archive is downloaded as a plain `.kpar` URL is (netrc, redirects, the size bound), every
+declared checksum is verified against the downloaded bytes before the archive is opened — a
+mismatch is an error and nothing is installed — and an algorithm this tool cannot compute is
+refused rather than skipped. The version is the archive's own `.project.json` `version`, checked
+against `@<version>` and `versionConstraint` as a plain IRI's is, and the purl `<name>` must be
+the archive's `name`. The archive installs as `lib/<publisher>-<name>_<version>.kpar`; the lock
+records the version-free identity `pkg:generic/<name>?download_url=encode(url)` as `resource`,
+the URL as `source` and the archive's SHA-256 as `sha256`, whatever algorithms the purl declares.
+`syspi add` writes the canonical form (version after the name, `checksum` before `download_url`,
+items sorted by algorithm) into `.project.json`; a later run with a declared checksum that
+disagrees with the lock or the installed archive fails. Example:
+
+- `pkg:generic/tools@3.0.0?checksum=sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08&download_url=https:%2F%2Fdist.example.org%2Freleases%2Facme-tools_3.0.0.kpar`
+
 ## What a package is
 
 The tagged tree — the repository root, or `#<subpath>` within it — is the project directory. It
@@ -138,7 +173,7 @@ and the lock as they were. A lock entry wins over the constraint in `.project.js
 .project.lock.json
 lib/
   mbse4u-sysmod_5.1.5/        exported tree of a tagged repository
-  acme-tools_3.0.0.kpar       archive source, verified and copied as is
+  acme-tools_3.0.0.kpar       archive source (a .kpar path, URL or download_url purl), verified and copied as is
 ```
 
 The directory name is `<publisher>-<name>_<version>` (`<name>_<version>` without a publisher),

internal/digest/checksum.go

@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: Apache-2.0
+
+package digest
+
+import (
+	"crypto/md5"
+	"crypto/sha1"
+	"crypto/sha256"
+	"crypto/sha3"
+	"crypto/sha512"
+	"encoding/hex"
+	"fmt"
+	"hash"
+	"io"
+	"os"
+	"sort"
+	"strings"
+)
+
+// Checksum is one ECMA-427 Annex B.3.5 checksum item: a lower-case algorithm
+// name and the lower-case hex of its value.
+type Checksum struct {
+	Algorithm, Value string
+}
+
+// String is the item as a checksum qualifier spells it, <algorithm>:<hex>.
+func (c Checksum) String() string { return c.Algorithm + ":" + c.Value }
+
+// algorithms are the Annex B.3.5 algorithms this tool can compute.
+var algorithms = map[string]func() hash.Hash{
+	"md5":      md5.New,
+	"sha1":     sha1.New,
+	"sha224":   sha256.New224,
+	"sha256":   sha256.New,
+	"sha384":   sha512.New384,
+	"sha512":   sha512.New,
+	"sha3-224": func() hash.Hash { return sha3.New224() },
+	"sha3-256": func() hash.Hash { return sha3.New256() },
+	"sha3-384": func() hash.Hash { return sha3.New384() },
+	"sha3-512": func() hash.Hash { return sha3.New512() },
+}
+
+// Algorithms lists the supported checksum algorithms, sorted.
+func Algorithms() []string {
+	out := make([]string, 0, len(algorithms))
+	for a := range algorithms {
+		out = append(out, a)
+	}
+	sort.Strings(out)
+	return out
+}
+
+// ParseChecksums reads a checksum qualifier value, <algorithm>:<hex> items
+// separated by commas (Annex B.3.5), normalised to lower case and sorted by
+// algorithm. An algorithm this tool cannot compute is an error: a declared
+// checksum that cannot be verified is not one that passes.
+func ParseChecksums(value string) ([]Checksum, error) {
+	var out []Checksum
+	seen := map[string]string{}
+	for _, item := range strings.Split(value, ",") {
+		algo, hexValue, ok := strings.Cut(item, ":")
+		algo, hexValue = strings.ToLower(strings.TrimSpace(algo)), strings.ToLower(strings.TrimSpace(hexValue))
+		if !ok || algo == "" || hexValue == "" {
+			return nil, fmt.Errorf("checksum %q is not <algorithm>:<hex>", item)
+		}
+		newHash, known := algorithms[algo]
+		if !known {
+			return nil, fmt.Errorf("checksum algorithm %q is not supported (%s are)", algo, strings.Join(Algorithms(), ", "))
+		}
+		raw, err := hex.DecodeString(hexValue)
+		if err != nil {
+			return nil, fmt.Errorf("checksum %q: the %s value is not hex", item, algo)
+		}
+		if want := newHash().Size(); len(raw) != want {
+			return nil, fmt.Errorf("checksum %q: a %s value is %d hex digits, not %d", item, algo, 2*want, len(hexValue))
+		}
+		if prev, dup := seen[algo]; dup && prev != hexValue {
+			return nil, fmt.Errorf("checksum declares two different %s values", algo)
+		} else if dup {
+			continue
+		}
+		seen[algo] = hexValue
+		out = append(out, Checksum{algo, hexValue})
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].Algorithm < out[j].Algorithm })
+	return out, nil
+}
+
+// FormatChecksums is the qualifier value for the items, comma-separated.
+func FormatChecksums(sums []Checksum) string {
+	parts := make([]string, len(sums))
+	for i, c := range sums {
+		parts[i] = c.String()
+	}
+	return strings.Join(parts, ",")
+}
+
+// Sums computes the file's digest under each item's algorithm in one pass,
+// keyed by algorithm.
+func Sums(path string, sums []Checksum) (map[string]string, error) {
+	out := map[string]string{}
+	if len(sums) == 0 {
+		return out, nil
+	}
+	f, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer f.Close()
+	hashes := map[string]hash.Hash{}
+	var writers []io.Writer
+	for _, c := range sums {
+		if _, dup := hashes[c.Algorithm]; !dup {
+			hashes[c.Algorithm] = algorithms[c.Algorithm]()
+			writers = append(writers, hashes[c.Algorithm])
+		}
+	}
+	if _, err := io.Copy(io.MultiWriter(writers...), f); err != nil {
+		return nil, err
+	}
+	for algo, h := range hashes {
+		out[algo] = hex.EncodeToString(h.Sum(nil))
+	}
+	return out, nil
+}
+
+// Verify checks the file's bytes against every declared checksum and reports
+// the first that does not match.
+func Verify(path string, sums []Checksum) error {
+	got, err := Sums(path, sums)
+	if err != nil {
+		return err
+	}
+	return Compare(sums, got)
+}
+
+// Compare holds the declared checksums to digests computed from the archive,
+// keyed by algorithm; an algorithm absent from got is not checked.
+func Compare(sums []Checksum, got map[string]string) error {
+	for _, c := range sums {
+		if g, ok := got[c.Algorithm]; ok && g != c.Value {
+			return fmt.Errorf("checksum mismatch: the resource declares %s %s, the archive has %s", c.Algorithm, c.Value, g)
+		}
+	}
+	return nil
+}

internal/digest/checksum_test.go

@@ -0,0 +1,55 @@
+// SPDX-License-Identifier: Apache-2.0
+
+package digest
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestParseChecksums(t *testing.T) {
+	sha := strings.Repeat("ab", 32)
+	md := strings.Repeat("CD", 16)
+	got, err := ParseChecksums("sha256:" + sha + ",md5:" + md + ",sha256:" + strings.ToUpper(sha))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if FormatChecksums(got) != "md5:"+strings.ToLower(md)+",sha256:"+sha {
+		t.Fatalf("got %s", FormatChecksums(got))
+	}
+	for _, tc := range []struct{ in, want string }{
+		{"sha256", "is not <algorithm>:<hex>"},
+		{"sha256:", "is not <algorithm>:<hex>"},
+		{":" + sha, "is not <algorithm>:<hex>"},
+		{"crc32:deadbeef", `algorithm "crc32" is not supported`},
+		{"sha256:xyz", "is not hex"},
+		{"sha256:abcd", "a sha256 value is 64 hex digits, not 4"},
+		{"sha256:" + sha + ",sha256:" + strings.Repeat("00", 32), "two different sha256 values"},
+	} {
+		if _, err := ParseChecksums(tc.in); err == nil || !strings.Contains(err.Error(), tc.want) {
+			t.Errorf("%q: got %v, want %q", tc.in, err, tc.want)
+		}
+	}
+}
+
+func TestVerify(t *testing.T) {
+	f := filepath.Join(t.TempDir(), "x.kpar")
+	os.WriteFile(f, []byte("hello"), 0o644)
+	want, _ := File(f)
+	sums, err := ParseChecksums("sha256:" + want + ",sha1:aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d,sha3-256:3338be694f50c5f338814986cdf0686453a888b84f424d792af4b9202398f392")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := Verify(f, sums); err != nil {
+		t.Fatal(err)
+	}
+	if err := Verify(f, nil); err != nil {
+		t.Fatal(err)
+	}
+	bad, _ := ParseChecksums("sha256:" + strings.Repeat("00", 32))
+	if err := Verify(f, bad); err == nil || !strings.Contains(err.Error(), "checksum mismatch: the resource declares sha256 "+strings.Repeat("00", 32)+", the archive has "+want) {
+		t.Fatalf("got %v", err)
+	}
+}