Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/workflows/scan.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,6 @@ jobs:
permissions:
# the index is committed by the custodian app, this job's token only reads
contents: read
# discover the org's packages with the GitHub Packages API
packages: read
steps:
- uses: actions/checkout@v7
- uses: actions-rust-lang/setup-rust-toolchain@v2
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ A directory of the Wasm components and WIT interfaces published by the Component

## How it works

1. `directory scan` finds the org's packages. It reads the GitHub Packages API when the token may (`packages: read`). It also reads the org's public repositories and their `components/*` directories.
1. `directory scan` finds the org's packages from its public repositories: a package for each repository, and one for each of its `components/*` directories. A package published from anywhere else can be added with `--package`.
2. It lists each package's tags on ghcr.io and indexes every stable release (`x.y.z`) and dev build (`x.y.z-dev`) that isn't in the index yet. It pulls the wasm layer, verifies its digest, and decodes it to WIT. A `_debug` tag is recorded as the debug build of its version. Other tags are skipped: other pre-releases, build metadata, `v`-prefixed tags, `dev`, and cosign signatures.
- A release is immutable, so it is indexed once.
- A dev tag moves with each build. It is re-indexed whenever its digest changes, until its release is published.
Expand Down Expand Up @@ -56,7 +56,7 @@ Once `x.y.z` is released, the site hides `x.y.z-dev` from version lists, search,
## Commands

```sh
# index new releases; a token lets discovery use the Packages API and raises rate limits
# index new releases; a token raises the GitHub API's rate limits
GITHUB_TOKEN=$(gh auth token) cargo run -- scan

# only the packages of a repository, e.g. while testing
Expand Down
2 changes: 1 addition & 1 deletion src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ enum Command {
/// The GitHub org whose ghcr.io packages are scanned, also their WIT namespace
#[arg(long, default_value = "componentized")]
org: String,
/// A GitHub token, `packages: read` lets discovery use the GitHub Packages API
/// A GitHub token, which raises the GitHub API's rate limits for discovery and releases
#[arg(long, env = "GITHUB_TOKEN", hide_env_values = true)]
github_token: Option<String>,
/// A package to scan in addition to the discovered ones, e.g. `logging/levels`
Expand Down
37 changes: 3 additions & 34 deletions src/scan.rs
Original file line number Diff line number Diff line change
Expand Up @@ -256,16 +256,10 @@ fn trim_nested(root: &std::path::Path) -> Result<()> {
Ok(())
}

/// Lists the container packages of the org: from the GitHub Packages API when the token may read
/// them, and from the `components/*` directories of the org's repositories, which is all an
/// anonymous scan can see.
/// Lists the container packages of the org: a package for each of its public repositories, and one
/// for each of their `components/*` directories. The GitHub Packages API would list them directly,
/// but it only accepts a personal access token (classic), not a workflow's or an app's token.
fn discover(github: &GitHub, org: &str) -> Result<BTreeSet<String>> {
#[derive(Deserialize)]
struct Package {
name: String,
#[serde(default)]
visibility: Option<String>,
}
#[derive(Deserialize)]
struct Repo {
name: String,
Expand All @@ -284,31 +278,6 @@ fn discover(github: &GitHub, org: &str) -> Result<BTreeSet<String>> {
}

let mut found = BTreeSet::new();
// the Packages API is optional: a token that can't read it, or an error reading it, falls back
// to discovering the packages from the repositories
match github.list::<Package>(&format!("/orgs/{org}/packages?package_type=container")) {
Ok(Some(packages)) => {
eprintln!(
"found {} packages with the GitHub Packages API",
packages.len()
);
found.extend(
packages
.into_iter()
.filter(|p| p.visibility.as_deref() != Some("private"))
.map(|p| p.name),
);
}
Ok(None) => eprintln!(
"the GitHub Packages API is not readable, `packages: read`, discovering packages from the repositories"
),
Err(err) => {
println!(
"::warning::the GitHub Packages API failed, discovering packages from the repositories: {err:#}"
);
eprintln!("the GitHub Packages API failed: {err:#}");
}
}
let repos = github
.list::<Repo>(&format!("/orgs/{org}/repos?type=public"))?
.ok_or_else(|| anyhow!("listing the repositories of {org}"))?;
Expand Down
Loading