Repository navigation
purl-next: Create PURL accuracy benchmark - Curate scans for correct PURLs #49
Description
Create PURL accuracy benchmark to ensure tools create correct and valid PURLs. This step is to design the input/output and process and get some minimal first step in place.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
- Status
Activity
chinyeungli commentedon Jul 29, 2025
@pombredanne for this batch, should it be focus on certain package such as
rust,npm,mavenetc? or doesn't matter?pombredanne commentedon Jul 29, 2025
@chinyeungli your call.
chinyeungli commentedon Jul 30, 2025
Perhaps this is a "parent" ticket for:
chinyeungli commentedon Aug 1, 2025
I am using the list from #62 (comment) to perform an accuracy benchmark. @pombredanne Is the process involves downloading the packages, extracting them, running sctk’s package scan to retrieve the PURLs, and verifying those PURLs to ensure they’re valid (supposing they are starting with
pkg:pypi/) and have a valid version number?chinyeungli commentedon Aug 5, 2025
PURL_accuracy_benchmark-batch1.xlsx
I've created a first pass of the accuracy benchmark.
@pombredanne please check to see if this is what we are looking for.
[-]purl-next: Create PURL accuracy benchmark - batch 1[/-][+]purl-next: Create PURL accuracy benchmark - step 1[/+]chinyeungli commentedon Aug 8, 2025
@pombredanne I ran the
analyze_docker_imagepipeline on Ubuntu 24.04 and downloaded the results. I verified that all ELF binaries are covered, and there are no invalid package types, versions, or namespaces. Additionally, I confirmed that there are no duplicate package names (i.e. I checked for duplicate PURLs with the same name but different/no versions).Does this align with what we're aiming to validate in the accuracy benchmark (i.e., ensuring no PURLs are missing and no invalid or extra present)?
chinyeungli commentedon Aug 8, 2025
purl_accuracy_benchmark.xlsx
Please check to see if this is what we want, and how much more do we want to examine.
chinyeungli commentedon Aug 12, 2025
I uploaded the benchmark files on google drive under PURL accuracy benchmark: https://docs.google.com/spreadsheets/d/13QbqWtztAmIS0oxWANMSaiQSXCs-NxU9/edit?usp=drive_link&ouid=111292578521869099532&rtpof=true&sd=true
It covers 22 projects:
Would that be enough for the current iteration?
chinyeungli commentedon Aug 12, 2025
Per @pombredanne comment, this task is covered by the work at #49 (comment)
[-]purl-next: Create PURL accuracy benchmark - step 1[/-][+]purl-next: Create PURL accuracy benchmark - Curate scans for correct PURLs[/+]pombredanne commentedon Sep 2, 2025
We finally published the expected PURLs in this new repo https://github.com/aboutcode-org/purl-benchmarks