Repository navigation
purl-next: Create PURL accuracy benchmarks #48
Description
We need to create PURL accuracy benchmark to ensure tools create accurate and valid PURLs.
Why? Most SCA tools provide poor results with invalid, or incorrect list of PURLs. We will handle invalid elsewhere, and the goal here is to create benchmarks using public projects, where we state what the correct output should be: that's not only about validating PURLs per se, that's validating scans as containing valid PURLs.
The plan is to perform the analysis in multiple steps or increments.
The output could be one of:
- a kissbom list of PURLs with license and copyright, OR
- a CycloneDX SBOM, OR
- an SPDX SBOM.
Ideally we should stay clear from a specific SBOM format to have something that is super simple to handle.
Some open source inputs to consider:
- A base RedHat container image (UBI)
- A base Debian or Ubuntu container image
- A large Go-based web application (with k8s)
- A large Python-based web application
- A large Java/JS-based application
With the input, we want to:
- Scan each
- validate that the PURLs are correct and are complete and exact (no more no less)
The final output should be as simple as a list of PURLs found in each (plus some metadata, URLs, etc.
The usage of these will be that any tool should be able to scan the inputs and validate that its output matches the list of PURLs.
Metadata
Metadata
Labels
Type
Projects
- Status
Activity
mjherzog commentedon Aug 6, 2025
@pombredanne We need clarity about which "tools" are to be evaluated for creating accurate PURLs. Are we evaluating FetchCode/PURLDB and the accuracy of the PURLs created by those tools?
Is the simple SBOM output from PURLDB or from somewhere else?
chinyeungli commentedon Aug 12, 2025
Per @pombredanne comment, the current work at #49 (comment) has covered #49 package-url#50 and package-url#51
DennisClark commentedon Aug 25, 2025
@pombredanne or @chinyeungli Do our plans include creating and providing an open-source public tool to perform the comparison between two lists of PURLs (comparing the AboutCode benchmark list .csv with the .csv list generated by an SCA tool)? Benefits would include (1) a method for standard scoring, and (2) instructional feedback to assist the analysts performing the comparisons. While working on the "Guide" it occurred to me that simply instructing analysts to "perform a diff" would not be as valuable as (1) specific detailed instructions regarding exactly what to look for, or (2) a tool that shows them how to do a useful comparison, or both.
For example, it would be valuable for the comparison tool to generate statistics not just for complete PURL values, but to identify patterns (both correct and incorrect) that exist for each of the various components (aka elements) of each PURL. So if I'm always getting my version wrong, or if my qualifiers are not all that good, I would see that in the results.
DennisClark commentedon Aug 25, 2025
@pombredanne please consider that @mjherzog has pointed out that a simple list of PURL values does not really need to be a .csv formatted file, it could be a simple text .txt file. What do you think? Certainly simplifies things.
5 remaining items
DennisClark commentedon Sep 1, 2025
@chinyeungli can you please investigate the best way to use SCIO to scan one example of each of these things
so that the scan options focus on package identification and PURL assignment of each package; we don't need to do licenses or copyrights for this. The results need to be such that they can be written to a simple text file (sorted by PURL), as described in
https://docs.google.com/document/d/15o5_6C1rks5gS2B21fyui_3f2688yPfZUP9pdZ91Moc/edit?usp=sharing
The part I'm struggling with is how to automate such an output (a simple text file designated by a path/filename) directly from the pipeline without needing to go through another export button in the SCIO UI, although that might be OK.
chinyeungli commentedon Sep 1, 2025
@DennisClark
In the doc at #49 (comment) , you'll find the "Input Download URL". Put that value into the "Download URLs" section on the New Project page in SCIO, and select the appropriate pipeline based on the URL format:
(if this is what you are asking)
by the way, I need to review the file, purl_accuracy_benchmark-2025-08-12.xlsx, again.
DennisClark commentedon Sep 1, 2025
Hi @chinyeungli the file at https://docs.google.com/spreadsheets/d/13QbqWtztAmIS0oxWANMSaiQSXCs-NxU9/edit?usp=drive_link&ouid=111292578521869099532&rtpof=true&sd=true is very informative and useful, but it's not exactly what we need for this benchmark project. Instead, we need one simple text file for each of the 22 projects that lists the PURLs, sorted by PURL value. For example, a text file for
https://github.com/elastic/elasticsearch/archive/refs/tags/v9.1.1.tar.gzwould contain:and nothing else, and the first thing of interest is why did we find 3 packages with duplicate PURLs? Different parts of the codebase I guess.
I am not sure what naming conventions we want to use for these benchmark text files; perhaps @pombredanne can make a suggestion. Should each simple text file be associated with a companion README file? or is the text file after all not quite so simple, and perhaps needs comment lines at the beginning to identify the Download URL and maybe other stuff?
chinyeungli commentedon Sep 1, 2025
okay, I will create text files for the projects that don't have "issues".
Format will be something like
will that work?
DennisClark commentedon Sep 1, 2025
@chinyeungli an excellent approach, thanks
chinyeungli commentedon Sep 2, 2025
@DennisClark I created some samples at https://drive.google.com/drive/folders/1akorAsBDWuC_zsaEDhEWHmSE8p139y5j?usp=drive_link , please check to see if this is what we want and let me know if the format need to be changed/updated.
pombredanne commentedon Sep 2, 2025
@chinyeungli I think for now we can go simpler with this approach:
For instance a table, with name, version, download_url and expected_purls_filename
With that data:
chinyeungli commentedon Sep 2, 2025
Out of the 22 projects created based on #49, only 7 are fully accurate—meaning they have no issues. The remaining projects either include layer commands from unrelated sources or contain ELF binaries that aren't marked as covered. I’ll try to add more projects to the list later.
pombredanne commentedon Sep 2, 2025
@chinyeungli 7 projects is awesome for a start and that's already a whole lot. We can expand make this at a later stage.
DennisClark commentedon Sep 2, 2025
@pombredanne @chinyeungli the enhancements to the Guide document look great, thanks!
pombredanne commentedon Sep 2, 2025
This is now all done. See: