Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: build
on:
workflow_dispatch:
pull_request:
branches: [ 'main' ]
branches: [ 'main', 'crc64' ]
push:
branches:
- '*'
Expand All @@ -23,7 +23,9 @@ jobs:
uses: actions/checkout@v3
with:
repository: scanoss/ldb
ref: master
# This branch requires the CRC64 release line of LDB (5.x-crc64).
# See inc/ldb_compat.h; building against ldb 'main' (4.x) is rejected.
ref: crc64
path: ldb

- name: Build LDB
Expand Down
15 changes: 13 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,22 @@ jobs:
uses: actions/checkout@v3
with:
repository: scanoss/ldb
# This branch requires the CRC64 release line of LDB (5.x-crc64).
ref: crc64
path: ldb
fetch-depth: 0

- name: Build & Install ldb
run: cd ldb && git checkout $(git describe --tags $(git rev-list --tags --max-count=1)) && make all && sudo make install
# Pick the latest tag of the CRC64 line specifically. Both LDB release
# lines are tagged in the same repository, so "the newest tag" may well
# be a 4.x one from the traditional line.
run: |
cd ldb
ldb_tag=$(git tag --list 'v*-crc64' --sort=-v:refname | head -1)
if [ -z "$ldb_tag" ] ; then echo "No -crc64 tag found in scanoss/ldb" >&2 ; exit 1 ; fi
echo "Building LDB $ldb_tag"
git checkout "$ldb_tag"
make all && sudo make install

- name: Build engine
run: |
Expand Down
29 changes: 29 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,35 @@ When reviewing your pull request, we will follow a checklist similar to this one

We will also verify that the functionality implemented change serves the general public and not a particular interest group.

### Release lines and tagging

The SCANOSS engine is maintained as two parallel release lines, and a pull request must target the right one:

| Release line | Branch | Engine versions | Tag format | Requires LDB |
|---|---|---|---|---|
| Traditional | `main` | 5.x | `vMAJOR.MINOR.PATCH` (e.g. `v5.5.1`) | 4.x, from the `main` branch of scanoss/ldb |
| CRC64-compatible | `crc64` | 6.x and later | `vMAJOR.MINOR.PATCH-crc64` (e.g. `v6.0.0-crc64`) | 5.x-crc64, from the `crc64` branch of scanoss/ldb |

The `-crc64` suffix is mandatory on the CRC64 line: both lines are tagged in the same repository, and the suffix is what keeps
their tags, release artifacts and installed package versions distinguishable. It is also carried by `SCANOSS_VERSION`, so
`scanoss -v` identifies which line a binary comes from.

Because RPM does not allow `-` in the `Version` field, `package.sh` translates the suffix to `_` for the spec file
(`6.0.0-crc64` becomes `6.0.0_crc64`). Debian packages keep the tag spelling as is.

A fix that applies to both lines should be submitted against `main` and then ported to `crc64`; the two branches are not merged
into each other. This includes CI workflow changes: keeping `.github/workflows/` in sync between the branches avoids one line
silently rotting while the other gets fixed.

### LDB version requirement

The minimum LDB version is declared once, in [`inc/ldb_compat.h`](inc/ldb_compat.h). Both the build time check
(`scripts/check_ldb_version.sh`, run from the Makefile) and the run time check (`ldb_compat_check()`, called from
`initialize_ldb_tables()`) read it from there. Do not hardcode a version anywhere else: the two checks must never be able to
disagree.

Bumping the requirement is a one line change to that header.

### Licensing

The SCANOSS Platform is released under the GPL-2.0 license. If you wish to contribute, you must accept that you are aware of the license under which the project is released, and that your contribution will be released under the same license. Sometimes the GPL-2.0 license is incompatible with other licenses chosen by other projects. Therefore, you must accept that your contribution can also be released under the MIT license, which is the license we choose for those situations. Unless you expressly request otherwise, we may use your name, email address, username or URL for your attribution notice text. The submission of your contribution implies that you agree with these licensing terms.
25 changes: 14 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,11 @@ endif

LDFLAGS+= -lldb -lm -lpthread -ldl

LDB_CURRENT_VERSION := $(shell ldb -v | sed 's/ldb-//' | head -c 3)
LDB_TARGET_VERSION := 4.2

VERSION_IS_LESS := $(shell echo $(LDB_CURRENT_VERSION) \< $(LDB_TARGET_VERSION) | bc)
# Minimum LDB requirement. The single source of truth is inc/ldb_compat.h, which
# is also what the run time check in src/ldb_compat.c compiles against, so the
# build time and run time checks cannot drift apart.
LDB_COMPAT_HEADER := inc/ldb_compat.h
LDB_VERSION_CHECK := scripts/check_ldb_version.sh

CCFLAGS ?= -O -lz -Wall -Wno-unused-result -Wno-deprecated-declarations -g -Iinc -Iexternal/inc -D_LARGEFILE64_SOURCE -D_GNU_SOURCE
SOURCES=$(wildcard src/*.c) $(wildcard src/**/*.c) $(wildcard external/*.c) $(wildcard external/**/*.c)
Expand All @@ -16,19 +17,21 @@ TARGET=scanoss


# Regla de prueba
$(TARGET): $(OBJECTS)
ifeq ($(VERSION_IS_LESS),1)
@echo "Current LDB version: $(LDB_CURRENT_VERSION) is too old, please update to the lastest version to continue."
exit 1
endif
$(TARGET): $(OBJECTS) | check_ldb_version
$(CC) -g -o $(TARGET) $(OBJECTS) $(LDFLAGS)

$(CC) -g -o $(TARGET) $^ $(LDFLAGS)
# Verify the installed LDB before anything is compiled. Declared as an
# order-only prerequisite of every object so it also runs under `make -j`, and
# kept out of the plain `make clean` path.
.PHONY: check_ldb_version
check_ldb_version: $(LDB_VERSION_CHECK) $(LDB_COMPAT_HEADER)
@$(LDB_VERSION_CHECK) $(LDB_COMPAT_HEADER)

VERSION=$(shell ./version.sh)

.PHONY: scanoss

%.o: %.c
%.o: %.c | check_ldb_version
$(CC) $(CCFLAGS) -o $@ -c $<

all: clean scanoss
Expand Down
62 changes: 54 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,21 +8,60 @@ With its open architecture that is easy to integrate into existing processes and

By freeing developers to focus on writing great, compliant code that they and their team can completely trust, applications are finished earlier, quality is consistently higher, and development costs are dramatically lower.

---

## ⚠️ Release line: CRC64-compatible (6.x)

**This branch (`crc64`) holds the CRC64-compatible release line of the SCANOSS engine, starting at v6.0.0-crc64.**

The engine is maintained as two parallel release lines:

| Release line | Branch | Engine versions | Tag format | Requires LDB |
|---|---|---|---|---|
| Traditional | `main` | 5.x | `v5.5.1` | 4.x (`main` branch of scanoss/ldb) |
| CRC64-compatible | `crc64` | 6.x and later | `v6.0.0-crc64` | 5.x-crc64 (`crc64` branch of scanoss/ldb) |

Releases of this line carry a mandatory `-crc64` suffix, since both lines are tagged in the same repository. `scanoss -v`
reports it too, so a binary always identifies the line it came from.

### LDB compatibility

> **Engine 6.x requires LDB 5.0.0-crc64 or later, from the [`crc64` branch of scanoss/ldb](https://github.com/scanoss/ldb/tree/crc64).**
>
> Engine 5.x and earlier go with the traditional LDB line (4.x, `main` branch). The two combinations are not interchangeable in
> either direction.

This constraint applies in both hash modes: it comes from the LDB on-disk table layout and the library API surface, not from the
choice of CRC64 vs MD5. An MD5 knowledge base is still perfectly usable with engine 6.x, as long as it was built with LDB 5.x-crc64.

The requirement is enforced twice, because libldb is linked dynamically and the LDB present at build time is not necessarily the
one loaded at run time:

* at build time, by `scripts/check_ldb_version.sh`, which validates both the `ldb.h` the compiler resolves and the `libldb.so`
the linker resolves, and aborts `make` if either falls short;
* at run time, by `ldb_compat_check()`, which validates the loaded library before any table is opened.

Both derive the minimum from a single source of truth, [`inc/ldb_compat.h`](inc/ldb_compat.h).

---

# Setup
The Scanoss engine requires a Knowledge database installed for retrieving results. Scanoss use the SCANOSS LDB (Linked-list database) as a shared library. LDB Source code and installation guide can be found on https://github.com/scanoss/ldb
The knowledge database is incrementally built using the SCANOSS mining tool (minr). It source code and installation guide can be found on https://github.com/scanoss/minr

# Prerequisites
- LDB shared library. Installation instructions: [https://github.com/scanoss/ldb/README.md](https://github.com/scanoss/ldb/blob/master/README.md). Minimum version 4.1.0.
- LDB shared library, **5.0.0-crc64 or later**, built from the `crc64` branch. Installation instructions: [LDB README (crc64 branch)](https://github.com/scanoss/ldb/blob/crc64/README.md).
- libgcrypt-dev
# Installation

The SCANOSS Engine is a command-line tool used for comparing a file or directory against the SCANOSS Knowledgebase. The source code can be downloaded and compiled as follows:

```
wget -O engine.zip https://github.com/scanoss/engine/archive/master.zip
unzip engine.zip
cd engine-master
git clone -b crc64 https://github.com/scanoss/ldb
cd ldb && make all && sudo make install && cd ..

git clone -b crc64 https://github.com/scanoss/engine
cd engine
make
sudo make install
cd ..
Expand Down Expand Up @@ -53,24 +92,31 @@ Syntax: scanoss [parameters] [TARGET]
* `-T, --tolerance NUM` - Set snippet scanning tolerance percentage (default: 0.1)
* `-r, --rank NUM` - Set maximum component rank accepted (default: 11)
* `--max-files NUM` - Set maximum number of files to fetch during matching (default: 12000)
* `--min-match-hits NUM` - Set minimum snippet ID hits for a match (default: 3, disables auto-adjust)
* `--min-match-lines NUM` - Set minimum matched lines for a range (default: 10, disables auto-adjust)
* `--min-snippet-hits NUM` - Set minimum snippet ID hits for a match (default: 3, disables auto-adjust)
* `--min-snippet-lines NUM` - Set minimum matched lines for a range (default: 10, disables auto-adjust)
* `--range-tolerance NUM` - Set max non-matched lines tolerated in a range (default: 5)
* `--ignore-file-ext` - Ignore file extension during snippet matching (default: honor extension)

### SBOM and Filtering
* `-s, --sbom FILE` - Include assets from a JSON SBOM file (CycloneDX/SPDX2.2 format) in identification
* `-b, --blacklist FILE` - Exclude matches from assets listed in JSON SBOM file (CycloneDX/SPDX2.2 format)
* `--force-snippet` - Same as "-b" but with forced snippet scanning
* `--force-snippet` - Force snippet scanning (no full file matching). Takes no argument
* `-c, --component HINT` - Add a component HINT to guide scan results

### Attribution and Licenses
* `-a, --attribution FILE` - Show attribution notices for the provided SBOM.json file
* `-k, --key KEY` - Show contents of the specified KEY file from MZ sources archive
* `--max-file-content-size MB` - Set maximum file content size in MB printed by `-k` (default: 50)
* `-l, --license LICENSE` - Display OSADL metadata for the given SPDX license ID
* `-L, --full-license` - Enable full license report
* `-F, --flags FLAGS` - Set engine scanning flags (see Engine Flags section below)

### Knowledgebase Queries
* `-P, --purl MD5` - Return the purls related to the given file MD5, with their url hashes and source paths (JSON)
* `-C, --url-hash MD5` - Return the details of the component(s) identified by the given url hash, or a comma-separated list (JSON)
* `-p, --project URL_HASH` - Reconstruct a project's file structure: list the md5 and path of each project file (requires the pivot table; the url hash may be MD5 or CRC64)
* `-S, --snippet-scan WFP` - Snippet-only scan of a single-file WFP block, returning candidate file_md5s and their line ranges. Use `-S -` to read the WFP from stdin

### General Options
* `-t, --test` - Run engine performance tests
* `-v, --version` - Show version information and exit
Expand Down Expand Up @@ -116,7 +162,7 @@ scanoss --flags 12 DIRECTORY
scanoss --sbom my_sbom.json TARGET

# Scan with custom snippet matching parameters
scanoss --min-match-hits 5 --min-match-lines 15 TARGET
scanoss --min-snippet-hits 5 --min-snippet-lines 15 TARGET

# Scan with custom range tolerance
scanoss --range-tolerance 10 TARGET
Expand Down
38 changes: 34 additions & 4 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,37 @@ With its open architecture that is easy to integrate into existing processes and

By freeing developers to focus on writing great, compliant code that they and their team can completely trust, applications are finished earlier, quality is consistently higher, and development costs are dramatically lower.

Release line: CRC64-compatible (6.x)
------------------------------------

This documentation covers the ``crc64`` release line of the SCANOSS engine, starting at v6.0.0-crc64.

The engine is maintained as two parallel release lines:

.. list-table::
:header-rows: 1

* - Release line
- Branch
- Engine versions
- Tag format
- Requires LDB
* - Traditional
- ``main``
- 5.x
- ``v5.5.1``
- 4.x (``main`` branch of scanoss/ldb)
* - CRC64-compatible
- ``crc64``
- 6.x and later
- ``v6.0.0-crc64``
- 5.x-crc64 (``crc64`` branch of scanoss/ldb)

Engine 6.x requires LDB 5.0.0-crc64 or later. Engine 5.x and earlier go with the traditional LDB line (4.x). The constraint
comes from the LDB on-disk table layout and the library API surface, not from the choice of CRC64 vs MD5, so it applies in both
hash modes. The requirement is enforced at build time (``scripts/check_ldb_version.sh``) and at run time
(``ldb_compat_check()``), both driven from ``inc/ldb_compat.h``.

Setup
-----

Expand All @@ -18,17 +49,16 @@ The knowledge database is incrementally built using the SCANOSS mining tool (min
Prerequisites
-------------

* LDB shared library. Installation instructions: `LDB README <https://github.com/scanoss/ldb/blob/master/README.md>`_. Minimum version 4.1.0.
* LDB shared library, **5.0.0-crc64 or later**, built from the ``crc64`` branch. Installation instructions: `LDB README (crc64 branch) <https://github.com/scanoss/ldb/blob/crc64/README.md>`_.
* libgcrypt-dev

Installation
-------------

The SCANOSS Engine is a command-line tool used for comparing a file or directory against the SCANOSS Knowledgebase. The source code can be downloaded and compiled as follows::

wget -O engine.zip https://github.com/scanoss/engine/archive/master.zip
unzip engine.zip
cd engine-master
git clone -b crc64 https://github.com/scanoss/engine
cd engine
make
sudo make install
cd ..
Expand Down
68 changes: 68 additions & 0 deletions inc/ldb_compat.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#ifndef __LDB_COMPAT_H
#define __LDB_COMPAT_H
/* SPDX-License-Identifier: GPL-2.0-or-later
*
* inc/ldb_compat.h
*
* Minimum LDB requirement for this engine release line.
*
* Copyright (C) 2018-2025 SCANOSS.COM
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 2 of the License, or
* (at your option) any later version.
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
* You should have received a copy of the GNU General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
*/

/**
* @file ldb_compat.h
* @date 2025
* @brief Single source of truth for the LDB version this engine requires.
*
* The engine links libldb dynamically (-lldb), so the LDB it is compiled
* against and the LDB it ends up loading at run time are not necessarily the
* same build. Both are validated:
*
* - build time: scripts/check_ldb_version.sh, driven from this header by
* the Makefile, reads LDB_VERSION out of the ldb.h the compiler resolves.
* - run time: ldb_compat_check(), called from initialize_ldb_tables(),
* reads the version reported by the libldb.so actually loaded.
*
* Both checks derive their requirement from the macros below, so they cannot
* drift apart. Do not hardcode a minimum version anywhere else.
*
* The four macros below are parsed by scripts/check_ldb_version.sh with a
* plain `sed`, so keep the `#define NAME VALUE` spelling on a single line.
*/

#define LDB_REQUIRED_VERSION_MAJOR 5
#define LDB_REQUIRED_VERSION_MINOR 0
#define LDB_REQUIRED_VERSION_PATCH 0

/* Mandatory release-line suffix. The CRC64 line of LDB tags every release
* with a "-crc64" suffix and carries it in LDB_VERSION, which is what tells a
* CRC64-capable LDB apart from the MD5-only 4.x line. An MD5-only LDB may well
* satisfy the version floor above and still be the wrong release line. */
#define LDB_REQUIRED_RELEASE_LINE "crc64"

#define LDB_COMPAT_STR_(x) #x
#define LDB_COMPAT_STR(x) LDB_COMPAT_STR_(x)

/** Human readable spelling of the minimum requirement, e.g. "5.0.0-crc64" */
#define LDB_REQUIRED_VERSION \
LDB_COMPAT_STR(LDB_REQUIRED_VERSION_MAJOR) "." \
LDB_COMPAT_STR(LDB_REQUIRED_VERSION_MINOR) "." \
LDB_COMPAT_STR(LDB_REQUIRED_VERSION_PATCH) "-" LDB_REQUIRED_RELEASE_LINE

#include <stdbool.h>

bool ldb_compat_parse(const char *version, int *major, int *minor, int *patch, char *suffix, int suffix_ln);
void ldb_compat_check(void);

#endif
2 changes: 1 addition & 1 deletion inc/scanoss.h
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
#define WFP_LN 4
#define WFP_REC_LN 18

#define SCANOSS_VERSION "5.5.2-beta"
#define SCANOSS_VERSION "6.0.0-crc64"

/* Log files */
#define SCAN_LOG "/tmp/scanoss_scan.log"
Expand Down
5 changes: 4 additions & 1 deletion package.sh
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,10 @@ if [ "$1" = "rpm" ] ; then
cp scanoss dist/.rpmpkg/scanoss
chmod +x ./dist/.rpmpkg/scanoss
cp scripts/rpmpkg/scanoss.spec dist/.rpmpkg/scanoss.spec
sed -i 's/\ENGINE_VERSION/'"$2"'/g' dist/.rpmpkg/scanoss.spec
# RPM forbids '-' in the Version field, and the crc64 release line tags carry a
# "-crc64" suffix (e.g. v6.0.0-crc64). Translate it to '_' for the spec file.
rpm_version="${2//-/_}"
sed -i 's/\ENGINE_VERSION/'"$rpm_version"'/g' dist/.rpmpkg/scanoss.spec
rpmbuild -ba --build-in-place --define "_topdir $(pwd)/dist/rpm" dist/.rpmpkg/scanoss.spec
fi

Loading
Loading