From edd2a0539993382a299d5adde352e69c8cd2d447 Mon Sep 17 00:00:00 2001 From: mscasso-scanoss Date: Thu, 17 Sep 2026 15:18:24 +0000 Subject: [PATCH] Release 6.0.0-crc64: enforce the LDB crc64 release line Open the CRC64 release line of the engine (6.x, branch crc64), mirroring the scanoss/ldb 5.0.0-crc64 release. Engine 6.x requires LDB 5.0.0-crc64 or later; engine 5.x (branch main) stays on LDB 4.x. The two lines are tagged in the same repository, so releases of this line carry a mandatory "-crc64" suffix, also carried by SCANOSS_VERSION so that `scanoss -v` identifies the line. LDB version validation ---------------------- The engine links libldb dynamically, so the LDB present at build time is not necessarily the one loaded at run time. Both are now validated, and neither check can drift from the other: the requirement is declared once, in inc/ldb_compat.h, and read by both. Build time (scripts/check_ldb_version.sh, run from the Makefile before any object is compiled). The previous check was `ldb -v | head -c 3` compared with `bc`, which: - truncated to 3 characters, so "5.0.0-crc64" became "5.0" and a two digit major ("10.0.0") became "10." and made bc fail with a syntax error; - compared as a decimal rather than a semantic version, ranking 4.10 below 4.2; - never inspected the suffix, so an MD5-only LDB passed; - queried the binary on PATH, which need not be the build the linker resolves; - depended on bc. It now compares major/minor/patch numerically with shell arithmetic, requires the crc64 suffix, and reports "too old" and "wrong release line" as distinct, actionable failures. It validates two sources and fails on either: the ldb.h the compiler resolves (located and expanded by the preprocessor, so it honours CPPFLAGS rather than assuming /usr/include) and the libldb.so the linker resolves (probed by building and running a program against -lldb). The header alone is not sufficient: a stale library earlier in the loader search path produces a binary that compiles cleanly and then fails at run time on the very machine that built it. `ldb -v` is kept as a non fatal hint only. Run time (src/ldb_compat.c, called from initialize_ldb_tables()). The previous check used strcmp against "4.1.0", which is lexicographic: "5.10.0" sorts below "5.9.0", so the engine would refuse to start against a newer LDB. That "5.0.0-crc64" passed at all was an accident of string length. The version is now parsed into major/minor/patch plus suffix and held to the same rule as the build. Documentation and consistency ----------------------------- - SCANOSS_VERSION 5.5.2-beta -> 6.0.0-crc64. - README.md and docs/source/index.rst: require LDB >= 5.0.0-crc64 and link the crc64 branch of the LDB README instead of master; document the release lines and the compatibility rule, noting it comes from the on-disk table layout and the library API, not from the hash mode, so it applies in MD5 too. - CONTRIBUTING.md: document the branch and tag scheme, and that the LDB minimum lives in a single header. - package.sh: RPM forbids '-' in the Version field, so the suffix is translated to '_' for the spec (6.0.0-crc64 -> 6.0.0_crc64). The deb keeps the hyphen. Fixes found while verifying the documentation against the code -------------------------------------------------------------- - --force-snippet was declared required_argument in long_options, but case 256 never reads optarg and the help text shows no argument, so `scanoss --force-snippet TARGET` silently consumed TARGET as the option's argument. Corrected to no_argument. - README documented --min-match-hits and --min-match-lines, which do not exist and are rejected by the parser. The real names are --min-snippet-hits and --min-snippet-lines. - README also described --force-snippet as "same as -b", omitted --max-file-content-size, -P/--purl, -C/--url-hash, -p/--project and -S/--snippet-scan, and pointed the install snippet at an archive of a nonexistent master branch. CI -- Both workflows cloned LDB from master (the 4.x MD5-only line), which the new check now rejects; they build from the crc64 branch instead. release.yml selected "the newest tag", which may belong to either line, and now picks the latest v*-crc64 tag specifically. build.yml also triggers on pull requests targeting crc64. The actions/checkout@v3 and ::set-output deprecations are left untouched on purpose: they are warnings, and fixing them on this branch alone would introduce drift from main. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/build.yml | 6 +- .github/workflows/release.yml | 15 ++- CONTRIBUTING.md | 29 ++++ Makefile | 25 ++-- README.md | 62 +++++++-- docs/source/index.rst | 38 +++++- inc/ldb_compat.h | 68 ++++++++++ inc/scanoss.h | 2 +- package.sh | 5 +- scripts/check_ldb_version.sh | 244 ++++++++++++++++++++++++++++++++++ src/ldb_compat.c | 159 ++++++++++++++++++++++ src/main.c | 20 +-- 12 files changed, 631 insertions(+), 42 deletions(-) create mode 100644 inc/ldb_compat.h create mode 100755 scripts/check_ldb_version.sh create mode 100644 src/ldb_compat.c diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 6496d071..f212fc29 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -3,7 +3,7 @@ name: build on: workflow_dispatch: pull_request: - branches: [ 'main' ] + branches: [ 'main', 'crc64' ] push: branches: - '*' @@ -23,7 +23,9 @@ jobs: uses: actions/checkout@v3 with: repository: scanoss/ldb - ref: master + # This branch requires the CRC64 release line of LDB (5.x-crc64). + # See inc/ldb_compat.h; building against ldb 'main' (4.x) is rejected. + ref: crc64 path: ldb - name: Build LDB diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 804dcfc7..c9aab363 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -25,11 +25,22 @@ jobs: uses: actions/checkout@v3 with: repository: scanoss/ldb + # This branch requires the CRC64 release line of LDB (5.x-crc64). + ref: crc64 path: ldb fetch-depth: 0 - + - name: Build & Install ldb - run: cd ldb && git checkout $(git describe --tags $(git rev-list --tags --max-count=1)) && make all && sudo make install + # Pick the latest tag of the CRC64 line specifically. Both LDB release + # lines are tagged in the same repository, so "the newest tag" may well + # be a 4.x one from the traditional line. + run: | + cd ldb + ldb_tag=$(git tag --list 'v*-crc64' --sort=-v:refname | head -1) + if [ -z "$ldb_tag" ] ; then echo "No -crc64 tag found in scanoss/ldb" >&2 ; exit 1 ; fi + echo "Building LDB $ldb_tag" + git checkout "$ldb_tag" + make all && sudo make install - name: Build engine run: | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 13e99fa2..0790d109 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,6 +25,35 @@ When reviewing your pull request, we will follow a checklist similar to this one We will also verify that the functionality implemented change serves the general public and not a particular interest group. +### Release lines and tagging + +The SCANOSS engine is maintained as two parallel release lines, and a pull request must target the right one: + +| Release line | Branch | Engine versions | Tag format | Requires LDB | +|---|---|---|---|---| +| Traditional | `main` | 5.x | `vMAJOR.MINOR.PATCH` (e.g. `v5.5.1`) | 4.x, from the `main` branch of scanoss/ldb | +| CRC64-compatible | `crc64` | 6.x and later | `vMAJOR.MINOR.PATCH-crc64` (e.g. `v6.0.0-crc64`) | 5.x-crc64, from the `crc64` branch of scanoss/ldb | + +The `-crc64` suffix is mandatory on the CRC64 line: both lines are tagged in the same repository, and the suffix is what keeps +their tags, release artifacts and installed package versions distinguishable. It is also carried by `SCANOSS_VERSION`, so +`scanoss -v` identifies which line a binary comes from. + +Because RPM does not allow `-` in the `Version` field, `package.sh` translates the suffix to `_` for the spec file +(`6.0.0-crc64` becomes `6.0.0_crc64`). Debian packages keep the tag spelling as is. + +A fix that applies to both lines should be submitted against `main` and then ported to `crc64`; the two branches are not merged +into each other. This includes CI workflow changes: keeping `.github/workflows/` in sync between the branches avoids one line +silently rotting while the other gets fixed. + +### LDB version requirement + +The minimum LDB version is declared once, in [`inc/ldb_compat.h`](inc/ldb_compat.h). Both the build time check +(`scripts/check_ldb_version.sh`, run from the Makefile) and the run time check (`ldb_compat_check()`, called from +`initialize_ldb_tables()`) read it from there. Do not hardcode a version anywhere else: the two checks must never be able to +disagree. + +Bumping the requirement is a one line change to that header. + ### Licensing The SCANOSS Platform is released under the GPL-2.0 license. If you wish to contribute, you must accept that you are aware of the license under which the project is released, and that your contribution will be released under the same license. Sometimes the GPL-2.0 license is incompatible with other licenses chosen by other projects. Therefore, you must accept that your contribution can also be released under the MIT license, which is the license we choose for those situations. Unless you expressly request otherwise, we may use your name, email address, username or URL for your attribution notice text. The submission of your contribution implies that you agree with these licensing terms. diff --git a/Makefile b/Makefile index d92d349b..1c862840 100644 --- a/Makefile +++ b/Makefile @@ -4,10 +4,11 @@ endif LDFLAGS+= -lldb -lm -lpthread -ldl -LDB_CURRENT_VERSION := $(shell ldb -v | sed 's/ldb-//' | head -c 3) -LDB_TARGET_VERSION := 4.2 - -VERSION_IS_LESS := $(shell echo $(LDB_CURRENT_VERSION) \< $(LDB_TARGET_VERSION) | bc) +# Minimum LDB requirement. The single source of truth is inc/ldb_compat.h, which +# is also what the run time check in src/ldb_compat.c compiles against, so the +# build time and run time checks cannot drift apart. +LDB_COMPAT_HEADER := inc/ldb_compat.h +LDB_VERSION_CHECK := scripts/check_ldb_version.sh CCFLAGS ?= -O -lz -Wall -Wno-unused-result -Wno-deprecated-declarations -g -Iinc -Iexternal/inc -D_LARGEFILE64_SOURCE -D_GNU_SOURCE SOURCES=$(wildcard src/*.c) $(wildcard src/**/*.c) $(wildcard external/*.c) $(wildcard external/**/*.c) @@ -16,19 +17,21 @@ TARGET=scanoss # Regla de prueba -$(TARGET): $(OBJECTS) -ifeq ($(VERSION_IS_LESS),1) - @echo "Current LDB version: $(LDB_CURRENT_VERSION) is too old, please update to the lastest version to continue." - exit 1 -endif +$(TARGET): $(OBJECTS) | check_ldb_version + $(CC) -g -o $(TARGET) $(OBJECTS) $(LDFLAGS) - $(CC) -g -o $(TARGET) $^ $(LDFLAGS) +# Verify the installed LDB before anything is compiled. Declared as an +# order-only prerequisite of every object so it also runs under `make -j`, and +# kept out of the plain `make clean` path. +.PHONY: check_ldb_version +check_ldb_version: $(LDB_VERSION_CHECK) $(LDB_COMPAT_HEADER) + @$(LDB_VERSION_CHECK) $(LDB_COMPAT_HEADER) VERSION=$(shell ./version.sh) .PHONY: scanoss -%.o: %.c +%.o: %.c | check_ldb_version $(CC) $(CCFLAGS) -o $@ -c $< all: clean scanoss diff --git a/README.md b/README.md index 33f371a2..8ac65d23 100644 --- a/README.md +++ b/README.md @@ -8,21 +8,60 @@ With its open architecture that is easy to integrate into existing processes and By freeing developers to focus on writing great, compliant code that they and their team can completely trust, applications are finished earlier, quality is consistently higher, and development costs are dramatically lower. +--- + +## ⚠️ Release line: CRC64-compatible (6.x) + +**This branch (`crc64`) holds the CRC64-compatible release line of the SCANOSS engine, starting at v6.0.0-crc64.** + +The engine is maintained as two parallel release lines: + +| Release line | Branch | Engine versions | Tag format | Requires LDB | +|---|---|---|---|---| +| Traditional | `main` | 5.x | `v5.5.1` | 4.x (`main` branch of scanoss/ldb) | +| CRC64-compatible | `crc64` | 6.x and later | `v6.0.0-crc64` | 5.x-crc64 (`crc64` branch of scanoss/ldb) | + +Releases of this line carry a mandatory `-crc64` suffix, since both lines are tagged in the same repository. `scanoss -v` +reports it too, so a binary always identifies the line it came from. + +### LDB compatibility + +> **Engine 6.x requires LDB 5.0.0-crc64 or later, from the [`crc64` branch of scanoss/ldb](https://github.com/scanoss/ldb/tree/crc64).** +> +> Engine 5.x and earlier go with the traditional LDB line (4.x, `main` branch). The two combinations are not interchangeable in +> either direction. + +This constraint applies in both hash modes: it comes from the LDB on-disk table layout and the library API surface, not from the +choice of CRC64 vs MD5. An MD5 knowledge base is still perfectly usable with engine 6.x, as long as it was built with LDB 5.x-crc64. + +The requirement is enforced twice, because libldb is linked dynamically and the LDB present at build time is not necessarily the +one loaded at run time: + +* at build time, by `scripts/check_ldb_version.sh`, which validates both the `ldb.h` the compiler resolves and the `libldb.so` + the linker resolves, and aborts `make` if either falls short; +* at run time, by `ldb_compat_check()`, which validates the loaded library before any table is opened. + +Both derive the minimum from a single source of truth, [`inc/ldb_compat.h`](inc/ldb_compat.h). + +--- + # Setup The Scanoss engine requires a Knowledge database installed for retrieving results. Scanoss use the SCANOSS LDB (Linked-list database) as a shared library. LDB Source code and installation guide can be found on https://github.com/scanoss/ldb The knowledge database is incrementally built using the SCANOSS mining tool (minr). It source code and installation guide can be found on https://github.com/scanoss/minr # Prerequisites -- LDB shared library. Installation instructions: [https://github.com/scanoss/ldb/README.md](https://github.com/scanoss/ldb/blob/master/README.md). Minimum version 4.1.0. +- LDB shared library, **5.0.0-crc64 or later**, built from the `crc64` branch. Installation instructions: [LDB README (crc64 branch)](https://github.com/scanoss/ldb/blob/crc64/README.md). - libgcrypt-dev # Installation The SCANOSS Engine is a command-line tool used for comparing a file or directory against the SCANOSS Knowledgebase. The source code can be downloaded and compiled as follows: ``` -wget -O engine.zip https://github.com/scanoss/engine/archive/master.zip -unzip engine.zip -cd engine-master +git clone -b crc64 https://github.com/scanoss/ldb +cd ldb && make all && sudo make install && cd .. + +git clone -b crc64 https://github.com/scanoss/engine +cd engine make sudo make install cd .. @@ -53,24 +92,31 @@ Syntax: scanoss [parameters] [TARGET] * `-T, --tolerance NUM` - Set snippet scanning tolerance percentage (default: 0.1) * `-r, --rank NUM` - Set maximum component rank accepted (default: 11) * `--max-files NUM` - Set maximum number of files to fetch during matching (default: 12000) -* `--min-match-hits NUM` - Set minimum snippet ID hits for a match (default: 3, disables auto-adjust) -* `--min-match-lines NUM` - Set minimum matched lines for a range (default: 10, disables auto-adjust) +* `--min-snippet-hits NUM` - Set minimum snippet ID hits for a match (default: 3, disables auto-adjust) +* `--min-snippet-lines NUM` - Set minimum matched lines for a range (default: 10, disables auto-adjust) * `--range-tolerance NUM` - Set max non-matched lines tolerated in a range (default: 5) * `--ignore-file-ext` - Ignore file extension during snippet matching (default: honor extension) ### SBOM and Filtering * `-s, --sbom FILE` - Include assets from a JSON SBOM file (CycloneDX/SPDX2.2 format) in identification * `-b, --blacklist FILE` - Exclude matches from assets listed in JSON SBOM file (CycloneDX/SPDX2.2 format) -* `--force-snippet` - Same as "-b" but with forced snippet scanning +* `--force-snippet` - Force snippet scanning (no full file matching). Takes no argument * `-c, --component HINT` - Add a component HINT to guide scan results ### Attribution and Licenses * `-a, --attribution FILE` - Show attribution notices for the provided SBOM.json file * `-k, --key KEY` - Show contents of the specified KEY file from MZ sources archive +* `--max-file-content-size MB` - Set maximum file content size in MB printed by `-k` (default: 50) * `-l, --license LICENSE` - Display OSADL metadata for the given SPDX license ID * `-L, --full-license` - Enable full license report * `-F, --flags FLAGS` - Set engine scanning flags (see Engine Flags section below) +### Knowledgebase Queries +* `-P, --purl MD5` - Return the purls related to the given file MD5, with their url hashes and source paths (JSON) +* `-C, --url-hash MD5` - Return the details of the component(s) identified by the given url hash, or a comma-separated list (JSON) +* `-p, --project URL_HASH` - Reconstruct a project's file structure: list the md5 and path of each project file (requires the pivot table; the url hash may be MD5 or CRC64) +* `-S, --snippet-scan WFP` - Snippet-only scan of a single-file WFP block, returning candidate file_md5s and their line ranges. Use `-S -` to read the WFP from stdin + ### General Options * `-t, --test` - Run engine performance tests * `-v, --version` - Show version information and exit @@ -116,7 +162,7 @@ scanoss --flags 12 DIRECTORY scanoss --sbom my_sbom.json TARGET # Scan with custom snippet matching parameters -scanoss --min-match-hits 5 --min-match-lines 15 TARGET +scanoss --min-snippet-hits 5 --min-snippet-lines 15 TARGET # Scan with custom range tolerance scanoss --range-tolerance 10 TARGET diff --git a/docs/source/index.rst b/docs/source/index.rst index 4e9e586f..89354d5e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -8,6 +8,37 @@ With its open architecture that is easy to integrate into existing processes and By freeing developers to focus on writing great, compliant code that they and their team can completely trust, applications are finished earlier, quality is consistently higher, and development costs are dramatically lower. +Release line: CRC64-compatible (6.x) +------------------------------------ + +This documentation covers the ``crc64`` release line of the SCANOSS engine, starting at v6.0.0-crc64. + +The engine is maintained as two parallel release lines: + +.. list-table:: + :header-rows: 1 + + * - Release line + - Branch + - Engine versions + - Tag format + - Requires LDB + * - Traditional + - ``main`` + - 5.x + - ``v5.5.1`` + - 4.x (``main`` branch of scanoss/ldb) + * - CRC64-compatible + - ``crc64`` + - 6.x and later + - ``v6.0.0-crc64`` + - 5.x-crc64 (``crc64`` branch of scanoss/ldb) + +Engine 6.x requires LDB 5.0.0-crc64 or later. Engine 5.x and earlier go with the traditional LDB line (4.x). The constraint +comes from the LDB on-disk table layout and the library API surface, not from the choice of CRC64 vs MD5, so it applies in both +hash modes. The requirement is enforced at build time (``scripts/check_ldb_version.sh``) and at run time +(``ldb_compat_check()``), both driven from ``inc/ldb_compat.h``. + Setup ----- @@ -18,7 +49,7 @@ The knowledge database is incrementally built using the SCANOSS mining tool (min Prerequisites ------------- -* LDB shared library. Installation instructions: `LDB README `_. Minimum version 4.1.0. +* LDB shared library, **5.0.0-crc64 or later**, built from the ``crc64`` branch. Installation instructions: `LDB README (crc64 branch) `_. * libgcrypt-dev Installation @@ -26,9 +57,8 @@ Installation The SCANOSS Engine is a command-line tool used for comparing a file or directory against the SCANOSS Knowledgebase. The source code can be downloaded and compiled as follows:: - wget -O engine.zip https://github.com/scanoss/engine/archive/master.zip - unzip engine.zip - cd engine-master + git clone -b crc64 https://github.com/scanoss/engine + cd engine make sudo make install cd .. diff --git a/inc/ldb_compat.h b/inc/ldb_compat.h new file mode 100644 index 00000000..4795e345 --- /dev/null +++ b/inc/ldb_compat.h @@ -0,0 +1,68 @@ +#ifndef __LDB_COMPAT_H +#define __LDB_COMPAT_H +/* SPDX-License-Identifier: GPL-2.0-or-later + * + * inc/ldb_compat.h + * + * Minimum LDB requirement for this engine release line. + * + * Copyright (C) 2018-2025 SCANOSS.COM + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +/** + * @file ldb_compat.h + * @date 2025 + * @brief Single source of truth for the LDB version this engine requires. + * + * The engine links libldb dynamically (-lldb), so the LDB it is compiled + * against and the LDB it ends up loading at run time are not necessarily the + * same build. Both are validated: + * + * - build time: scripts/check_ldb_version.sh, driven from this header by + * the Makefile, reads LDB_VERSION out of the ldb.h the compiler resolves. + * - run time: ldb_compat_check(), called from initialize_ldb_tables(), + * reads the version reported by the libldb.so actually loaded. + * + * Both checks derive their requirement from the macros below, so they cannot + * drift apart. Do not hardcode a minimum version anywhere else. + * + * The four macros below are parsed by scripts/check_ldb_version.sh with a + * plain `sed`, so keep the `#define NAME VALUE` spelling on a single line. + */ + +#define LDB_REQUIRED_VERSION_MAJOR 5 +#define LDB_REQUIRED_VERSION_MINOR 0 +#define LDB_REQUIRED_VERSION_PATCH 0 + +/* Mandatory release-line suffix. The CRC64 line of LDB tags every release + * with a "-crc64" suffix and carries it in LDB_VERSION, which is what tells a + * CRC64-capable LDB apart from the MD5-only 4.x line. An MD5-only LDB may well + * satisfy the version floor above and still be the wrong release line. */ +#define LDB_REQUIRED_RELEASE_LINE "crc64" + +#define LDB_COMPAT_STR_(x) #x +#define LDB_COMPAT_STR(x) LDB_COMPAT_STR_(x) + +/** Human readable spelling of the minimum requirement, e.g. "5.0.0-crc64" */ +#define LDB_REQUIRED_VERSION \ + LDB_COMPAT_STR(LDB_REQUIRED_VERSION_MAJOR) "." \ + LDB_COMPAT_STR(LDB_REQUIRED_VERSION_MINOR) "." \ + LDB_COMPAT_STR(LDB_REQUIRED_VERSION_PATCH) "-" LDB_REQUIRED_RELEASE_LINE + +#include + +bool ldb_compat_parse(const char *version, int *major, int *minor, int *patch, char *suffix, int suffix_ln); +void ldb_compat_check(void); + +#endif diff --git a/inc/scanoss.h b/inc/scanoss.h index 742dc8ec..ada3d28b 100644 --- a/inc/scanoss.h +++ b/inc/scanoss.h @@ -33,7 +33,7 @@ #define WFP_LN 4 #define WFP_REC_LN 18 -#define SCANOSS_VERSION "5.5.2-beta" +#define SCANOSS_VERSION "6.0.0-crc64" /* Log files */ #define SCAN_LOG "/tmp/scanoss_scan.log" diff --git a/package.sh b/package.sh index 41e0ba37..187b5ba3 100755 --- a/package.sh +++ b/package.sh @@ -53,7 +53,10 @@ if [ "$1" = "rpm" ] ; then cp scanoss dist/.rpmpkg/scanoss chmod +x ./dist/.rpmpkg/scanoss cp scripts/rpmpkg/scanoss.spec dist/.rpmpkg/scanoss.spec - sed -i 's/\ENGINE_VERSION/'"$2"'/g' dist/.rpmpkg/scanoss.spec + # RPM forbids '-' in the Version field, and the crc64 release line tags carry a + # "-crc64" suffix (e.g. v6.0.0-crc64). Translate it to '_' for the spec file. + rpm_version="${2//-/_}" + sed -i 's/\ENGINE_VERSION/'"$rpm_version"'/g' dist/.rpmpkg/scanoss.spec rpmbuild -ba --build-in-place --define "_topdir $(pwd)/dist/rpm" dist/.rpmpkg/scanoss.spec fi diff --git a/scripts/check_ldb_version.sh b/scripts/check_ldb_version.sh new file mode 100755 index 00000000..4583b2b3 --- /dev/null +++ b/scripts/check_ldb_version.sh @@ -0,0 +1,244 @@ +#!/bin/bash +### +# SPDX-License-Identifier: GPL-2.0-or-later +# +# Copyright (C) 2018-2025 SCANOSS.COM +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 2 of the License, or +# (at your option) any later version. +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . +### +# +# Build time validation of the installed LDB. +# +# The requirement is read from inc/ldb_compat.h, the same header the run time +# check (src/ldb_compat.c) compiles against, so the two can never disagree. +# +# Two independent sources are validated, because the engine links libldb +# dynamically and they can disagree on the same machine: +# +# 1. The ldb.h the compiler resolves for `#include `, expanded by the +# preprocessor itself. This is literally what the engine compiles against, +# which makes it a far more faithful source than `ldb -v`: the shell binary +# found on PATH is a separate artifact that may come from another install, +# from another release line, or not be installed at all, while the build +# would still happily use /usr/include/ldb.h and link -lldb. +# +# 2. The libldb.so the linker actually resolves, probed by building and +# running a two line program against it. A stale library earlier in the +# loader search path (a leftover /usr/local/lib/libldb.so shadowing +# /usr/lib/libldb.so, say) produces a binary that compiles cleanly and then +# fails the run time check on the very machine that built it. +# +# `ldb -v` is consulted too, but only as a non fatal hint: it is neither +# compiled against nor linked in. +# +# Usage: check_ldb_version.sh [path/to/inc/ldb_compat.h] +# + +set -u + +COMPAT_HEADER="${1:-inc/ldb_compat.h}" +CC_BIN="${CC:-gcc}" +CPPFLAGS="${CPPFLAGS:-}" + +if [ ! -f "$COMPAT_HEADER" ] ; then + echo "ERROR: cannot read $COMPAT_HEADER, unable to determine the required LDB version." >&2 + exit 1 +fi + +# Read a `#define NAME VALUE` (quoted or not) out of the compat header +compat_define() { + sed -n "s/^[[:space:]]*#define[[:space:]]\+$1[[:space:]]\+\"\{0,1\}\([^\"[:space:]]*\)\"\{0,1\}.*/\1/p" \ + "$COMPAT_HEADER" | head -1 +} + +REQ_MAJOR=$(compat_define LDB_REQUIRED_VERSION_MAJOR) +REQ_MINOR=$(compat_define LDB_REQUIRED_VERSION_MINOR) +REQ_PATCH=$(compat_define LDB_REQUIRED_VERSION_PATCH) +REQ_LINE=$(compat_define LDB_REQUIRED_RELEASE_LINE) + +if [ -z "$REQ_MAJOR" ] || [ -z "$REQ_MINOR" ] || [ -z "$REQ_PATCH" ] || [ -z "$REQ_LINE" ] ; then + echo "ERROR: could not parse the LDB requirement out of $COMPAT_HEADER." >&2 + exit 1 +fi + +REQUIRED="${REQ_MAJOR}.${REQ_MINOR}.${REQ_PATCH}-${REQ_LINE}" +REQUIRED_WEIGHT=$(( REQ_MAJOR * 1000000 + REQ_MINOR * 1000 + REQ_PATCH )) + +INSTALL_HINT=" git clone -b ${REQ_LINE} https://github.com/scanoss/ldb && cd ldb && make all && sudo make install" + +# Split MAJOR[.MINOR[.PATCH]][-SUFFIX] into its components +parse_version() { + local v="${1#ldb-}" + local base="${v%%-*}" + local suffix="" + case "$v" in *-*) suffix="${v#*-}" ;; esac + local major="${base%%.*}" + local rest="${base#*.}" + local minor=0 patch=0 + if [ "$rest" != "$base" ] ; then + minor="${rest%%.*}" + local rest2="${rest#*.}" + [ "$rest2" != "$rest" ] && patch="${rest2%%.*}" + fi + case "$major$minor$patch" in + ''|*[!0-9]*) return 1 ;; + esac + printf '%s %s %s %s\n' "$major" "$minor" "$patch" "$suffix" + return 0 +} + +# validate_version +# Exits with a message the user can act on if does not meet the +# requirement declared in inc/ldb_compat.h. +validate_version() { + local found="$1" origin="$2" parsed + local major minor patch line weight + + if ! parsed=$(parse_version "$found") ; then + cat >&2 <&2 <&2 <\nLDB_COMPAT_PROBE LDB_VERSION\n' > "$probe_src" + +preprocessed=$($CC_BIN $CPPFLAGS -E "$probe_src" 2>/dev/null) +if [ -z "$preprocessed" ] ; then + cat >&2 < not found. The SCANOSS engine compiles and links against LDB. + Install LDB ${REQUIRED} or later from the '${REQ_LINE}' branch of + https://github.com/scanoss/ldb: +${INSTALL_HINT} + +EOM + exit 1 +fi + +LDB_HEADER=$(printf '%s\n' "$preprocessed" | \ + sed -n 's/^#[[:space:]]*[0-9]\{1,\}[[:space:]]*"\([^"]*ldb\.h\)".*/\1/p' | head -1) +LDB_HEADER="${LDB_HEADER:-}" + +# Re-run with -P: without it, the line markers gcc emits around the expansion +# split "LDB_COMPAT_PROBE" and the version string onto different lines. +HEADER_VERSION=$($CC_BIN $CPPFLAGS -E -P "$probe_src" 2>/dev/null | tr '\n' ' ' | \ + sed -n 's/.*LDB_COMPAT_PROBE[[:space:]]*"\([^"]*\)".*/\1/p' | head -1) + +if [ -z "$HEADER_VERSION" ] ; then + cat >&2 <\n#include \nint main(void){char *v=NULL;ldb_version(&v);printf("%%s\\n",v?v:"");return 0;}\n' > "$probe_src" + +if $CC_BIN $CPPFLAGS "$probe_src" -o "$probe_bin" -lldb -lm -lpthread -ldl >/dev/null 2>&1 ; then + LINKED_VERSION=$("$probe_bin" 2>/dev/null | head -1 | tr -d '\n') + LINKED_PATH=$(ldd "$probe_bin" 2>/dev/null | sed -n 's/.*libldb\.so[^=]*=> \([^ ]*\).*/\1/p' | head -1) + LINKED_PATH="${LINKED_PATH:-the libldb.so resolved by the loader}" + + if [ -z "$LINKED_VERSION" ] ; then + echo "WARNING: could not read a version out of ${LINKED_PATH}; relying on ${LDB_HEADER} alone." >&2 + else + if [ "$LINKED_VERSION" != "$HEADER_VERSION" ] ; then + cat >&2 <&2 +fi + +# --- 3. The shell binary on PATH (hint only) -------------------------------- + +if command -v ldb >/dev/null 2>&1 ; then + SHELL_VERSION=$(ldb -v 2>/dev/null | head -1 | tr -d '\n') + SHELL_VERSION="${SHELL_VERSION#ldb-}" + if [ -n "$SHELL_VERSION" ] && [ "$SHELL_VERSION" != "$HEADER_VERSION" ] ; then + echo "WARNING: the 'ldb' binary on PATH ($(command -v ldb)) reports ${SHELL_VERSION}, but the engine builds against ${HEADER_VERSION}." >&2 + fi +fi + +echo "LDB ${HEADER_VERSION} found (requires ${REQUIRED} or later) - OK" +exit 0 diff --git a/src/ldb_compat.c b/src/ldb_compat.c new file mode 100644 index 00000000..27fa86fd --- /dev/null +++ b/src/ldb_compat.c @@ -0,0 +1,159 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * src/ldb_compat.c + * + * Run time validation of the LDB library the engine is loaded against. + * + * Copyright (C) 2018-2025 SCANOSS.COM + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation, either version 2 of the License, or + * (at your option) any later version. + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +/** + * @file ldb_compat.c + * @date 2025 + * @brief Validate, at run time, that the loaded libldb belongs to the release + * line this engine requires. + * + * libldb is linked dynamically, so the library resolved at load time may be a + * different build from the one the engine was compiled against. The build time + * counterpart of this check lives in scripts/check_ldb_version.sh; both derive + * the requirement from inc/ldb_compat.h so they cannot diverge. + */ + +#include +#include +#include +#include + +#include "ldb_compat.h" +#include "scanoss.h" +#include "debug.h" + +/** + * @brief Parse an LDB version string into its semantic version components. + * + * Accepted spelling is MAJOR[.MINOR[.PATCH]][-SUFFIX], optionally preceded by + * the "ldb-" prefix that the ldb shell prints. Missing MINOR/PATCH default to + * zero. Anything after the first '-' is the release line suffix. + * + * @param version version string to parse + * @param major output, major number + * @param minor output, minor number + * @param patch output, patch number + * @param suffix output buffer for the release line suffix (empty if none) + * @param suffix_ln size of the suffix output buffer + * @return true if the numeric components could be parsed + */ +bool ldb_compat_parse(const char *version, int *major, int *minor, int *patch, char *suffix, int suffix_ln) +{ + *major = *minor = *patch = 0; + if (suffix_ln > 0) *suffix = 0; + + if (!version) return false; + + /* Skip an optional "ldb-" prefix */ + if (!strncmp(version, "ldb-", 4)) version += 4; + + /* A version has to start with a digit */ + if (!isdigit((unsigned char) *version)) return false; + + int *component[3] = {major, minor, patch}; + int i = 0; + const char *p = version; + + while (i < 3) + { + if (!isdigit((unsigned char) *p)) return false; + + long value = strtol(p, (char **) &p, 10); + if (value < 0 || value > 1000000) return false; + *component[i++] = (int) value; + + if (*p != '.' || i == 3) break; + p++; + } + + /* Anything left has to be the suffix, introduced by '-' */ + if (*p == '-') + { + p++; + if (suffix_ln > 0) + { + strncpy(suffix, p, suffix_ln - 1); + suffix[suffix_ln - 1] = 0; + } + } + /* A trailing '\n' (or nothing at all) is fine, anything else is not */ + else if (*p && *p != '\n') return false; + + return true; +} + +/** + * @brief Abort the engine unless the loaded LDB satisfies inc/ldb_compat.h + * + * Reports "too old" and "wrong release line" as two distinct failures: they + * call for different fixes on the user's side. + */ +void ldb_compat_check(void) +{ + char *ldb_ver = NULL; + ldb_version(&ldb_ver); + scanlog("ldb version: %s\n", ldb_ver ? ldb_ver : "(unknown)"); + + int major = 0, minor = 0, patch = 0; + char suffix[64] = "\0"; + + if (!ldb_ver || !ldb_compat_parse(ldb_ver, &major, &minor, &patch, suffix, sizeof(suffix))) + { + fprintf(stderr, + "ERROR: could not determine the version of the loaded LDB library (reported: %s).\n" + " SCANOSS engine %s requires LDB %s or later.\n", + ldb_ver ? ldb_ver : "nothing", SCANOSS_VERSION, LDB_REQUIRED_VERSION); + free(ldb_ver); + exit(EXIT_FAILURE); + } + + /* Numeric floor, compared component by component (never lexicographically: + * strcmp() would rank "5.10.0" below "5.9.0"). */ + long found = (long) major * 1000000 + (long) minor * 1000 + patch; + long required = (long) LDB_REQUIRED_VERSION_MAJOR * 1000000 + + (long) LDB_REQUIRED_VERSION_MINOR * 1000 + LDB_REQUIRED_VERSION_PATCH; + + if (found < required) + { + fprintf(stderr, + "ERROR: the loaded LDB library is version %s, which is too old.\n" + " SCANOSS engine %s requires LDB %s or later.\n" + " Update LDB from the '%s' branch of https://github.com/scanoss/ldb\n", + ldb_ver, SCANOSS_VERSION, LDB_REQUIRED_VERSION, LDB_REQUIRED_RELEASE_LINE); + free(ldb_ver); + exit(EXIT_FAILURE); + } + + /* Release line. An MD5-only LDB can clear the numeric floor above and still + * be the wrong library: the table layout and the API differ. */ + if (strcmp(suffix, LDB_REQUIRED_RELEASE_LINE)) + { + fprintf(stderr, + "ERROR: the loaded LDB library is version %s, which belongs to the %s release line.\n" + " SCANOSS engine %s requires the '%s' release line of LDB (%s or later).\n" + " Install LDB from the '%s' branch of https://github.com/scanoss/ldb\n", + ldb_ver, *suffix ? "wrong" : "MD5-only", SCANOSS_VERSION, + LDB_REQUIRED_RELEASE_LINE, LDB_REQUIRED_VERSION, LDB_REQUIRED_RELEASE_LINE); + free(ldb_ver); + exit(EXIT_FAILURE); + } + + free(ldb_ver); +} diff --git a/src/main.c b/src/main.c index c9eb932e..53eaeb25 100644 --- a/src/main.c +++ b/src/main.c @@ -43,6 +43,7 @@ #include "purl_scan.h" #include "scanoss.h" #include "util.h" +#include "ldb_compat.h" #include "component.h" #include #include "hpsm.h" @@ -76,7 +77,6 @@ int scan_ranking_threshold = -1; //disable by defaults bool scan_honor_file_extension = SNIPPETS_DEFAULT_HONOR_FILE_EXTENSION; bool lib_encoder_present = false; -#define LDB_VER_MIN "4.1.0" void * lib_encoder_handle = NULL; bool lib_encoder_load() @@ -123,17 +123,11 @@ bool lib_encoder_load() void initialize_ldb_tables(char *name) { - char * ldb_ver = NULL; - ldb_version(&ldb_ver); - scanlog("ldb version: %s\n", ldb_ver); - - if (!ldb_ver || strcmp(ldb_ver, LDB_VER_MIN) < 0) - { - fprintf(stderr, "The current ldb version %s is too old, please upgrade to %s to proceed\n", ldb_ver, LDB_VER_MIN); - exit(EXIT_FAILURE); - } - free(ldb_ver); - + /* Validate the LDB actually loaded: libldb is linked dynamically, so it is + not necessarily the build the engine was compiled against. The minimum is + declared once, in inc/ldb_compat.h, and shared with the build time check. */ + ldb_compat_check(); + char oss_db_name[MAX_ARGLN]; if (name) strcpy(oss_db_name, name); @@ -295,7 +289,7 @@ static struct option long_options[] = { {"tolerance", required_argument, 0, 'T'}, {"sbom", required_argument, 0, 's'}, {"blacklist", required_argument, 0, 'b'}, - {"force-snippet", required_argument, 0, 256}, /* Long option only, no short form */ + {"force-snippet", no_argument, 0, 256}, /* Long option only, no short form */ {"snippet-scan", required_argument, 0, 'S'}, {"component", required_argument, 0, 'c'}, {"key", required_argument, 0, 'k'},