[jwrosewell]

Makes the licence files in this repository the same as in 51degrees-cxx, with one standard LICENSE at the root and the two additional terms stated only in each source file's header, and takes pipelineOptions.xsd from configuration-validation as a submodule rather than keeping a copy.

Problem

The root LICENSE (blob 4153cd3775, 13,827 bytes) held the EUPL v1.2 text with its title in capitals and one quotation mark different from the text in 51degrees-cxx, and five identical copies sat in device-detection/, examples/device-detection/, examples/ip-intelligence/, ip-intelligence/ and pipeline/. location/LICENSE (blob 43159c60bc) held the 51Degrees header form, being the copyright comment, then the EUPL text, then the two additional terms. No standalone file at the root carried the two additional terms.

pipeline/pipeline.core/src/main/resources/pipelineOptions.xsd (blob e04f6dc414) was a copy of configuration-validation's pipelineOptions.xsd with no record of where it came from, whilst the .NET pipeline takes the same repository as the submodule pipeline/Core/Configuration/Validation.

The decision

James Rosewell decided on 11 November 2026, in these words, "Follow .NET and CXX. Single LICENSE on root that is the standard text. Then each file has the standard 51Degrees header with the exception of the files with patents in them where the patents are also listed." At about 20:05 UTC the same day James decided, in these words, "LICENSE only, remove NOTICE", for every repository, 51degrees-cxx included. So the two additional terms, the Compatible Licences clause and the network application clause, stay in each source file's header, which is where they apply, and no standalone file carries them. The standard text is the one 51degrees-cxx #105 put in cxx's LICENSE.

The first commit here added a NOTICE as cxx has it and named it in the README. The second commit takes both out again after the 20:05 decision, so the change as a whole adds no NOTICE and leaves the README's licence line as it was.

What changed

1. LICENSE is now the EUPL v1.2 text byte for byte as 51degrees-cxx has it (blob a74f04438a, 13,865 bytes, UTF-8).
2. The six other copies are removed, being device-detection/LICENSE, examples/device-detection/LICENSE, examples/ip-intelligence/LICENSE, ip-intelligence/LICENSE, location/LICENSE and pipeline/LICENSE. Nothing in the tree referred to them.
3. pipeline/pipeline.core/configuration-validation is a submodule of ../configuration-validation (relative, as .NET has it) at b2740eddf2, the head of its main. The copy in src/main/resources is removed, and pipeline/pipeline.core/pom.xml adds the submodule's pipelineOptions.xsd, and only that file, as a resource. The file lands at the root of the class path as before, which is where PipelineOptionsFactory.java line 52 loads it with getClassLoader().getResource("pipelineOptions.xsd"). The submodule's pipelineOptions.xsd is blob e04f6dc414, the same bytes as the removed copy. .NET pins 49fd2e7540, one commit behind, and that commit changes only .github/workflows/only-rc-into-main.yml. src/main/resources stays in the resource list, so a file added there later is still built in.

The blob hashes were checked with git ls-files -s after staging, so they are what git holds and not a line ending conversion of them.

Packages

Maven Central takes the licence from the pom. Its requirements say "You need to declare the license(s) used for distributing your components." and ask for a <licenses> element, not a file. The eight parent poms declare "European Union Public License 1.2" with the URL https://opensource.org/licenses/EUPL-1.2, and the other modules inherit it. None of the removed copies sat in a src/main/resources folder, so none was in a jar. pipeline.core 4.5.130 and 4.6.0 and pipeline.javascriptbuilder 4.6.0 on Maven Central hold no licence file, and both pipeline.core jars hold pipelineOptions.xsd at their root. No package changes from the licence files.

A build from a clone without submodules now has no pipelineOptions.xsd on the class path, so getOptionsFromFile has no schema to load. The tests in pipeline.core's OptionsTest call it, and the full build below checks out the submodules and runs them.

Source headers

Every source file on rc at 53ccddd was checked against licenseText and the three patent variants in tools CopyrightUpdater appsettings.json, with files selected as the updater selects them (Program.cs lines 349 to 356). That means the .java files git tracks, less files under a folder named docs, build, debug, release, .vs, obj, packages, bin, wwwroot, node_modules, swig or vendor, or named like the last folder of a submodule path. Years are ignored, because the monthly copyright update sets them.

Files checked	Before	After
Standard header	698	698
Patent variant	0	0
Other wording	0	0
No header	0	0
Total	698	698

No source file needed a change. Left alone, because the updater skips them:

• 88 SWIG wrappers in the swig packages of the two on-premise engines, 37 in device-detection/device-detection.hash.engine.on-premise and 51 in ip-intelligence/ip-intelligence.engine.on-premise, which SWIG generates.
• pipeline/pipeline.javascriptbuilder/src/main/java/fiftyone/pipeline/javascriptbuilder/templates/JavaScriptResource.java, which the updater skips because the submodule folder beside it is also called templates. It already carries the standard header.

The other licence files are left as they are, being pipeline/pipeline.did/src/main/resources/META-INF/LICENSE-owid.txt (Apache 2.0, OWID) and NOTICE.txt beside it, which is the attribution the Apache licence asks for when the OWID source is bundled, not the two additional terms. Submodule contents (cxx, the JavaScript templates, pipeline/owid-java and the new configuration-validation) were not checked, because each repository keeps its own.

No file here carries a patent variant, and none was added. The Hash and IP Intelligence engines here call the C code in 51degrees-cxx through the SWIG wrappers, and the cxx files that implement the patented methods carry the variants (for example src/dd/hash/hash.c and src/ipi/graph/graph.c). If any file in this repository should carry one, that is a question for review.

The script that classed the headers

Run as python -I headers.py appsettings.json <clone> java report --skipped, with appsettings.json from tools at 0605b1d5.

"""Class every source file's licence header against the tools CopyrightUpdater.

Usage: python -I headers.py <appsettings.json> <repository root> <language>
           [report|variants|fix] [--skipped] [--list <category>]

<language> is dotnet, java, node or python. The file selection follows
CopyrightUpdater Program.cs: the files git tracks with the language's
extensionsToUpdate, minus extensionsToIgnore, minus any file under a folder
named in dirsToIgnore or named as the last part of a submodule path. Files
with the language's extensions that the updater skips are counted
separately, and --skipped classes them too.

Categories:
  standard         the licenseText header, in the language's comment form
  standard-plus    Python only, the standard header followed in the same run
                   of comment lines by other comments
  patent-standard  a patent variant, word for word
  other            a copyright comment that differs from licenseText
  patent-other     a patent variant that differs from its text
  none             no copyright comment
  generated        SWIG output, a minified file, or marked as generated
Years are wildcards, so a header differing only in its year is standard.

fix replaces an "other" or "patent-other" comment with the matching text,
keeping its year, and adds licenseText with the current year to a "none"
file, after any shebang and Python coding line.
"""
import datetime
import json
import re
import subprocess
import sys
from collections import Counter, defaultdict

LANG = {
    "dotnet": [".cs", ".cshtml"],
    "java": [".java"],
    "node": [".js"],
    "python": [".py"],
}
PATENTS = [("2871816", "patternLicenseText"), ("3438848", "hashLicenseText"),
           ("2506025.2", "ipgraphLicenseText")]
YEAR = "(?P<year>[0-9]{4})"


def form(text, path, lang):
    """The header in the comment syntax the updater writes for the file."""
    if path.endswith(".cshtml"):
        text = text.replace("/*", "@*").replace("*/", "*@")
    if lang == "python":
        text = text.replace("/*", "#").replace(" */", "")
        text = text.replace("\n *", "\n#")
    return text


def template_regex(text):
    return re.compile(re.escape(text).replace(re.escape("{0}"), YEAR))


def submodule_names(root):
    names = []
    try:
        for line in open(f"{root}/.gitmodules", encoding="utf-8"):
            s = line.strip()
            if s.startswith("path"):
                names.append(re.split(r"[=/]", s)[-1].strip())
    except FileNotFoundError:
        pass
    return names


def select(root, lang, settings):
    out = subprocess.run(["git", "-C", root, "ls-files", "-z", "-s"],
                         stdout=subprocess.PIPE, check=True).stdout
    files, skipped = [], defaultdict(list)
    ignore_dirs = [d.lower() for d in settings["dirsToIgnore"]]
    sub = [s.lower() for s in submodule_names(root)]
    for entry in out.decode("utf-8").split("\0"):
        if not entry:
            continue
        meta, path = entry.split("\t", 1)
        if meta.startswith("160000"):
            continue
        low = path.lower()
        if not any(low.endswith(e) for e in LANG[lang]):
            continue
        dirs = low.split("/")[:-1]
        hit = [d for d in dirs if d in ignore_dirs]
        hit_sub = [d for d in dirs if d in sub]
        ext = [e for e in settings["extensionsToIgnore"] if low.endswith(e)]
        if hit:
            skipped[f"folder {hit[0]}"].append(path)
        elif hit_sub:
            skipped[f"submodule name {hit_sub[0]}"].append(path)
        elif ext:
            skipped[f"extension {ext[0]}"].append(path)
        else:
            files.append(path)
    return files, skipped


def blocks(text, lang, path):
    """Comment blocks near the top of the file as (start, end) spans."""
    found = []
    if lang == "python":
        pos, start, prev = 0, None, None
        for line in text.splitlines(keepends=True):
            if line.startswith("#") and not (pos == 0 and line.startswith("#!")):
                if start is None:
                    start = pos
                prev = pos + len(line)
            elif start is not None:
                found.append((start, prev))
                start = None
            pos += len(line)
            if pos > 20000:
                break
        if start is not None:
            found.append((start, prev))
        return found
    i = 0
    limit = min(len(text), 20000)
    while i < limit - 1:
        if text[i] in "/@" and text[i + 1] == "*":
            j = text.find("*/" if text[i] == "/" else "*@", i + 2)
            if j < 0:
                break
            found.append((i, j + 2))
            i = j + 2
        else:
            i += 1
    return found


def classify(raw, path, lang, regex):
    text = raw.replace("\r\n", "\n")
    spans = blocks(text, lang, path)
    first = spans[0] if spans else None
    cop = next((s for s in spans if "copyright" in text[s[0]:s[1]].lower()),
               None)
    if cop is None:
        if path.endswith(".min.js"):
            return "generated", None, text
        head = text[:3000].lower()
        if (first and "swig" in text[first[0]:first[1]].lower()) or \
                "automatically generated" in head or "auto-generated" in head \
                or "<auto-generated" in head or "generated by" in head:
            return "generated", None, text
        return "none", None, text
    block = text[cop[0]:cop[1]]
    low = block.lower()
    if "patent" in low:
        key = next((k for n, k in PATENTS if n in block), None)
        if key and regex[key].fullmatch(block.rstrip("\n")) or (
                key and lang == "python"
                and regex[key].match(block)):
            return "patent-standard", (cop, key, first == cop), text
        return "patent-other", (cop, key, first == cop), text
    if "swig" in low:
        return "generated", None, text
    if lang == "python":
        m = regex["licenseText"].match(block)
        if m and (block[m.end():] in ("", "\n")):
            return "standard", (cop, "licenseText", first == cop), text
        if m:
            return "standard-plus", (cop, "licenseText", first == cop), text
    elif regex["licenseText"].fullmatch(block):
        return "standard", (cop, "licenseText", first == cop), text
    return "other", (cop, "licenseText", first == cop), text


def main():
    settings_path, root, lang = sys.argv[1:4]
    mode = sys.argv[4] if len(sys.argv) > 4 else "report"
    settings = json.load(open(settings_path, encoding="utf-8"))
    files, skipped = select(root, lang, settings)
    counts = Counter()
    by = defaultdict(list)
    variants = defaultdict(list)
    regex_by_ext = {}
    results = []
    for path in files:
        ext = ".cshtml" if path.endswith(".cshtml") else "other"
        if ext not in regex_by_ext:
            regex_by_ext[ext] = {
                k: template_regex(form(settings[k], path, lang))
                for k in ("licenseText", "patternLicenseText",
                          "hashLicenseText", "ipgraphLicenseText")}
        data = open(f"{root}/{path}", "rb").read()
        raw = data.decode("latin-1")
        cat, info, text = classify(raw, path, lang, regex_by_ext[ext])
        counts[cat] += 1
        by[cat].append(path)
        if info and not info[2]:
            by["not-first-comment"].append(path)
        if cat in ("other", "patent-other", "standard-plus"):
            span = info[0]
            block = text[span[0]:span[1]]
            norm = re.sub(r"Copyright\s+[0-9][0-9 ,\-]*[0-9]", "Copyright YEAR",
                          block)
            variants[norm].append(path)
        results.append((path, cat, info))
    if mode == "report":
        print(f"selected {len(files)}")
        for k, v in sorted(counts.items()):
            print(f"  {k}: {v}")
        print(f"  not the first comment: {len(by['not-first-comment'])}")
        print("skipped by the updater's rules")
        for k, v in sorted(skipped.items()):
            print(f"  {k}: {len(v)}")
        if "--skipped" in sys.argv:
            for k, v in sorted(skipped.items()):
                c = Counter()
                for path in v:
                    ext = ".cshtml" if path.endswith(".cshtml") else "other"
                    reg = {kk: template_regex(form(settings[kk], path, lang))
                           for kk in ("licenseText", "patternLicenseText",
                                      "hashLicenseText", "ipgraphLicenseText")}
                    raw = open(f"{root}/{path}", "rb").read().decode("latin-1")
                    c[classify(raw, path, lang, reg)[0]] += 1
                print(f"  {k}: {dict(c)}")
        if "--list" in sys.argv:
            want = sys.argv[sys.argv.index("--list") + 1]
            for p in (by[want] if want in by else skipped.get(want, [])):
                print("   ", p)
    elif mode == "variants":
        for norm, paths in sorted(variants.items(), key=lambda kv: -len(kv[1])):
            print(f"===== {len(paths)} files, e.g. {paths[0]}")
            print(norm)
    elif mode == "fix":
        fix(root, lang, settings, results, regex_by_ext)


def newline_of(raw):
    i = raw.find("\n")
    return "\r\n" if i > 0 and raw[i - 1] == "\r" else "\n"


def fix(root, lang, settings, results, regex_by_ext):
    done = Counter()
    for path, cat, info in results:
        if cat not in ("other", "patent-other", "none"):
            continue
        full = f"{root}/{path}"
        data = open(full, "rb").read()
        raw = data.decode("latin-1")
        nl = newline_of(raw)
        crlf = nl == "\r\n"
        if cat == "none":
            header = form(settings["licenseText"], path, lang).replace(
                "{0}", str(datetime.date.today().year))
            header = header.replace("\n", nl)
            bom = ""
            body = raw
            if body.startswith("\xef\xbb\xbf"):
                bom, body = body[:3], body[3:]
            prefix = ""
            lines = body.splitlines(keepends=True)
            k = 0
            if lines and lines[0].startswith("#!"):
                k = 1
            if lang == "python" and len(lines) > k and re.match(
                    r"^#.*coding[:=]", lines[k]):
                k += 1
            prefix = "".join(lines[:k])
            rest = "".join(lines[k:])
            new = bom + prefix + header + nl + nl + rest.lstrip("\r\n")
            open(full, "wb").write(new.encode("latin-1"))
            done["added"] += 1
            continue
        span, key, _ = info
        if key is None:
            print(f"SKIP unknown patent {path}")
            continue
        # Map the normalised span back to the raw text.
        text = raw.replace("\r\n", "\n")
        block = text[span[0]:span[1]]
        m = re.search(r"Copyright\s+([0-9]{4})", block)
        if not m:
            print(f"SKIP no year {path}")
            continue
        header = form(settings[key], path, lang).replace("{0}", m.group(1))
        if lang == "python" and not block.endswith("\n"):
            pass
        elif lang == "python":
            header += "\n"
        new_text = text[:span[0]] + header + text[span[1]:]
        if crlf:
            if "\n" in raw.replace("\r\n", ""):
                print(f"SKIP mixed line endings {path}")
                continue
            new_text = new_text.replace("\n", "\r\n")
        open(full, "wb").write(new_text.encode("latin-1"))
        done["updated " + cat] += 1
    print(dict(done))


if __name__ == "__main__":
    main()

Full build

Full build 43111846204 is green. It started at 22:26:55 UTC on 11 November 2026 and finished at 23:34:23 UTC, on the throwaway branch tmp/pr-matrix-109-c, which was taken from rc at 03c3dd9 and held one temporary workflow running common-ci's pull request build on this draft with merging turned off. The branch is still in the repository at 68d938b, and whoever merges this draft should delete it, as the two branches of the earlier builds already have been.

Eleven jobs are logged and every one of them is a success, being Configure, the nine Build and Test jobs (Ubuntu Java 8, 11, 17 and 21, Ubuntu ARM Java 17, and Windows Java 8, 11, 17 and 21) and Complete. The two Compare Performance jobs were skipped, as they are on a draft. No job printed BUILD FAILURE.

All ten Setting 'PullRequestSha' lines, being Configure and the nine build jobs, read 5a43e068751e38479096c7bbd6d8d83d54e2ad00, which is this draft's head, so every job built this head and nothing else. Configure printed "Left out 6 macOS entries".

OptionsTest is the evidence that the schema still reaches the class path from the submodule. It ran in each of the nine build jobs and printed "Tests run: 5, Failures: 0, Errors: 0, Skipped: 0" in all nine (for example Ubuntu Java 17). Those five tests go through PipelineOptionsFactory.getOptionsFromFile, which reads the schema with getClassLoader().getResource("pipelineOptions.xsd"), so if the submodule's copy had not been built into the jar that call would give null and the tests would fail.

There is no red run for this change, because it moves a file and changes no behaviour, so there is no new test to write and show failing first. The standing test that would catch the move going wrong is OptionsTest, for the reason above, and it passes on every platform and every Java version the build covers.

macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.

Two earlier full builds of this draft never ran to completion and nothing here relies on them. 43091443200, on tmp/pr-matrix-109 at 5ff1b36, built an earlier head and was cancelled when the NOTICE came out of the change, and 43096291105, on tmp/pr-matrix-109b at d46a979, was created at 20:28 UTC inside the GitHub Actions outage and has sat queued ever since without starting. Both branches are deleted.

What remains

• The Environment variable names check fails on this head (run 43111207600) as it does on rc at 53ccddd (run 43088786083). Both stop at pipeline/pipeline.engines/src/test/java/fiftyone/pipeline/engines/services/DataUpdateServiceLoggingTests.java line 76, where the Java constant LICENSE_KEY starts a continued line and the check reads it as the old variable name. The file came with #79, this draft does not touch it, and the fix is #111, a change of its own. The check on this head only goes green once Name the data update logging test's key so it is not an old variable name #111 is in rc and this draft has taken rc.
• The branch is behind rc by #107 (03c3dd9) and #97 (4bec158). Between them they change six .java files in pipeline/pipeline.cloudrequestengine, none of which this draft touches, and each of the six still carries the standard header, so the merged tree has no file without one. GitHub reads the draft as MERGEABLE.
• The eight poms give the licence as <name>European Union Public License 1.2</name> with <url>https://opensource.org/licenses/EUPL-1.2</url>, and that does declare EUPL-1.2, because "European Union Public License 1.2" is word for word the full name the SPDX licence list gives for the identifier EUPL-1.2 (list version 3.29.0), and the identifier itself is in the URL. Maven Central asks for a name and a URL rather than an identifier, so the poms are left as they are. Putting the bare EUPL-1.2 in the <name> instead would be a one-word change in eight files, and that is a question for James rather than something to add to a licence change.

[jwrosewell]

Closing and reopening at once to start the checks again, because the runs queued during GitHub's Actions outage this evening never started.