Skip to content

Give each package its own version and release only what changed - #37

Draft
jwrosewell wants to merge 9 commits into
rcfrom
feature/own-package-versions
Draft

jwrosewell wants to merge 9 commits into
rcfrom
feature/own-package-versions

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Step 3.1 of common-ci#306. ci/build-package.ps1 line 35 runs npm version with the run's one version for every package, so all 16 packages published 4.5.102 on 4 November 2026 (Publish run 41918901313) whatever had changed. That spends every package's allowance under the weekly release rule.

The decisions in the run book apply. Each package has its own version, tagged <package>/<version>, continuing from the version it last published, and only the patch number moves. A package is released when a file under its folders changed since its last tag, and every package that depends on it is released with it, with its requirement raised to the new version.

Change

  • ci/packages.json lists the 16 packages with Name, Path, ExtraPaths, DependsOn and Publish, read by common-ci's steps/release-plan.ps1 from common-ci#307. DependsOn is the packages here that each remote_package.json requires. Both on-premise packages have cxx in ExtraPaths, because their native modules compile from it.
  • Each package's examples folder is in a field of its own, ExcludePaths, so that a change only to examples releases nothing, which is check 4 of the run book. The examples sit inside the package folders here, unlike in 51degrees-python, where they are a package of their own that is never published. The two shared packages keep their examples folders, because their index.js exports the modules in them. location/docs is excluded too, as it is the only documentation folder inside a package.
  • ci/release-plan.ps1 loads common-ci's step from the common-ci root that the shared workflows run these scripts from. It gives git each ExcludePaths folder as an exclude pathspec after the package's own, and it adds the build plan and the versions the requirements are set from. Only the publish scripts load it, so the pull request builds, which run ci/build-project.ps1, do not need common-ci's step.
  • ci/get-next-package-version.ps1 gives the run the highest planned version as its label, and the new ci/package-update-required.ps1 prints the plan and answers 1 when it is empty, as common-ci's nightly-publish.configure.ps1 expects.
  • ci/build-package.ps1 packs only the planned packages, each at its own version. Each requirement on a package here becomes ^ and that package's version once the release is out, which is its planned version, or its last released version when it is not in the plan. The build stops when a manifest and DependsOn disagree. When nothing is planned, every package is packed at its released version, as 51degrees-python does, so the nightly publish still tests packed packages.
  • ci/install-package.ps1 builds a native module from source when its package has no tarball, because that package is not being released.
  • ci/publish-package.ps1 publishes only the planned tarballs, then tags each package and makes one GitHub release per package with its tarball attached, in place of common-ci's single tag.
  • ci/check-version-bump.ps1, modelled on 51degrees-dotnet's, runs common-ci's steps/check-version-bump.ps1 for each planned package against its own tags, and again against the version npm's latest dist-tag gives, so that a tag pushed by hand or a FirstVersion cannot set where a package starts.
  • ci/check-release-interval.ps1 runs common-ci's steps/check-release-interval.ps1 with -TagPrefix for each planned package. 51degrees-dotnet has no such script on rc, so this follows the comment above that step in common-ci's nightly-publish.yml.
  • ci/packages.ps1 loses $PublishedPackages, which nothing reads now, and AGENTS.md describes the scheme in place of the single version.

Both check scripts run only once this repository is in the $OwnCheck lists of common-ci's nightly-publish.yml. Until then the shared check compares the run's label with the plain tags, which still refuses a minor move of any package, because the label is the highest planned version.

Before this reaches main

The publish scripts need steps/release-plan.ps1, which is on common-ci's rc only, and the workflows here take common-ci from main. So common-ci's release of common-ci#307 to main has to come before the release of this repository's rc into main that carries this change. Merging into rc first changes nothing that runs, because the publish runs on main.

The first release

Step 4 of the run book. Before the first publish that runs these scripts, each package is tagged at the version it last published, so that each continues from there. All 16 published 4.5.102 from commit 384bff2d, which the Package job of run 41918901313 cloned before it published the 16 packages and pushed tag 4.5.102, and npm's latest is 4.5.102 for every one of them. The tags, all on 384bff2d52cacba2e3315fdaaa1eca74a58be6ec, are:

fiftyone.pipeline.core/4.5.102
fiftyone.pipeline.engines/4.5.102
fiftyone.pipeline.engines.fiftyone/4.5.102
fiftyone.pipeline.cloudrequestengine/4.5.102
fiftyone.pipeline.did/4.5.102
fiftyone.pipeline.translation/4.5.102
fiftyone.devicedetection.shared/4.5.102
fiftyone.devicedetection.cloud/4.5.102
fiftyone.devicedetection.onpremise/4.5.102
fiftyone.devicedetection/4.5.102
fiftyone.geolocation/4.5.102
fiftyone.ipintelligence.shared/4.5.102
fiftyone.ipintelligence.cloud/4.5.102
fiftyone.ipintelligence.onpremise/4.5.102
fiftyone.ipintelligence.translation/4.5.102
fiftyone.ipintelligence/4.5.102

They should be lightweight tags with no GitHub release, for example gh api -X POST repos/51Degrees/51degrees-node/git/refs -f ref=refs/tags/fiftyone.pipeline.core/4.5.102 -f sha=384bff2d52cacba2e3315fdaaa1eca74a58be6ec. The weekly check dates an annotated tag by when it was made and a release by when it was published, which would hold every package for a week from the day of tagging. A lightweight tag with no release is dated by its commit, 4 November 2026. The scripts now on main read only plain tags such as 4.5.102, so the tags can be pushed at any time before that publish. No tag has been pushed.

Checks

Run 42368202986 laid the repositories out as common-ci's publish does, with common-ci's rc (853b913d, the merge of common-ci#307) at the root and this branch inside it, made the 16 tags above locally and ran the publish's scripts in order. The plan it printed:

Name                                 Previous Version Reason
fiftyone.pipeline.core               4.5.102  4.5.103 changed since 4.5.102
fiftyone.pipeline.engines            4.5.102  4.5.103 a package it depends on is released
fiftyone.pipeline.translation        4.5.102  4.5.103 a package it depends on is released
fiftyone.ipintelligence.translation  4.5.102  4.5.103 a package it depends on is released
fiftyone.pipeline.cloudrequestengine 4.5.102  4.5.103 a package it depends on is released
fiftyone.pipeline.engines.fiftyone   4.5.102  4.5.103 changed since 4.5.102
fiftyone.devicedetection.cloud       4.5.102  4.5.103 a package it depends on is released
fiftyone.devicedetection.onpremise   4.5.102  4.5.103 a package it depends on is released
fiftyone.geolocation                 4.5.102  4.5.103 a package it depends on is released
fiftyone.ipintelligence.cloud        4.5.102  4.5.103 a package it depends on is released
fiftyone.ipintelligence.onpremise    4.5.102  4.5.103 a package it depends on is released
fiftyone.devicedetection             4.5.102  4.5.103 a package it depends on is released
fiftyone.ipintelligence              4.5.102  4.5.103 a package it depends on is released

Since 4.5.102, rc has changed fiftyone.pipeline.core (#31) and fiftyone.pipeline.engines.fiftyone (#35), and 13 packages depend on one of them. fiftyone.pipeline.did, fiftyone.devicedetection.shared and fiftyone.ipintelligence.shared stay at 4.5.102.

  • The version check passed every package against its own tags and against npm. The weekly check reported that each package may next go from 11 November 2026 and did not refuse, as WEEKLY_RELEASES_FROM is not set.
  • The build packed those 13 tarballs only. Each requires the planned packages at ^4.5.103, and fiftyone.devicedetection.shared and fiftyone.ipintelligence.shared at ^4.5.102, their last release.
  • The dry-run publish ran npm publish --dry-run for the 13 tarballs, tagged each <package>/4.5.103 locally, pushed nothing and set common-ci's SkipUpdateTag.

The same run then made local commits on top of tags at one commit, every package at 4.5.102 with fiftyone.geolocation at 4.5.110 and fiftyone.devicedetection.cloud at 4.5.107, and checked each plan. Every check passed.

  1. A change to fiftyone.geolocation planned it alone at 4.5.111, and one to fiftyone.ipintelligence.translation planned it alone at 4.5.103.
  2. A change to fiftyone.pipeline.engines planned it and its 9 dependents, each after the packages it depends on. A change to fiftyone.pipeline.core planned 13 packages. A move of the cxx submodule planned both on-premise packages, fiftyone.devicedetection and fiftyone.ipintelligence. A change to the exported examples of fiftyone.devicedetection.shared planned it and its 3 dependents.
  3. Each planned version was its own last tag plus one patch, fiftyone.devicedetection.cloud 4.5.108 and fiftyone.geolocation 4.5.111 among them. The tarballs of the fiftyone.pipeline.engines plan required fiftyone.pipeline.core, which was not planned, at ^4.5.102, and fiftyone.devicedetection.cloud at ^4.5.108. With a tag fiftyone.pipeline.engines/4.6.0 pushed by hand, the plan gave 4.6.1, which passed against the package's tags and was refused against npm's 4.5.102 as a minor move.
  4. A change to ci/options.json, a workflow, README.md, AGENTS.md, three docs folders, a product readme.md and the examples of five packages planned nothing, and ci/package-update-required.ps1 answered 1.

Run 42368758379 checked the publish's Test job for a plan of one package. With a change to fiftyone.geolocation alone, the build packed fiftyone.geolocation-4.5.103.tgz only, and ci/install-package.ps1 installed it from that tarball, left fiftyone.pipeline.core linked to the source, and built both native modules from source.

Both runs tested the first commit, 27823384. The second commit, b863ad0e, corrects two comments and changes no code, and run 42369537155 repeats the dry run on it.

No package was built for real and nothing was published or pushed. The temporary branches are deleted once their runs finish.

The release rule of 7 November

Commit b6d4f8d applies the decision that a package is not released when the only changes under its folders since its last tag are to documentation, tests, examples or manifest metadata such as links. It needs common-ci#310, which leaves out **/*.md, **/docs/**, **/tests/**, **/test/**, **/examples/** and **/LICENSE* unless a package sets IgnoreDefaults to false, adds each package's Ignore globs, and calls ci/release-plan-filter.ps1 with the files left.

  • ci/release-plan-filter.ps1 does not count a package.json or remote_package.json whose only changes since the tag are to description, keywords, homepage, repository, bugs, author, license, files or readme. A change to any other field counts, version, the dependencies, main, exports and scripts among them, as do a manifest added or removed and one that is not JSON. Fields are compared as values with their keys sorted, so a change only to the layout or the order of the fields does not count.
  • Every package's Ignore holds **/__tests__/**, the other folder name jest finds tests in. No package has one today.
  • The two shared packages hold their shipped modules in lib, not in examples and tests, so they use the default ignore list like every other package. Their index.js used to export tests/keyUtils.js, tests/testConstants.js, tests/exampleOutput.js and the modules in examples, which is why they had set IgnoreDefaults to false. __tests__/layout.test.js in each package checks the layout. No file in the repository imports the old paths, index.js is the main entry and the only documented way in, and the generated types moved with the code. A deep import of an old path such as fiftyone.devicedetection.shared/examples/dataExtension stops working, and no re-export was left because it would have to sit inside examples.
  • ci/release-plan-filter.Tests.ps1 holds 25 Pester tests of the filter. Nothing in this repository runs them yet, so run them with Invoke-Pester ./ci/release-plan-filter.Tests.ps1.

Run 42396135517 checked out common-ci at feature/release-plan-ignore (7eb3d89, the head of common-ci#310) and this branch with the 16 tags pushed on 7 November. The Pester tests passed, 25 of 25. The plan at HEAD was the 13 packages above, with the reasons changed since 4.5.102 (2 files) for fiftyone.pipeline.core and (6 files) for fiftyone.pipeline.engines.fiftyone, and ci/package-update-required.ps1 answered 0. Each of 14 local commits, reset after each, planned what it should.

Change Planned
fiftyone.pipeline.did package.json homepage only no
fiftyone.pipeline.did package.json description and keywords no
fiftyone.pipeline.did package.json dependency yes, 4.5.103
fiftyone.pipeline.did package.json version yes, 4.5.103
fiftyone.pipeline.did package.json scripts yes, 4.5.103
fiftyone.pipeline.did package.json homepage and usage.js yes, 4.5.103 (1 file)
fiftyone.pipeline.did a test, the readme and an example no
fiftyone.pipeline.did a new __tests__ folder no
fiftyone.pipeline.did remote_package.json homepage only no
fiftyone.pipeline.did remote_package.json dependency yes, 4.5.103
fiftyone.devicedetection.shared lib/keyUtils.js yes, 4.5.103
fiftyone.devicedetection.shared readme.md only no
fiftyone.ipintelligence.shared lib/keyUtils.js yes, 4.5.103
fiftyone.ipintelligence.shared lib/dataExtension.js yes, 4.5.103

Shared packages hold their code outside examples and tests (9 November)

James Rosewell decided that examples, tests and docs must never cause a package release. fiftyone.devicedetection.shared and fiftyone.ipintelligence.shared shipped modules from their own examples and tests folders through index.js, so the default ignore either skipped a real change to shipped code or, with the defaults off as before, released the package for an example change.

  • Commit 6832fd5 moves dataExtension.js, exampleConstants.js and optionsExtension.js from examples, and exampleOutput.js, keyUtils.js and testConstants.js from tests, into a lib folder in each package, with the generated types. It removes IgnoreDefaults and the extra **/*.md and **/LICENSE* entries from both packages in ci/packages.json, so both use the default ignore list like every other package. tests/exampleOutput.js for IP Intelligence arrived on rc afterwards, and the merge of rc (98a1a02) puts it in lib too.
  • The old paths keep working. Commit 71f7427 leaves a one-line file at each old path, for example examples/keyUtils.js holding module.exports = require('../lib/keyUtils');, so require('fiftyone.devicedetection.shared/examples/keyUtils') and the other five paths in each package return the same module as before. files in package.json lists lib, exactly those twelve files (six per package), *.js and types. The release plan still ignores examples and tests, which is right because the files never change, and the code in lib decides whether a package is released. The generated types are not duplicated at the old paths, so only the JavaScript paths are kept.
  • __tests__/layout.test.js in each package checks that index.js exports every module, that index.js requires nothing from an examples or tests folder, that npm pack --dry-run lists every lib file and the old-path files and nothing else under examples or tests, that each old path returns the same module as the file in lib, and that each old file is a one-line re-export.
  • Red: run 42685837747 failed 4 of 6 tests on the first layout test before the move. For the old-path tests, run 42712988878 ran the new tests on 4a12263 without the files and both suites failed to run, because the examples and tests folders did not exist.
  • Green: run 42713016047 at head 71f7427 plus a workflow. Both suites passed, 10 of 10 tests. The earlier green run 42690191355 showed tsc leaving the types unchanged, require from the on-premise and IP Intelligence cloud packages returning every export, and eslint passing.
  • Full build: run 42714138681, Setting 'PullRequestSha' 71f7427 on both lines. All six Ubuntu, Ubuntu ARM and Windows jobs passed. macOS was left out of this draft's full build under the 9 November decision. The rc head is built with macOS before it merges into main.
  • Not run: the plan itself against these packages, which needs common-ci#312 on main.

Outstanding

  • This repository needs adding to both $OwnCheck lists in common-ci's nightly-publish.yml, and common-ci#307 has to reach common-ci's main, as above.
  • A change only to a package's own readme.md no longer releases it, because common-ci's defaults leave out **/*.md, so a README fix reaches npm with the package's next release for another reason.
  • ExcludePaths and common-ci's default **/examples/** and **/docs/** now leave out the same folders, so ExcludePaths and the code in ci/release-plan.ps1 that reads it could go.
  • npm publishes the tarballs in name order, as before, not in dependency order. All of them go in one step.
  • If the publish fails part way, the packages already on npm are not tagged, as before this change, and the next publish would plan them again at the same version, which npm refuses.
  • The PreBuild jobs still build both native modules on every entry when neither on-premise package is planned.

Each of the 16 packages is listed in ci/packages.json and tagged
<package>/<version>. common-ci's steps/release-plan.ps1 works out which
packages a publish releases. A package is released when a file under its
folders changed since its last tag, and every package depending on it is
released with it. Only the patch number moves.

ci/build-package.ps1 packs only the planned packages, each at its own
version, and requires each package here at the version it has once the
release is out. ci/publish-package.ps1 publishes only those tarballs and
tags each package. ci/check-version-bump.ps1 and
ci/check-release-interval.ps1 check each planned package on its own.

Step 3.1 of 51Degrees/common-ci#306.
ci/release-plan.ps1 looks at COMMON_CI before the common-ci root, and
ci/check-version-bump.ps1 runs the shared check twice for each package.
common-ci's release plan now leaves out documentation, tests, examples
and LICENSE files, and calls a repository's ci/release-plan-filter.ps1
for the files left. This repository's filter does not count a
package.json or remote_package.json whose only changes since the
package's last tag are to description, keywords, homepage, repository,
bugs, author, license, files or readme. A change to any other field,
such as version, dependencies, main, exports or scripts, counts.

Every package also ignores __tests__ folders, the other folder name
jest finds tests in. The two shared packages export modules from their
tests and examples folders, so they turn the defaults off and ignore
only their Markdown and LICENSE files.

ci/release-plan-filter.Tests.ps1 holds the Pester tests of the filter.
fiftyone.devicedetection.shared and fiftyone.ipintelligence.shared
published the modules in their examples and tests folders through
index.js, so the release plan could not ignore those folders for them.
The modules now live in lib, the packages use the default ignore list,
and a test checks the layout.
# Conflicts:
#	ip-intelligence/fiftyone.ipintelligence.shared/index.js
#	ip-intelligence/fiftyone.ipintelligence.shared/lib/exampleOutput.js
#	ip-intelligence/fiftyone.ipintelligence.shared/types/lib/exampleOutput.d.ts
Each old path under examples and tests is a one-line re-export of the module
now in lib, and is listed in files so that it is published. The release plan
still ignores both folders, and the stubs never change, so the code in lib
decides whether a package is released.
…ports fixed

A change to files in package.json decides what a package ships, so it counts
towards a release. The layout test now names the six re-exports, so deleting
one fails, and each re-export carries the licence header. AGENTS.md no longer
calls the shared packages an exception.
@jwrosewell jwrosewell mentioned this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant