Different commands not deterministic

[thejuan]

We just ran into an issue where we run multiple steps in a job i.e. a build step and a package step (that are separate)
If not every package has a package step, the packages in each partition are different to those in the build as packagesWithCommand is used. This causes problems if the second step is expecting artifacts from the first to be there

I'd propose a change whereby the partitions are calculated on all packages, then applied to the packagesWithCommand.

We can fix locally with a no-op command added to every package

[buschtoens]

Thanks for your suggestion and use-case, @thejuan. I'm sorry, that I'm only getting back to you now. It seems that I missed this notification.

I see your point and would like for lerna-parallelism to support your use-case as well, but I think your proposed solution has unintended side-effects.

the partitions are calculated on all packages, then applied to the packagesWithCommand.

This would cause uneven / empty partitions and — in the worst case — could result in all scripts being run in the same partition, while all other partitions are empty.

As a quick fix, we could add the behavior you suggested as a CLI flag users can opt-in to. Do you have a good name suggestion? I have these ideas, but am not satisfied with them: --stable-split, --global-split, --unbalanced

Ideally, we could intercept this filter step in RunCommand#initialize() to allow more sophisticated filtering. The only possible way I currently see is declaring a setter trap for packagesWithScript and abusing script === 'env', so that we can do our own filtering, but this feels quite hacky. 😧

An alternative direction we could explore is adding parallelism support to lerna list or providing an enhanced, parallelism-aware filter & list command. This way we could first generate the list of packages and then pass it to e.g. run as --scope.