Skip to content

fix(deployer): refuse a hot upgrade built on a different toolchain - #271

Closed
thiagoesteves wants to merge 1 commit into
mainfrom
thiagoesteves/reject-toolchain-mismatch
Closed

fix(deployer): refuse a hot upgrade built on a different toolchain#271
thiagoesteves wants to merge 1 commit into
mainfrom
thiagoesteves/reject-toolchain-mismatch

Conversation

@thiagoesteves

Copy link
Copy Markdown
Owner

The failure this prevents

Applying the OTP-28 artifact to an installation running the OTP-27 build:

Unpacked successfully: "0.9.10"
Could not open file /opt/deployex/lib/elixir-1.19.5/ebin/elixir.appup
systools:make_relup failed, reason: :error

Nothing there says the wrong file was chosen. It surfaces two steps after the mistake, as a missing file inside systools.

Why it cannot work

A hot upgrade replaces application code inside the running VM. It cannot replace the VM or the language runtime beneath it. systools:make_relup/4 demands an .appup for every application whose version changes between the two releases, and neither Elixir nor the OTP applications ship one - so any toolchain difference is fatal by construction, not by accident.

For DeployEx that maps directly onto otp_version:

devops/releases/otp-27/.tool-versions   elixir 1.18.4-otp-27
devops/releases/otp-28/.tool-versions   elixir 1.19.5-otp-28

The check

check/1 already extracts the release before validating it, so both versions are sitting on disk. It now compares them against the running system:

running = %{elixir: System.version(), erts: to_string(:erlang.system_info(:version))}
release = %{elixir: version_from_dir(new_path, "lib/elixir-*", "elixir-"),
            erts:   version_from_dir(new_path, "erts-*", "erts-")}

and refuses when either differs, naming both sides:

Hot upgrade refused, the release was built on a different toolchain:
%{elixir: %{running: "1.18.4", release: "1.19.5"}}. A hot upgrade cannot change the
Erlang or Elixir version under a running system. Use the artifact matching the
otp_version this installation runs, or apply it as a full deployment.

The UI shows wrong artifact for this installation: elixir 1.18.4 -> 1.19.5 instead of invalid release.

Only the versions that actually differ are reported - an ERTS-only change reports ERTS alone.

Deliberately permissive in one place

A release whose layout cannot be read (no lib/elixir-*, no erts-*) falls through rather than being rejected. The checks that follow report a malformed release far better than a guess here would, and refusing on a failed Path.wildcard would turn an unrelated packaging problem into a confusing toolchain error.

Risk assessment

Impact: choosing the wrong artifact is rejected at the point of choosing it, naming the versions. An artifact built on the same toolchain behaves exactly as before.

Blast radius: the DeployEx self-upgrade check and the error branch that displays it. The monitored-application path is untouched - it has its own check.

Regression risk: low. The new step only rejects an upgrade that would have failed at make_relup anyway, and an unreadable layout falls through rather than blocking.

Rollback: plain commit revert.

Checks

Full umbrella suite green (foundation 236 + 25 doctests, host 21, deployer 174, sentinel 84, deployex_web 177 + 6 doctests), mix credo --strict clean, mix format --check-formatted clean, compiles with --warnings-as-errors.

Two new tests: one builds a release layout with a different Elixir and the same ERTS - exactly the shape of the real failure - and asserts only elixir is reported; the other builds a matching layout and asserts it passes through.

Not addressed here

The other half of that incident: Deployex.execute/2 logged Hot upgrade in deployex installed with success three seconds before the unpack started, because with sync_execution: false it logs on the cast returning :ok rather than on the upgrade completing. A failed self-upgrade still reports success. Worth its own PR.

🤖 Generated with Claude Code

A hot upgrade replaces application code inside the running VM, it cannot replace the VM
or the language runtime under it. systools:make_relup/4 needs an appup for every
application whose version changes, and neither Elixir nor the OTP applications ship one.

Applying the OTP 28 artifact to an installation running the OTP 27 build therefore fails
partway through, after the release has been unpacked, with

  Could not open file /opt/deployex/lib/elixir-1.19.5/ebin/elixir.appup
  systools:make_relup failed, reason: :error

which says nothing about the actual mistake, that the wrong file was chosen.

check/1 already extracts the release before validating it, so both versions are on disk
and answerable before anything is installed. It now compares the Elixir and ERTS versions
in the release against the running system and refuses when either differs, naming both
sides. The UI reports which versions changed rather than "invalid release".

A release whose layout cannot be read is left to the checks that follow, which report it
better than a guess would.

Risk assessment

Impact: choosing the wrong artifact is rejected at the point of choosing it, with a
message naming the versions. An artifact built on the same toolchain behaves as before.

Blast radius: the deployex self upgrade check and the error branch that displays it. The
monitored application path is untouched, it has its own check.

Regression risk: low. The new step only rejects an upgrade that would have failed at
make_relup anyway, and an unreadable layout falls through rather than blocking.

Rollback: plain commit revert.
@thiagoesteves

Copy link
Copy Markdown
Owner Author

Superseded: the check was specific to Elixir and ERTS. Replaced by a single OTP comparison against the erts the release bundles.

@thiagoesteves
thiagoesteves deleted the thiagoesteves/reject-toolchain-mismatch branch August 11, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant