Skip to content

pyln: relax the grpcio/protobuf pins and regenerate the protos - #9490

Open
cdecker wants to merge 2 commits into
masterfrom
2026m09-dependency-min
Open

pyln: relax the grpcio/protobuf pins and regenerate the protos#9490
cdecker wants to merge 2 commits into
masterfrom
2026m09-dependency-min

Conversation

@cdecker

@cdecker cdecker commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

grpcio, grpcio-tools and protobuf were hard-pinned to exact versions, which kept the
tree off every newer release. Relax them to floors and regenerate the checked-in stubs with
the resulting toolchain.

Why the floors move up

This looks backwards for a change whose point is to relax pins, so it is worth spelling out.
Generated protobuf code asserts runtime >= gencode at import time, and the grpc stubs
refuse to load below the grpcio-tools that produced them. Regenerating therefore raises the
minimum, and the manifests have to say so or a fresh install resolves an older runtime and
dies on import rather than at some later, stranger point:

  • protobuf >= 6.33.5 (was == 6.32.1)
  • grpcio >= 1.81.1, grpcio-tools >= 1.81.1 (were == 1.75.1)

Comments next to each bound record that they must be bumped together with a regeneration.

The comment claiming the protobuf pin matched .github/scripts/setup.sh was stale; that file
carries no protobuf version.

Makefile fix

contrib/pyln-grpc-proto/Makefile invoked a bare python for the primitives.proto step,
which is not on PATH in the uv-managed environment, so make protos died with
make: python: No such file or directory after regenerating node.proto. The rule declares
primitives_pb2.pyi and node_pb2.pyi as outputs, but because the recipe aborted there they
were never produced and so were never checked in — even though the package ships py.typed.
Both stub files are now generated and committed.

Testing

cargo check --all is clean. pyln.grpc imports and round-trips a message against the
regenerated stubs on grpcio 1.83.1 / protobuf 6.33.6.

Note

cargo-udeps reports no unused dependencies anywhere in the workspace.

Separately, and not addressed here: the tree carries a duplicate HTTP stack
(reqwest 0.11 pulling hyper 0.14, h2 0.3, http 0.2, rustls 0.21) entirely from
plugins/bip353-plugin. The obvious fix — bitcoin-payment-instructions 0.7, which drops
reqwest for the much smaller bitreq — is not viable: bitreq's Proxy supports only
HTTP CONNECT, with no SOCKS support and no feature to add it. The plugin uses socks5h://
for Tor, where the h also prevents DNS leaks, so taking that upgrade would silently break
Tor for BIP-353 lookups. Worth revisiting if bitreq gains SOCKS.

cdecker and others added 2 commits September 8, 2026 17:10
The primitives.proto invocation called a bare `python`, which is not on
PATH in the uv-managed environment, so `make protos` died with
`make: python: No such file or directory` after regenerating node.proto.
The rule declares primitives_pb2.pyi and node_pb2.pyi as outputs, but
because the recipe aborted here they were never produced and so were
never checked in, even though the package ships py.typed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZoAzNVajT8Udj7XYrurTa
grpcio, grpcio-tools and protobuf were hard-pinned to exact versions,
which kept the tree off every newer release and made it the tighter side
when compared against Greenlight. Relax them to floors and regenerate the
stubs with the resulting toolchain (grpcio-tools 1.81.1).

The floors have to move *up* rather than down: generated protobuf code
asserts `runtime >= gencode` at import time, and the grpc stubs refuse to
load below the grpcio-tools that produced them. So the new gencode pins
the floors at protobuf 6.33.5 and grpcio 1.81.1; installing anything older
now fails at import rather than at some later, stranger point. Comments
next to each bound record that they must be bumped together with a
regeneration.

The comment claiming the protobuf pin matched .github/scripts/setup.sh
was stale; that file carries no protobuf version.

Also check in node_pb2.pyi and primitives_pb2.pyi, which the Makefile has
always declared as outputs and which the package's py.typed advertises,
but which were never generated before the preceding fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YZoAzNVajT8Udj7XYrurTa
@daywalker90

Copy link
Copy Markdown
Collaborator

We use protoc-bin-vendored in cln-grpc but that is pinned to 31.5 and does not let you pick another version. I searched again on crates.io and i found https://crates.io/crates/protoc-fetcher which does let you pick a version. We should switch these in this PR and pick 33.5 like you did or i can open another PR for that if you want.

@daywalker90

Copy link
Copy Markdown
Collaborator

Neither uv run make nor uv run make gen generate those .pyi files, so they will be out of sync.

@daywalker90

Copy link
Copy Markdown
Collaborator

Ok uv run make check-gen-updated would fail the CI but still they should get generated during above commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants