Skip to content

Run the test suite against local servers only - #551

Merged
ericmj merged 1 commit into
mainfrom
local-test-servers
Nov 17, 2026
Merged

ericmj merged 1 commit into
mainfrom
local-test-servers

Conversation

@ericmj

@ericmj ericmj commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

16 tests connected to badssl.com, twitter.com, facebook.com, robynthinks.wordpress.com, www.shopify.com, letsencrypt.org, nghttp2.org and rabbitmq.com. Every PR's CI hits them, and it fails whenever one of those hosts is slow or changes its setup, unrelated to the change. For example, the OTP 28.1 and 29.0 jobs of https://github.com/elixir-mint/mint/actions/runs/44065266797 on #549 failed on three different badssl.com tests (unexpected TLS alerts, a :timeout, and later a :closed), and passed on rerun. The suite also couldn't run offline.

The certificate verification, TLS version and connect timeout tests now use a local :ssl server (Mint.TLSTestServer) with certificates generated by :public_key.pkix_test_data/1. Mint.TestCertificates.pkix_test_chain/1 takes a :hostname option for the wrong-hostname case. A local connect can complete before a timeout of 0 fires, so the plain TCP connect timeout test fills a listen socket's accept queue first. Linux then drops the SYN and the connect can't complete; other systems reset the connection instead, so the test is skipped there.

letsencrypt.org no longer serves the DST Root CA X3 chain that the partial chain test depended on; it now serves a chain ending in ISRG Root X2 cross-signed by ISRG Root X1. The replacement builds a chain with an intermediate signed by an expired root and trusts both that root and a second root with the intermediate's key. Mint connects, and the same connection with Mint's partial_chain replaced fails with certificate_expired, so the test exercises Mint's code. dst_and_isrg.pem is deleted.

The :public_key.cacerts_get/0 test loads a generated CA with :public_key.cacerts_load/1 and restores the OS store in on_exit. That store is global to the VM, so the test lives in its own async: false module.

The HTTP/2 tests against third-party sites are removed. The #349 informational response case is covered by the "interim responses (1xx)" tests in conn_test.exs, and #171 has a new test checking that request header names are lowercased before HPACK encoding; it fails if Headers.lower_raws/1 is removed from HTTP2.request/5. After this, no test checks Mint against live third-party HTTP/2 servers.

The :requires_internet_connection tag and the README instructions for excluding it are removed. The ALPN tests against the local Caddy, which were grouped under describe "nghttp2.org", are now under describe "protocol selection". The HTTP/2 "certificate verification" TODO is removed, because Mint.Core.Transport.SSL verifies certificates for both protocols and the local verification tests in Mint.IntegrationTest cover it.

@coveralls

coveralls commented Nov 17, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 263326

Coverage decreased (-0.2%) to 90.343%

Details

  • Coverage decreased (-0.2%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • 6 coverage regressions across 4 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

6 previously-covered lines in 4 files lost coverage.

File Lines Losing Coverage Coverage
lib/mint/http2.ex 3 95.22%
lib/mint/core/util.ex 1 89.29%
lib/mint/http2/frame.ex 1 99.09%
lib/mint/negotiate.ex 1 81.82%

Coverage Stats

Coverage Status
Relevant Lines: 2040
Covered Lines: 1843
Line Coverage: 90.34%
Coverage Strength: 725.81 hits per line

💛 - Coveralls

@ericmj
ericmj marked this pull request as ready for review November 17, 2026 14:46
@ericmj
ericmj force-pushed the local-test-servers branch from 1c777fe to f847bdf Compare November 17, 2026 14:52
16 tests connected to badssl.com, twitter.com, facebook.com,
robynthinks.wordpress.com, www.shopify.com, letsencrypt.org,
nghttp2.org and rabbitmq.com. CI failed whenever one of them was slow
or changed its configuration, unrelated to the change under test, and
the suite couldn't run offline.

The certificate verification, TLS version and connect timeout tests
now use a local :ssl server with certificates generated by
:public_key.pkix_test_data/1. The partial chain test builds a chain
with an intermediate signed by an expired root, which Mint's
partial_chain function accepts and :ssl alone rejects with
certificate_expired; letsencrypt.org no longer serves the DST Root CA
X3 chain the old test depended on. A local connect can complete before
a timeout of 0 fires, so the plain TCP connect timeout test fills a
listen socket's accept queue first; Linux then drops the SYN, and the
test is skipped on other systems, which reset the connection instead.
The :public_key.cacerts_get/0 test
loads a generated CA with :public_key.cacerts_load/1 in a synchronous
module and restores the OS store afterwards.

The HTTP/2 tests against third-party sites are removed. The #349
informational response case is covered by the interim response tests
in conn_test.exs, and #171 now has a test that request header names
are lowercased before HPACK encoding. Without these tests nothing
checks Mint against live third-party HTTP/2 servers.

The :requires_internet_connection tag is gone along with the README
instructions for excluding it. The ALPN tests against the local Caddy
were grouped under "nghttp2.org" and are now under "protocol
selection", and the HTTP/2 certificate verification TODO is removed
because Mint.Core.Transport.SSL verifies certificates for both
protocols and the local verification tests cover it.
@ericmj
ericmj force-pushed the local-test-servers branch from f847bdf to 263326a Compare November 17, 2026 15:00
@ericmj
ericmj merged commit bd0b65b into main Nov 17, 2026
3 checks passed
@ericmj
ericmj deleted the local-test-servers branch November 17, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants