tests: reset freed pointers in mem-fail TLS tests to avoid double-free

[sameehj]

Description

This is a test-only change. It does not change library code or run-time behavior.

The nightly mem-fail job builds with WOLFSSL_MEM_FAIL_COUNT and makes each memory allocation fail in turn. When an allocation fails, an Expect* assertion fails and latches the test result. After this point, the test framework does not evaluate any more Expect* statements. It therefore skips the pointer assignment inside a later ExpectNotNull(ptr = wolfSSL_new(...)).

Several unit tests run more than one setup/free block against the same ssl and ctx variables. If the framework skips the re-assignment in a later block, the variable still holds the pointer that the previous block freed. The unconditional cleanup at the end of the block then frees that pointer a second time. The result is a double-free or use-after-free. Because the failure depends on heap state, it appears as an intermittent SIGSEGV (139) or abort (134), not a stable failure.

The fix sets each pointer to NULL immediately after the test frees it. A later skipped assignment then leaves NULL, and wolfSSL_free(NULL)/wolfSSL_CTX_free(NULL) do nothing. Three tests also hold objects that the WOLFSSL owns (an ECH struct, pre-shared keys, and TLSX extension pointers) and dereference them in a later block through a bare if (ptr != NULL) guard. These handles are cleared for the same reason. In test_tls_msgtype_sni_parse_ech_public, a failed TLSX_Push() does not transfer ownership of ech (TLSX_New() returns NULL without storing data), so the test now frees ech on that path instead of leaking it.

The change applies the same pattern that test_ProcessChainOCSPRequest_bounds already uses in this file. Two multi-block tests (test_wolfSSL_SetTlsHmacInner_bounds and test_ProcessChainOCSPRequest_bounds) already reset their pointers and are left unchanged.

Affected tests:

• test_tls_msgtype.c: certificate_ext_offered, sni_parse_match, sni_parse_ech_public, psk_ch_id_gates, psk_sh_index, psk_sh_resumption, psk_ch_binder_gates, cookie_parse_gates, tca_parse_gates, tca_find, tca_new_alloc, psk_write_chosen.
• test_tls_bounds.c: test_TLSX_PopulateExtensions_bounds, test_TLSX_ext_msgtype_dispatch_bounds, test_TLSX_CSR_Parse_bounds, test_TLSX_ALPN_GetSize_overflow, test_TLSX_PopulateSupportedGroups_bounds, test_TLSX_SecureRenegotiation_Write_bounds.

Note: the crashes are not deterministic, so a single scan can miss one. For this reason the change hardens all multi-block tests in these two files, not only the tests that crashed in one run.

Fixes zd#

Testing

Build:

./configure --disable-shared --enable-static --enable-all \
  --enable-secure-renegotiation --enable-debug \
  C_EXTRA_FLAGS=-DWOLFSSL_MEM_FAIL_COUNT
make

Method: for each affected test, read its baseline allocation count, then run the test once per injection point (MEM_FAIL_CNT=1..MAX). Repeat each scan several times, because the failure is not deterministic. Record the exit code and compare MemFailCount Total against MemFailCount Frees.

Results:

• Before: the listed tests produce an intermittent SIGSEGV (139) or abort (134) under injection. A single scan first missed test_TLSX_CSR_Parse_bounds; a repeated scan then showed it crashes at MEM_FAIL_CNT=137.
• After: 0 crashes across all listed tests, over the full injection range, at 4 to 10 repetitions per point.

Dependency: the same nightly job also reports two Free/Alloc mismatches (test_tls_msgtype_tca_new_alloc at 22 and test_ProcessChainOCSPRequest_bounds at 137). These are memory-count and leak problems, not crashes, and PR #11397 fixes them. The nightly job is fully clean only when both PR #11397 and this PR are present.

Checklist

• added tests — not applicable; this change fixes existing tests and adds no new test.
• updated/added doxygen — not applicable; no API change.
• updated appropriate READMEs — not applicable.
• Updated manual and documentation — not applicable.

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-openssl-compat
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11398

Scan targets checked: wolfssl-bugs, wolfssl-src

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

[sameehj]

retest this please