Skip to content

Fix memory leaks in signature and key handling - #534

Merged
mtrojnar merged 2 commits into
masterfrom
fix-memory-leaks
Nov 16, 2026
Merged

mtrojnar merged 2 commits into
masterfrom
fix-memory-leaks

Conversation

@mtrojnar

@mtrojnar mtrojnar commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Pull Request Type

  • Bug fix

Related Issue

Issue number: N/A

Current Behavior

Error paths, repeated values and failed OpenSSL ownership transfers leave signatures, credentials and buffers allocated. PE and script processing can leave input mappings partially allocated. TSA parameter-loading failures incorrectly report success. Base64 BIO allocation failures can also leak a timestamp response BIO or crash script extraction and timestamp decoding, and signature content creation can leak the PKCS#7 structure or crash when an allocation fails.

New Behavior

Release owned objects on failure and before replacement, preserve MSI buffers when growth fails, and unmap the complete original input. Invalid or unreadable TSA parameters now fail the command and remove its output. Base64 BIO allocation failures abort decoding cleanly, and signature content creation reports allocation failures without leaking or crashing.

Scope of Changes

Cover signature processing, digest BIOs, credential/provider loading, engine references and repeated options. Catalog member, duplicate APPX hash and private key selection remain unchanged. The libcurl header fix applies only to legacy OpenSSL < 3.0 builds. A separate comments-only commit documents ownership transfers, cleanup responsibility and mapping-length invariants.

Testing

  • Existing tests
  • New tests added
  • Manual testing

All 424 existing CTest tests passed on Linux with both OpenSSL 3.6 and 1.1.1. The modern build's 407 Valgrind process checks and 192-run fault-injection matrix (132 triggered failures) reported zero errors. Separate legacy libcurl checks covered both timestamp protocols and failure of each header append, with no definite or indirect leaks. Another 21 focused script/timestamp decoding checks passed under Valgrind, including filter and memory-BIO allocation failures. A further 25 fault-injection checks covered signature content creation across PE, MSI, CAB, APPX and script formats, with no leaks or crashes. Focused checks covered failed writes, malformed signatures, engine login, MSI reallocations and complete input unmapping. Repeated APPX hash extraction and allocation failures passed Valgrind checks for all five hash types. GitHub CI, including the CodeQL security gate and Linux/macOS/Windows builds, passed.

Additional Notes

Validation used temporary local reproductions; no test files are included.

License Declaration

  • I hereby agree to license my contribution under the project's license.

@mtrojnar
mtrojnar force-pushed the fix-memory-leaks branch 7 times, most recently from 45a016a to d023046 Compare November 15, 2026 17:18
Release signatures, timestamp credentials and APPX buffers on error, and
free unused catalog members. Repeated hashes, private keys and command
options must release the values they replace. Finish initialized engines
when forced login fails.

Handle failed ownership transfers to OpenSSL attributes and stacks,
preserve MSI buffers after failed growth, and unmap the complete input
when PE or script processing shortens its content length. Check digest
BIO allocation before linking sink BIOs.

Route built-in TSA loading failures through common cleanup. Preserve the
failure result so unreadable or invalid TSA parameters no longer leave
an apparently successful output without the requested timestamp.
Make success-only ownership transfers and always-consuming helper
contracts explicit at call boundaries. Explain signature cleanup,
context-owned APPX buffers and immutable mapping sizes so future changes
do not reintroduce leaks or double frees.

Comments only; no behavior changes.

@olszomal olszomal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@mtrojnar
mtrojnar merged commit 8cdb811 into master Nov 16, 2026
11 checks passed
@mtrojnar
mtrojnar deleted the fix-memory-leaks branch November 16, 2026 14:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants