[5.3] Fix DB2Decoder NPE when chained DSS response is split across TCP segments - #1716
Merged
Merged
Conversation
…ents Fixes eclipse-vertx#899 A DRDA response from DB2 can consist of multiple chained DSS (Data Stream Structure) segments, linked via a chain bit (0x40) in byte 3 of each segment's header. The computeLength() method is supposed to calculate the total length of all chained segments before the decoder processes them as a single logical response. Bug: When TCP delivers only the first segment(s) of a chained response (with the chain bit set) but the next chained segment has not arrived yet, computeLength() returns the partial length (equal to readableBytes). Since decode() only waits when payloadLength > readableBytes, it proceeds to call decodePayload() with incomplete data. The command handler processes the partial response and either: 1. Calls ensureALayerDataInBuffer() to read the next DSS header but the buffer is empty: "Needed to have 6 in buffer but only had 0" 2. Calls fireCommandSuccess()/fireCommandFailure() which polls the command from the inflight queue. When the remaining chained segments arrive, inflight.peek() returns the WRONG command: "Invalid correlator ID. Got 2 expected 1" 3. Or inflight.peek() returns null: NullPointerException in decodePayload() 4. Leftover DSS data from command A is fed to command B's handler, which encounters unexpected DRDA codepoints: "Found unknown codepoint: 0x2411" The crash sequence: 1. TCP delivers DSS_1 (chain bit set), DSS_2 not yet received 2. computeLength() returns len(DSS_1) == readableBytes 3. decode() proceeds (payloadLength <= readableBytes) 4. Command processes partial data, polls itself from inflight 5. DSS_2 arrives -> dispatched to wrong command or null -> error This is intermittent because TCP segmentation is non-deterministic - whether the OS delivers all DSS segments in one read() depends on timing, buffer sizes, and network conditions. Fix: After the while loop, check if dssContinues is still true (meaning the chain bit was set on the last DSS we read but the next segment hasn't arrived). If so, return readableBytes + 1 to force decode() to wait for more data. Also fix a secondary bug: use getUnsignedShort() instead of getShort() for reading DSS lengths. The DRDA spec defines DSS lengths as unsigned 16-bit values (0-65535), but getShort() returns a signed Java short (-32768 to 32767), which would produce negative values for segments larger than 32767 bytes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #899
A DRDA response from DB2 can consist of multiple chained DSS (Data Stream Structure) segments, linked via a chain bit (0x40) in byte 3 of each segment's header. The computeLength() method is supposed to calculate the total length of all chained segments before the decoder processes them as a single logical response.
Bug: When TCP delivers only the first segment(s) of a chained response (with the chain bit set) but the next chained segment has not arrived yet, computeLength() returns the partial length (equal to readableBytes). Since decode() only waits when payloadLength > readableBytes, it proceeds to call decodePayload() with incomplete data. The command handler processes the partial response and either:
Calls ensureALayerDataInBuffer() to read the next DSS header but the buffer is empty: "Needed to have 6 in buffer but only had 0"
Calls fireCommandSuccess()/fireCommandFailure() which polls the command from the inflight queue. When the remaining chained segments arrive, inflight.peek() returns the WRONG command: "Invalid correlator ID. Got 2 expected 1"
Or inflight.peek() returns null: NullPointerException in decodePayload()
Leftover DSS data from command A is fed to command B's handler, which encounters unexpected DRDA codepoints: "Found unknown codepoint: 0x2411"
The crash sequence:
This is intermittent because TCP segmentation is non-deterministic - whether the OS delivers all DSS segments in one read() depends on timing, buffer sizes, and network conditions.
Fix: After the while loop, check if dssContinues is still true (meaning the chain bit was set on the last DSS we read but the next segment hasn't arrived). If so, return readableBytes + 1 to force decode() to wait for more data.
Also fix a secondary bug: use getUnsignedShort() instead of getShort() for reading DSS lengths. The DRDA spec defines DSS lengths as unsigned 16-bit values (0-65535), but getShort() returns a signed Java short (-32768 to 32767), which would produce negative values for segments larger than 32767 bytes.
Motivation:
Explain here the context, and why you're making that change, what is the problem you're trying to solve.
Conformance:
You should have signed the Eclipse Contributor Agreement as explained in https://github.com/eclipse/vert.x/blob/master/CONTRIBUTING.md
Please also make sure you adhere to the code style guidelines: https://github.com/vert-x3/wiki/wiki/Vert.x-code-style-guidelines