Skip to content

[5.1] Fix DB2Decoder NPE when chained DSS response is split across TCP segments - #1715

Merged
tsegismont merged 1 commit into
eclipse-vertx:5.1from
DavideD:899-dss-split
Oct 2, 2026
Merged

tsegismont merged 1 commit into
eclipse-vertx:5.1from
DavideD:899-dss-split

Conversation

@DavideD

@DavideD DavideD commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #899

I asked Claude to solves #899 and this is what happened.
My knowledge in this area is null but I tested the solution with the Hibernate Reactive build and it seems to work now

A DRDA response from DB2 can consist of multiple chained DSS (Data Stream Structure) segments, linked via a chain bit (0x40) in byte 3 of each segment's header. The computeLength() method is supposed to calculate the total length of all chained segments before the decoder processes them as a single logical response.

Bug: When TCP delivers only the first segment(s) of a chained response (with the chain bit set) but the next chained segment has not arrived yet, computeLength() returns the partial length (equal to readableBytes). Since decode() only waits when payloadLength > readableBytes, it proceeds to call decodePayload() with incomplete data. The command handler processes the partial response and either:

  1. Calls ensureALayerDataInBuffer() to read the next DSS header but the buffer is empty: "Needed to have 6 in buffer but only had 0"

  2. Calls fireCommandSuccess()/fireCommandFailure() which polls the command from the inflight queue. When the remaining chained segments arrive, inflight.peek() returns the WRONG command: "Invalid correlator ID. Got 2 expected 1"

  3. Or inflight.peek() returns null: NullPointerException in decodePayload()

  4. Leftover DSS data from command A is fed to command B's handler, which encounters unexpected DRDA codepoints: "Found unknown codepoint: 0x2411"

The crash sequence:

  1. TCP delivers DSS_1 (chain bit set), DSS_2 not yet received
  2. computeLength() returns len(DSS_1) == readableBytes
  3. decode() proceeds (payloadLength <= readableBytes)
  4. Command processes partial data, polls itself from inflight
  5. DSS_2 arrives -> dispatched to wrong command or null -> error

This is intermittent because TCP segmentation is non-deterministic - whether the OS delivers all DSS segments in one read() depends on timing, buffer sizes, and network conditions.

Fix: After the while loop, check if dssContinues is still true (meaning the chain bit was set on the last DSS we read but the next segment hasn't arrived). If so, return readableBytes + 1 to force decode() to wait for more data.

Also fix a secondary bug: use getUnsignedShort() instead of getShort() for reading DSS lengths. The DRDA spec defines DSS lengths as unsigned 16-bit values (0-65535), but getShort() returns a signed Java short (-32768 to 32767), which would produce negative values for segments larger than 32767 bytes.

Motivation:

Explain here the context, and why you're making that change, what is the problem you're trying to solve.

Conformance:

You should have signed the Eclipse Contributor Agreement as explained in https://github.com/eclipse/vert.x/blob/master/CONTRIBUTING.md
Please also make sure you adhere to the code style guidelines: https://github.com/vert-x3/wiki/wiki/Vert.x-code-style-guidelines

…ents

Fixes eclipse-vertx#899

A DRDA response from DB2 can consist of multiple chained DSS (Data
Stream Structure) segments, linked via a chain bit (0x40) in byte 3 of
each segment's header. The computeLength() method is supposed to
calculate the total length of all chained segments before the decoder
processes them as a single logical response.

Bug: When TCP delivers only the first segment(s) of a chained response
(with the chain bit set) but the next chained segment has not arrived
yet, computeLength() returns the partial length (equal to
readableBytes). Since decode() only waits when payloadLength >
readableBytes, it proceeds to call decodePayload() with incomplete
data. The command handler processes the partial response and either:

  1. Calls ensureALayerDataInBuffer() to read the next DSS header but
     the buffer is empty:
     "Needed to have 6 in buffer but only had 0"

  2. Calls fireCommandSuccess()/fireCommandFailure() which polls the
     command from the inflight queue. When the remaining chained
     segments arrive, inflight.peek() returns the WRONG command:
     "Invalid correlator ID. Got 2 expected 1"

  3. Or inflight.peek() returns null:
     NullPointerException in decodePayload()

  4. Leftover DSS data from command A is fed to command B's handler,
     which encounters unexpected DRDA codepoints:
     "Found unknown codepoint: 0x2411"

The crash sequence:
1. TCP delivers DSS_1 (chain bit set), DSS_2 not yet received
2. computeLength() returns len(DSS_1) == readableBytes
3. decode() proceeds (payloadLength <= readableBytes)
4. Command processes partial data, polls itself from inflight
5. DSS_2 arrives -> dispatched to wrong command or null -> error

This is intermittent because TCP segmentation is non-deterministic -
whether the OS delivers all DSS segments in one read() depends on
timing, buffer sizes, and network conditions.

Fix: After the while loop, check if dssContinues is still true (meaning
the chain bit was set on the last DSS we read but the next segment
hasn't arrived). If so, return readableBytes + 1 to force decode() to
wait for more data.

Also fix a secondary bug: use getUnsignedShort() instead of getShort()
for reading DSS lengths. The DRDA spec defines DSS lengths as unsigned
16-bit values (0-65535), but getShort() returns a signed Java short
(-32768 to 32767), which would produce negative values for segments
larger than 32767 bytes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DavideD

DavideD commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

I don't know why, but I cannot login to the Eclipse foundation website

@DavideD

DavideD commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

I don't know why, but I cannot login to the Eclipse foundation website

It seems the verification of the License is working now. @vietj or @tsegismont, can you have a look at this and let me know if it makes sense, it would be nice close this issue.
Thanks

@tsegismont tsegismont left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @DavideD

@tsegismont
tsegismont merged commit 9fb735f into eclipse-vertx:5.1 Oct 2, 2026
18 checks passed
@tsegismont tsegismont modified the milestones: 5.1.10, 5.1.9 Oct 2, 2026
@tsegismont

Copy link
Copy Markdown
Member

@DavideD could you please create corresponding PRs for main, 5.2 and 4.x branches? Thanks

@DavideD DavideD changed the title Fix DB2Decoder NPE when chained DSS response is split across TCP segments [5.1] Fix DB2Decoder NPE when chained DSS response is split across TCP segments Oct 2, 2026
@DavideD

DavideD commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Awesome, done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants