fill the lookahead buffer on short reads from a chunked source#625
Open
saleemno1 wants to merge 1 commit into
Open
fill the lookahead buffer on short reads from a chunked source#625saleemno1 wants to merge 1 commit into
saleemno1 wants to merge 1 commit into
Conversation
A Reader that delivers data in chunks makes the buffered read return early, so the delimiter lookahead was compared against a partially filled buffer and multi-character delimiters were missed.
garydgregory
requested changes
Jul 20, 2026
garydgregory
left a comment
garydgregory
left a comment
Member
There was a problem hiding this comment.
Hello @saleemno1
Please see my questions and comments.
| @Test | ||
| void testParseWithDelimiterStringFromChunkedReader() throws IOException { | ||
| // A reader that hands out one character at a time and never reports itself ready, like a socket or pipe. | ||
| final Reader chunked = new Reader() { |
Member
There was a problem hiding this comment.
If this is the same as the behavior of the ChunkedReader in the other test, then refactor for reuse.
| /** | ||
| * A reader that returns one character per call and never reports itself ready, like a socket or pipe that delivers data in chunks. | ||
| */ | ||
| private static final class ChunkedReader extends java.io.Reader { |
Member
There was a problem hiding this comment.
Would it be simpler to extend StringReader and override those methods?
| final int len = super.read(buf, offset, length); | ||
| int len = super.read(buf, offset, length); | ||
| // The underlying buffered reader stops early once the source reports it is not ready, so a stream that delivers data in chunks (a socket or a pipe) | ||
| // yields a short read. Callers match multi-character sequences against this buffer, so keep reading until it is full or the source is exhausted. |
Member
There was a problem hiding this comment.
Isn't there still a risk that even filling to the end of the buffer breaks up multi-character sequences?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for maintaining this!
ExtendedBufferedReader.peek(char[])andread(char[], int, int)inherit a buffered read that returns as soon as the source reports it is not ready, so aReaderbacked by a socket or a pipe hands back a short read and the delimiter lookahead is matched against a half-filled buffer.Two things go wrong once the buffer is short:
Lexer.isDelimiter/isEscapeDelimitercompare every slot, so the still-zeroed tail makes a multi-character delimiter that is really present look like ordinary text. Parsinga[|]bwith delimiter[|]yields one fielda[|]bfrom a chunked reader and two fieldsa,bfrom aStringReaderfor the same bytes, which is a parser differential if the split feeds any filtering or routing decision.CSVFormat.printWithEscapes(Reader, Appendable)builds its lookahead the same way, so printing the single valuex[|]yfrom a chunked reader writes the delimiter unescaped and the record reads back as two fields.Both call sites share the same reader, so the loop lives there rather than in each caller. Added a regression test at the reader level and one at the parser level; both fail without the change.
mvnis green including the coverage gate.