Skip to content

KAFKA-10561: Preserve sub-millisecond precision in TimestampConverter - #23432

Open
michal-swiatowy wants to merge 1 commit into
apache:trunkfrom
michal-swiatowy:timestamp-converter-sub-ms-precision
Open

michal-swiatowy wants to merge 1 commit into
apache:trunkfrom
michal-swiatowy:timestamp-converter-sub-ms-precision

Conversation

@michal-swiatowy

@michal-swiatowy michal-swiatowy commented Sep 11, 2026

Copy link
Copy Markdown

NOTE: I used https://issues.apache.org/jira/browse/KAFKA-10561 for my PR but not sure how much it matches, I'm happy to change or create a new issue if required. Maybe https://issues.apache.org/jira/browse/KAFKA-10627?

Main idea
Replace java.util.Date (millisecond precision) with java.time.Instant (nanosecond precision) as the internal representation in TimestampConverter. This fixes loss of microsecond and nanosecond data when converting between unix and string timestamp formats.

To do this a new format.type configuration option is added that selects the string parsing/formatting engine:

  • SimpleDateFormat (default) - for backward compatibility,
  • DateTimeFormatter - uses java.time.format.DateTimeFormatter

Key invariants

  • Default behavior is unchanged - SimpleDateFormat remains the default format.type
  • Conversions involving Date, Time, or Timestamp Connect logical types still go through java.util.Date at the boundary and remain limited to millisecond precision (this is a Connect API constraint).
  • Using only JDK java.time classes.

Testing

  • Added new tests for data format support and various convert cases
  • Full connect:transforms test suite ran ok

Sample usage

transforms.ts.type=org.apache.kafka.connect.transforms.TimestampConverter$Value
transforms.ts.target.type=unix
transforms.ts.unix.precision=microseconds
transforms.ts.format=yyyy-MM-dd'T'HH:mm:ss.SSSSSS'Z'
transforms.ts.format.type=DateTimeFormatter

Replace java.util.Date (millisecond precision) with java.time.Instant
(nanosecond precision) as the internal intermediate representation in
TimestampConverter.
@github-actions github-actions Bot added triage PRs from the community connect labels Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connect triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant