Skip to content

[ruby-compat/08] Match ruby's timestamp-to-INTEGER/FLOAT/STRING output - #117

Merged
chikamura merged 6 commits into
feature/ruby-compatibilityfrom
ruby-compat/08-timestamp-conversion-match-ruby
Oct 8, 2026
Merged

chikamura merged 6 commits into
feature/ruby-compatibilityfrom
ruby-compat/08-timestamp-conversion-match-ruby

Conversation

@chikamura

Copy link
Copy Markdown
Contributor

Summary

  • Ruby's timestamp_converter outputs epoch seconds for INTEGER/FLOAT (val.to_i/val.to_f), and its DEFAULT_TIMESTAMP_FORMAT has no timezone offset. The java port previously output epoch milliseconds for INTEGER/FLOAT and appended a timezone offset (%:z) to the default STRING format.
  • Change INTEGER to output getEpochSecond() and FLOAT to output getEpochSecond() + getNano() / 1e9, matching ruby's to_i/to_f.
  • Change default_timestamp_format's @ConfigDefault from "%Y-%m-%d %H:%M:%S.%6N %:z" to "%Y-%m-%d %H:%M:%S.%6N", matching ruby's DEFAULT_TIMESTAMP_FORMAT (this also brings the code in line with README, which already documented the no-offset format).
  • Breaking change: existing configs that emit timestamp columns as INTEGER/FLOAT, or as STRING without an explicit timestamp_format, will see different output values/format.

Test plan

  • ./gradlew test (added testConvertTimestampToFloat_withFraction to cover the new sub-second FLOAT arithmetic)
  • ./gradlew spotlessApply

🤖 Generated with Claude Code

chikamura and others added 2 commits September 16, 2026 10:44
INTEGER/FLOAT now output epoch seconds instead of milliseconds, and
default_timestamp_format no longer appends a timezone offset,
matching ruby's timestamp_converter and DEFAULT_TIMESTAMP_FORMAT.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…mp-conversion-match-ruby

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@chikamura
chikamura marked this pull request as ready for review September 25, 2026 04:38
@chikamura chikamura changed the title [WIP][ruby-compat/08] Match ruby's timestamp-to-INTEGER/FLOAT/STRING output [ruby-compat/08] Match ruby's timestamp-to-INTEGER/FLOAT/STRING output Sep 25, 2026
@chikamura
chikamura requested a review from john-abre September 25, 2026 05:27

@john-abre john-abre left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ありがとうございます。変換ロジックの本体は ruby の value_converter_factory.rb(val.to_i / val.to_f / DEFAULT_TIMESTAMP_FORMAT = "%Y-%m-%d %H:%M:%S.%6N")と正確に一致しています。1970 年以前の INTEGER の floor 挙動や、現実的な値での FLOAT の精度も ruby と揃っていることを確認しました。破壊的変更が PR 本文と README に明記されている点も良いと思います。

マージ前に対応をお願いしたい点が 2 つあります。

  1. timestamp → STRING/DATE/DATETIME で default_timezone が依然として無視されており、デフォルトフォーマットから %:z を外したことでそれが見えなくなっています。BigqueryColumnOption.getTimezone() は "UTC" 固定のデフォルトで、PluginTask.getDefaultTimezone() は src/main のどこからも参照されていません。ruby は @timezone = timezone || default_timezone としています。default_timezone: Asia/Tokyo と column_options: [{name: ts, type: STRING}] の組み合わせで、ruby は 2020-05-01 09:00:00.000000 を出力しますが、このブランチは 2020-05-01 00:00:00.000000 を出力し、オフセットもないためズレに気づけません。既存の問題ではありますが、この PR のタイトルは STRING 出力が ruby と一致すると謳っています。この PR でフォールバックを実装する(column option を Optional<String> + @ConfigDefault("null") にして orElse(task.getDefaultTimezone()) する)か、PR 本文と #122 に「後回しにする」と明記するか、どちらかをお願いします。

  2. テストが新しい演算の挙動を固定できていません。 インラインコメントを参照してください。INTEGER のケースは秒単位ぴったりの値を使っているため、切り捨てか四捨五入か、1970 年以前の floor 挙動かが検証されておらず、FLOAT の許容誤差も必要以上に緩くなっています。

ブロッカーではない nit: INTEGER/FLOAT 列でミリ秒に依存していたユーザー向けの移行案内がありません。STRING のユーザーは default_timestamp_format: "%Y-%m-%d %H:%M:%S.%6N %:z" で元に戻せるので、README に一行あると親切です。INTEGER/FLOAT でミリ秒が必要なユーザーには戻す手段がなく、失敗も静かに起きます(下流の TIMESTAMP_MILLIS(col) が 1970 年の日付を返すなど)。

node.put(name, src.getEpochSecond() + src.getNano() / 1_000_000_000.0);
break;
case STRING:
String format = columnOption.getTimestampFormat().orElse(task.getDefaultTimestampFormat());

@john-abre john-abre Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

数行下で使われているタイムゾーン(timezone = columnOption.getTimezone())は UTC 固定のデフォルトで、ruby の timezone || default_timezone と違い task.getDefaultTimezone() へのフォールバックがありません。デフォルトフォーマットに %:z がなくなったため、default_timezone: Asia/Tokyo を設定しているユーザーは、ruby の出力から 9 時間ズレたタイムゾーンなしの文字列を静かに受け取ることになります。下の DATE / DATETIME 分岐も同様です。ここでフォールバックを実装するか、PR 本文で「後回し」と明記するかをお願いします。

break;
case FLOAT:
node.put(name, Double.valueOf(src.toEpochMilli()));
node.put(name, src.getEpochSecond() + src.getNano() / 1_000_000_000.0);

@john-abre john-abre Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: この加算は epoch 直近では正しく丸められません。Instant.ofEpochMilli(-1) だと -1 + 0.999 = -0.0010000000000000009 になり、ruby の to_f は -0.001 を返します。実際の日付では差は最大 1 ulp なのでこのままでも問題ありません。完全に一致させたい場合は BigDecimal.valueOf(src.getEpochSecond()).add(BigDecimal.valueOf(src.getNano(), 9)).doubleValue() で実現できます。

BigqueryTimestampConverter.convertAndSet(
node, "key", ts, BigqueryColumnOptionType.INTEGER, columnOption, task);
assertEquals(1588291200000L, node.get("key").asLong());
assertEquals(1588291200L, node.get("key").asLong());

@john-abre john-abre Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

この値は秒単位ぴったりなので、getEpochSecond() でも toEpochMilli() / 1000 でも Math.round(toEpochMilli() / 1000.0) でも通ってしまいます。ruby の to_i と同じ挙動を固定するケースを追加してください。

  • ofEpochMilli(1588291200999L) → 1588291200L(四捨五入ではなく切り捨て)
  • ofEpochMilli(-1L) → -1L(ゼロ方向の切り捨てではなく floor)

org.embulk.spi.time.Timestamp ts = org.embulk.spi.time.Timestamp.ofEpochMilli(1588291200123L);

BigqueryTimestampConverter.convertAndSet(
node, "key", ts, BigqueryColumnOptionType.FLOAT, columnOption, task);

@john-abre john-abre Sep 25, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1588291200L + 123000000 / 1e9 は double として正確に 1588291200.123 になるので、delta は 0.0(または 1e-9)にできます。1970 年以前のケース(ofEpochMilli(-500L) → -0.5)もここに追加しておくと良いと思います。

chikamura and others added 3 commits September 28, 2026 14:05
…havior

Cover the review feedback on the timestamp conversion change:

- INTEGER: sub-second values are truncated (999ms -> same second) and
  pre-1970 values are floored (-1ms -> -1), matching ruby's Time#to_i.
- FLOAT: the fraction case now asserts an exact double, plus pre-1970
  cases (-500ms -> -0.5, -1ms -> -0.001 as ruby's Time#to_f rounds once).
- default_timezone: timestamp -> STRING/DATE/DATETIME, string -> TIMESTAMP,
  and nested RECORD TIMESTAMP fields fall back to task default_timezone
  when the column/field has no timezone, as ruby's
  `timezone || default_timezone` does.

The -1ms FLOAT case and the default_timezone cases fail until the converter
is updated; the rest already pass and only guard the behavior.

TestBigqueryRecordConverter drops its @before fixture for a createTask()
helper so a test can build a task with a different default_timezone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
column_options[].timezone and fields[].timezone defaulted to "UTC" on the
option itself, so task-level default_timezone was never read and
timestamp -> STRING/DATE/DATETIME (and string/nested-field parsing) ignored
it. Make timezone Optional and resolve `timezone || default_timezone` at
every use, as ruby's ValueConverterFactory does.

FLOAT conversion added epoch seconds and a nanosecond fraction as doubles,
which rounds twice and differs from ruby's Time#to_f near the epoch
(-1ms gave -0.0010000000000000009). Sum via BigDecimal and round once.

README gains an upgrade note: how to restore the old offset-bearing STRING
default, and that INTEGER/FLOAT milliseconds can't be restored.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The base INTEGER test now uses a sub-second value so it pins truncation on
its own, replacing the separate truncation test; the pre-1970 floor test
moves next to it. For FLOAT, the -1ms case already covers a negative value,
so it takes over the beforeEpoch name and the redundant -500ms case is
dropped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@chikamura

chikamura commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author

@john-abre
Thanks for the detailed review. Addressed in 87d3839, 64bef3d and cd1dffc.

  • default_timezone fallback: column_options[].timezone and fields[].timezone are now Optional with a null
    default, and every converter resolves timezone.orElse(task.getDefaultTimezone()) — timestamp → STRING/DATE/DATETIME,
    string → TIMESTAMP/DATE/DATETIME, and nested RECORD fields — matching ruby's timezone || default_timezone. Added tests
    for each path with default_timezone: Asia/Tokyo, plus one confirming a column-level timezone still wins.
  • INTEGER tests: testConvertTimestampToInteger now uses …999ms so it pins truncation by itself, and a new
    _floorsBeforeEpoch case covers -1ms → -1.
  • FLOAT tests / rounding: tightened the fraction delta to 0.0, and took your BigDecimal suggestion so the sum is
    rounded once like ruby's Time#to_f. The pre-1970 case uses -1ms → -0.001, which covers both the negative value and
    the single-rounding behavior (so I didn't add a separate -500ms case).
  • README: added an upgrade note on restoring the previous STRING default via default_timestamp_format: "%Y-%m-%d %H:%M:%S.%6N %:z", and that INTEGER/FLOAT milliseconds cannot be restored (e.g. TIMESTAMP_MILLIS(col) →
    TIMESTAMP_SECONDS(col)).

@chikamura
chikamura requested a review from john-abre September 28, 2026 05:53
…mp-conversion-match-ruby

# Conflicts:
#	README.md
john-abre

This comment was marked as duplicate.

@john-abre
john-abre self-requested a review October 6, 2026 11:10

@john-abre john-abre left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chikamura ありがとうございます!
Blockerはないですが、#124 のマージでの微調整でどう?

(メモ to self: timestamp_formatありtimezone無いカラムはこの修正でdefault_timezoneになりますので、今までJava版プラグインを使っていた顧客の状況は要確認 👀 )

@chikamura

Copy link
Copy Markdown
Contributor Author

@john-abre
Thanks for the proposal. I agree with the direction — sharing the formatter resolution and caching the formatters both make sense.

That said, while we're working through the ruby-compatibility changes, I'd like to keep the scope of feature/ruby-compatibility limited to closing behavioral gaps with the ruby plugin, so I'd prefer to put this PR on hold for now. Once the ruby-compat series has been merged, I'd like to pick this up again, rebased onto feature/ruby-compatibility (or master).

@chikamura
chikamura requested a review from john-abre October 7, 2026 04:48

@john-abre john-abre left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙇

@chikamura
chikamura merged commit 78bd4dc into feature/ruby-compatibility Oct 8, 2026
@chikamura
chikamura deleted the ruby-compat/08-timestamp-conversion-match-ruby branch October 8, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants