[CI] Extract flink-backward stage from lake to avoid 60min timeout#3692
[CI] Extract flink-backward stage from lake to avoid 60min timeout#3692Guosmilesmile wants to merge 4 commits into
Conversation
|
similar to #3273. |
@luoyuxia Thanks for pointing out. I made some change based on the suggestions from your issue. Reduce the ITCase coverage on the backward-compatibility Flink versions (1.18 and 1.19) to a 6-test smoke subset that covers the essential read/write paths. The full test suite continues to run on the representative version ( Kept (smoke) vs. skipped
|
935723c to
ba431c8
Compare
|
@Guosmilesmile This PR is very useful, but seems the flink/spark3/spark3-scala213 still last about nearly 1 hour, i guess we need to optimize them too. |
|
Flink Common run in 1.20 version, we also run 1.20 in full test-coverage. So reduce the ITCase coverage for 1.20 align with 1.18 and 1.19 reduce cost time for flink module. |
|
Regarding the Spark part, I would prefer to address it in the next PR. Let's focus on handling the Flink part for now. |
Thanks, that makes sense. If you don't have the bandwidth, I'd be happy to take it and open a spark related PR. Would that be okay with you? |
Sure, go ahead. That'd be great, thanks! |
Description
Problem
The
lakestage in CI (defined inci-template.yaml) ran 6 sub-modules sequentially in a single Maven command:This consistently triggered the step-level
timeout-minutes: 60limit, causing the job to fail with:Solution
Split the
lakestage into two independent jobs that run in parallel via the GitHub Actions matrix:lake: fluss-lake, fluss-lake-paimon, fluss-lake-iceberg, fluss-lake-lance (~12 min)flink-backward: fluss-flink-1.19, fluss-flink-1.18 (~47 min)Since both jobs run in parallel (
fail-fast: false), the wall-clock time drops from ~60 min to ~47 min, well within the timeout.Changes
stage.sh: AddedSTAGE_FLINK_BACKWARD/MODULES_FLINK_BACKWARD, split Flink 1.18/1.19 out ofMODULES_LAKE, updatedget_test_modules_for_stage()with new case and updatedcorenegation.ci-template.yaml: Addedflink-backwardto the matrix.