Skip to content

add Streaming - #1533

Merged
Slach merged 5 commits into
masterfrom
streaming
Sep 4, 2026
Merged

add Streaming#1533
Slach merged 5 commits into
masterfrom
streaming

Conversation

@Slach

@Slach Slach commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

fix #780

Prepare for --streaming mode (#780): split Upload/Download/Restore into
prologue, per-table body and epilogue, extract createOneTable,
uploadOneTable, restoreOneTable, saveLocalBackupMetadata and a
removeTableLocal helper. Behavior of existing commands is unchanged.
…780

create_remote --streaming pipelines freeze -> upload -> remove local copy
per table, restore_remote --streaming pipelines download -> attach ->
remove local copy per table. The create/upload and download/restore
contracts stay the on-disk layout plus per-table metadata.json; the
orchestrators reuse the extracted per-table steps. Streaming runs keep
their own resumable state files: create_upload_streaming.state2 and
download_restore_streaming.state2. watch --streaming / server
--watch-streaming pass the mode through to create_remote.

Also bump Dockerfile builder to golang:1.27-alpine to match go.mod.
Old 1.x ClickHouse has no {database}/{table} macros and no two-argument
numbers(): substitute the macros only when getMacro is absent (same check
as createTestSchema) and use number + offset FROM numbers(rows).
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 33858353957

Coverage increased (+0.4%) to 66.816%

Details

  • Coverage increased (+0.4%) from the base build.
  • Patch coverage: 376 uncovered changes across 10 files (880 of 1256 lines covered, 70.06%).
  • 20 coverage regressions across 10 files.

Uncovered Changes

File Changed Covered %
pkg/backup/restore_remote.go 286 164 57.34%
pkg/backup/create_remote.go 210 136 64.76%
pkg/server/server.go 60 12 20.0%
pkg/backup/restore.go 178 132 74.16%
pkg/backup/download.go 135 106 78.52%
pkg/backup/create.go 204 180 88.24%
pkg/backup/upload.go 149 128 85.91%
pkg/backup/watch.go 8 3 37.5%
pkg/config/priority.go 6 2 33.33%
pkg/backup/delete.go 8 5 62.5%
Total (13 files) 1256 880 70.06%

Coverage Regressions

20 previously-covered lines in 10 files lost coverage.

File Lines Losing Coverage Coverage
pkg/backup/download.go 4 77.16%
pkg/storage/gcs.go 3 54.46%
pkg/backup/create.go 2 74.43%
pkg/backup/restore.go 2 71.9%
pkg/metadata/backup_metadata.go 2 71.43%
pkg/storage/cos.go 2 55.66%
pkg/storage/object_disk/object_disk.go 2 68.63%
pkg/backup/list.go 1 70.48%
pkg/backup/upload.go 1 72.27%
pkg/server/server.go 1 57.05%

Coverage Stats

Coverage Status
Relevant Lines: 25603
Covered Lines: 17107
Line Coverage: 66.82%
Coverage Strength: 35701.02 hits per line

💛 - Coveralls

@Slach
Slach merged commit 99eddff into master Sep 4, 2026
81 of 90 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add --streaming option to create_remote and restore_remote

2 participants