cas improvements - #2300
Draft
filimonov wants to merge 8 commits into
Draft
Conversation
…blication Two defects surfaced by the `content_addressed_garbage_collection_log` scenario cards for issue #2233. Any `S3_ERROR` timeout during a GC round was recorded as an indistinguishable `Failed` outcome with a free-text error, and a failed round zeroed out the real counters and cleared `i_am_leader`, suppressing the heartbeat and provoking leadership ping-pong on a flaky backend. Transient error codes (`S3_ERROR`, `NETWORK_ERROR`, `ABORTED`, timeouts, `MEMORY_LIMIT_EXCEEDED`) now produce an `Aborted` outcome while keeping leadership, and `system.cas_gc_log` gains an `error_code` column alongside the `Aborted` outcome. Separately, the emulated blob-publication path materialized the whole blob body in memory (about 1 GiB for a 512 MiB blob) under a global mutex; it now streams the body instead. Related: #2233 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…_ERROR A fetch-by-relink that loses the offer-to-confirm race -- the source's ref moved (a merge, a mutation, an outdated-part drop) between the offer and the confirm -- is a designed, fail-closed outcome: the receiver abandons the relink and the replication queue retries, re-selecting the source and the covering part. It was thrown as `NETWORK_ERROR`, which misdescribes it three ways: - both queue executors (`processQueueEntry`, `ReplicatedMergeTreeQueue`-driven `ReplicatedMergeMutateTaskBase`) treat `NETWORK_ERROR` as an unclassified failure, so every refusal printed an Error-level log line with a full stack trace; issue #2219 records a multi-hour false triage chasing a network fault that was never there (up to 53% of relink proofs refuse under small-part load); - stateless `part_log` hygiene checks tolerate the fetch-transient class under the code upstream fetches use for it, `NO_REPLICA_HAS_PART` (e.g. `02265_column_ttl` whitelists exactly that code), so a refusal landing in `part_log` as `NETWORK_ERROR` fails them -- this is what broke `02265_column_ttl` in the CAS lanes on PR #2159 (13/14 reruns under `prefer_fetch_merged_part_size_threshold=1`); - the label suggests retrying the transport, while the one recovery that is unsound here is a byte re-request to the same source. Both relink retry-later throw sites (taxonomy row 3, the confirm refusal, and row 5b, the unresolved promote) now throw `NO_REPLICA_HAS_PART`. The queue behavior is unchanged -- the exception is stored on the entry, backed off, and re-executed -- but both executors demote it to INFO with no stack trace. Unlike `ABORTED` (the other demoted code), it keeps `need_to_save_exception`, so a refusal storm stays visible in `system.replication_queue`; `ABORTED`'s save-nothing shape is the known pathology where a refusal loop runs invisibly with no backoff accounting. `test_confirm_refuses_when_source_dropped_in_window` now pins the classification: the refusal must not appear at Error level, must appear at Information level, and must reach `part_log` only as `NO_REPLICA_HAS_PART`. No message text changed; no generic queue code changed. Closes: #2219 Related: #2159 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…eys renamed yet) Behavior-preserving groundwork for an atomic rename of the CAS wire-format JSON keys from abstract letters (`t`, `k`, `s`, ...) to semantic names (`kind`, `outcome`, `state`, ...), landed as its own phase so the rename itself is a single reviewable diff. Adds `WireKey` and per-encoding field write helpers, and `EnumWireTable` — a table pairing each enum value with its wire word, proven complete against the enum by a set-equality coverage check with a failing witness for every member. `kMinBlobHeaderLen` gets one compile-time owner instead of several hand-kept constants. `TokenType`, `ObjectKind`, and `BlobHashAlgo` move onto `EnumWireTable`, and the blob-meta, pool-meta, GC state/heartbeat/ maintenance, server-root, blob-envelope, ref-log/ref-ckpt/ref-snapshot/ ref-catalog, run, fold-seal, and gc-outcomes codecs are all migrated onto the carriers — every one of them still writing its existing wire spelling. `RunMarker` becomes a typed enum, and the format test battery is closed out with a set-equality check over the codec registry. No wire-format bytes change in this phase; the follow-up phase (next commit) performs the actual key cut. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The actual key rename, on top of the phase-1 carrier infrastructure. The
format generation history is first reset to a `{1, 1}` baseline, since CAS
has no released, persisted data yet and pre-release generations exist only
to prove the evolution machinery.
Every CAS wire format switches its JSON keys from single-letter/abbreviated
spellings to descriptive names in one pass: the shared `BlobRef`/`Token`/
`ManifestRef`/binding fields, `cas_blob_meta`, `cas_pool_meta` (`algos_used`
becomes a JSON word array instead of a bitmask), GC state/heartbeat/
maintenance state, the server-root record (`MountLease::min_active` becomes
`min_active_build_sequence`), `cas_ref_ckpt`, `cas_ref_log` (the seal link
becomes `!prev_epoch`/`!prev_seq`), `cas_ref_snapshot`, `cas_part_manifest`,
`cas_run`, `cas_gc_outcomes` (`kind`/`outcome`), the fold-seal record and its
`CoverageClass` words, the blob descriptor (with its 239-byte worst case
proved at compile time against the 240-byte floor), and `cas_ref_catalog`.
Golden tests are re-pinned to the new bytes throughout.
Token-group requiredness is unified through `TokenFields::build`: an outcome
missing its token now fails closed instead of serializing a partial group.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…enchmark Small correctness and review follow-ups to the wire-key cut: the part manifest names its namespace field the way every other object does, the algorithm set is read from the proven `EnumWireTable` instead of two independent hand-kept lists, the GC lease and heartbeat keep their separate owner spellings (documented, not merged), and the wire-format word writer gets the contract it was always assumed to have. Extends the `benchmark_cas_ref_protocol` harness to cover every format and direction the wire-keys design measures, plus a review-round fix to that harness. Also fixes `c++expr`: the generated work function needs internal linkage, without which ClickHouse-mode compilation did not work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…speedup Full before/after measurement of the wire-key cut found decode of four of the five formats barely slower, with `cas_fold_seal` the exception (its short strings make the longer keys dominate). Chasing that, the JSON object reader and the per-format row reader are now reused across a stream's rows instead of rebuilt for each one, cutting decode time 57-81% (53-79% net of the key-length cost). A separate copy-free string-read attempt was measured at a 6-7% regression on `cas_ref_catalog` and is not included here. Also lets the full stateless test suite run locally: `functional_tests.py` turns on verbose output for the dataset-attach step (so a `DNS_ERROR` that only fires outside CI doesn't get swallowed and misread as a Kafka failure downstream) and extends the "skip stateful tests when running locally" guard to a local run with no test selector at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ALTER TABLE ... EXPORT PARTITION` was refused with `SUPPORT_IS_DISABLED` on a content-addressed disk because it is absent from the partition-command allowlist in `MergeTreeData`. The rejection it fell into says the command "clones parts file-by-file with no transaction, which would corrupt the clone", and that reason does not describe exporting. `ExportPartTask` reads the source part through `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) under `readLockParts` and writes rows into the destination through a `SinkToStorage` on an ordinary query pipeline. Nothing is hard-linked or copied on the source disk; the command's own bookkeeping is in ZooKeeper. So the allowlist was rejecting it by omission rather than by an argument that applies to it, which the code around it already half concedes: `EXPORT_PARTITION` is listed among the commands permitted to target `PARTITION ALL` a few lines above. Verified end to end rather than by inspection, on a server built from this change: a `ReplicatedMergeTree` source on a CAS disk holding (1,2020), (2,2020), (3,2021), exported to an `IcebergLocal` destination. `EXPORT PARTITION ID '2020'` succeeds and the destination holds exactly (1,2020) and (2,2020) — the right partition, and the 2021 row correctly absent. Two limitations surfaced on the way and are NOT addressed here, because neither is about CAS. Export is implemented only for `ReplicatedMergeTree`: a plain `MergeTree` source now returns `Code: 48 NOT_IMPLEMENTED` instead of the CAS refusal, so the reproduction in the report — which uses a plain MergeTree — will still fail, just for its real reason. And the operation remains behind the server setting `allow_experimental_export_merge_tree_partition`. Closes: #2291 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KiHKrvEVy8u4nA1A8qYFUY Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
TBD
Documentation entry for user-facing changes
TBD
CI/CD Options
Exclude tests:
Regression jobs to run: