Skip to content

[fix](fe) Avoid reusing dropped column unique ids - #67239

Open
Yukang-Lian wants to merge 1 commit into
apache:masterfrom
Yukang-Lian:codex/fix-dropped-column-unique-id
Open

[fix](fe) Avoid reusing dropped column unique ids#67239
Yukang-Lian wants to merge 1 commit into
apache:masterfrom
Yukang-Lian:codex/fix-dropped-column-unique-id

Conversation

@Yukang-Lian

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: None

Related PR: #63173

Problem Summary:

Light schema change uses column unique IDs to match FE slots with columns stored
in existing segments. Some supported legacy metadata can contain a
maxColUniqueId lower than an ID that is still present in the current schema.

The DROP COLUMN path replaced the index schema before preserving its unique-ID
high-water mark. Dropping the highest-ID column could therefore lose that
historical ID, and a later ADD COLUMN could assign it again. Existing segments
could then be interpreted as containing values for the new column.

This change normalizes maxColUniqueId when metadata is deserialized and
preserves the maximum from both the old and new schemas during light schema
changes. It prevents future drops from losing a high-water mark that is still
available in current metadata.

This is a preventive fix for metadata whose high-water mark is still inferable.
It cannot reconstruct an ID that was already absent from both the schema and
maxColUniqueId before upgrade.

Release note

Prevent future light schema changes from reusing dropped column unique IDs when
a legacy high-water mark is stale. Metadata that had already lost its historical
high-water mark before upgrade is not repaired by this change.

Check List (For Author)

  • Test

    • Regression test
      • Added a Docker fault-injection test that writes an old segment, drops
        and re-adds a column, verifies the old row returns NULL, and verifies
        a new row returns the newly written value.
      • Ran LOCAL_DORIS_PATH=$PWD/.doris-compose ./run-regression-test.sh --run -d fault_injection_p0 -s test_readd_dropped_column_unique_id -image doris-pr63173-test:latest -runMode not_cloud.
    • Unit Test
      • Added DDL coverage for non-reused IDs and metadata deserialization
        coverage for repairing a stale maximum while preserving a higher one.
      • Ran ./run-fe-ut.sh --run 'org.apache.doris.alter.SchemaChangeHandlerTest#testReaddDroppedValueColumnUsesNewUniqueId,org.apache.doris.catalog.MaterializedIndexMetaTest'.
      • Ran NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe.
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Light schema changes preserve the column unique-ID high-water
      mark so newly added columns do not reuse IDs still known to the metadata.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#63173

Problem Summary: Light schema change uses column unique IDs to match FE slots
with columns stored in existing segments. Some supported legacy metadata can
contain a maxColUniqueId lower than an ID that is still present in the current
schema.

The DROP COLUMN path replaced the index schema before preserving its unique-ID
high-water mark. Dropping the highest-ID column could therefore lose that
historical ID, and a later ADD COLUMN could assign it again. Existing segments
could then be interpreted as containing values for the new column.

This change normalizes maxColUniqueId when metadata is deserialized and
preserves the maximum from both the old and new schemas during light schema
changes. It prevents future drops from losing a high-water mark that is still
available in current metadata. It cannot reconstruct an ID that was already
absent from both the schema and maxColUniqueId before upgrade.

### Release note

Prevent future light schema changes from reusing dropped column unique IDs when
a legacy high-water mark is stale. Metadata that had already lost its historical
high-water mark before upgrade is not repaired by this change.

### Check List (For Author)

- Test: Regression test / Unit Test
    - `./run-fe-ut.sh --run 'org.apache.doris.alter.SchemaChangeHandlerTest#testReaddDroppedValueColumnUsesNewUniqueId,org.apache.doris.catalog.MaterializedIndexMetaTest'`
    - `LOCAL_DORIS_PATH=$PWD/.doris-compose ./run-regression-test.sh --run -d fault_injection_p0 -s test_readd_dropped_column_unique_id -image doris-pr63173-test:latest -runMode not_cloud`
    - `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe`
- Behavior changed: Yes. Light schema changes preserve the column unique-ID
  high-water mark so newly added columns do not reuse IDs still known to the
  metadata.
- Does this need documentation: No

Co-authored-by: Siyang Tang <tangsiyang@selectdb.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants