Skip to content

[Feature] Support TAG value filtering for table subscriptions - #18639

Open
Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:feat/subscription-tag-filter
Open

Caideyipi wants to merge 1 commit into
apache:masterfrom
Caideyipi:feat/subscription-tag-filter

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Table subscriptions can select databases, tables, and output columns, but they cannot filter individual rows by TAG value. This PR adds the tag-filter topic attribute for table-model subscriptions. Its default value is true, so existing topics keep their current behavior.

Filter semantics

  • Parse a restricted SQL predicate over TAG columns. The supported forms are AND, OR, NOT, =, !=, IN, LIKE, regexp_like, and IS NULL, evaluated with SQL three-valued logic.
  • Validate expressions on CREATE/ALTER and bind referenced fields against table schema. Missing schema, stale bindings, unknown fields, and non-TAG fields fail closed instead of leaking unfiltered rows.
  • Filter Tablet rows while preserving row order, data types, timestamps, and null bitmaps. TAG filtering runs before column-filter projection.

Delivery paths and configuration changes

  • Apply the filter to Record/Tablet and TsFile delivery, including normal subscription queues, consensus subscription queues, and Tree View projection.
  • Snapshot filter configuration per event batch so ALTER TOPIC takes effect on subsequent batches without changing the meaning of an in-flight batch.
  • Rebind matchers when table schema versions change, and clean up temporary TsFile rewrite resources on empty output and failure paths.

Validation

  • DataNode Spotless and Checkstyle: passed.
  • TopicConfigTest: 10 tests passed.
  • SubscriptionConsumerMultiProviderPollTest: 3 tests passed.
  • Focused DataNode subscription tests: 36 tests passed before the final review fixes.
  • git diff --check origin/master...HEAD: passed after rebasing onto current origin/master.
  • A fresh DataNode compile reached the recompilation of 2,987 sources but was stopped after about 20 minutes because of local resource limits; the clean reactor build and integration tests are left to CI.

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added comments explaining the intent where it would not be obvious to an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • added integration tests.

Key changed/added classes (or packages if there are too many classes) in this PR
  • org.apache.iotdb.db.subscription.tagfilter
  • SubscriptionFilterSnapshot
  • SubscriptionTreeViewProjector
  • SubscriptionBrokerAgent
  • ConsensusLogToTabletConverter
  • PipeTabletEventTsFileBatch
  • TopicConfig and TopicMeta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant