[draft] Master catalog spi 11 hive#65474
Open
morningman wants to merge 356 commits into
Open
Conversation
…t sub-steps (WC1-WC4) + flip residuals Code-grounded recon (3 dimension readers + completeness/ordering critics, all HEAD-verified) of the remaining pre-flip write-chain items. Key output: a three-way dormancy taxonomy (A=unreachable, B=value-identical, C=live-with-no-op-default) and a risk-ascending order -- WC1 relocate BIND_BROKER_NAME (B), WC2 engine-map hms cases (A), WC3 partition-spec INSERT reject (C, needs live-connector no-regression test), WC4 read-ACID query-finish commit wiring (C, the hard flip prerequisite). Items 2b/3 carry no dormant work (flip-time deletes R1/R2). Authoritative plan for the write-chain build-out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…_NAME to BrokerProperties (single source) Value-identical relocation (both literals are "broker.name"). The generic ExternalCatalog base no longer depends on the HMS subclass for the broker-name key: - BrokerProperties.BIND_BROKER_NAME_KEY: private -> public (now the single source of truth, read by both guessIsMe case-insensitively and ExternalCatalog.bindBrokerName case-sensitively). - ExternalCatalog.bindBrokerName(): reads BrokerProperties.BIND_BROKER_NAME_KEY instead of HMSExternalCatalog.BIND_BROKER_NAME; drop the now-unused HMSExternalCatalog import, add BrokerProperties. - HMSExternalCatalog.BIND_BROKER_NAME: deleted (verified single external reference at HEAD). Lookup semantics kept byte-identical (case-sensitive Map.get) -- deliberately NOT harmonized to case-insensitive, which would silently change the resolved broker name for a differently-cased key. Iron-rule cleanup: removes a fe-core base-class -> HMS-subclass constant dependency. Dormant/no-op (value provably unchanged for every catalog). Test: BrokerPropertiesTest pins BIND_BROKER_NAME_KEY == "broker.name" (the shared-key regression surface). Green: fe-core BrokerPropertiesTest 6 + HiveScanNodeTest 6; checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…IND_BROKER_NAME relocation) DONE; next = WC2 Records the write-chain decomposition doc and WC1 (6c01805109c). Sets the risk-ascending path WC2 (engine-map hms cases, unreachable) -> WC3 (partition-spec reject, needs live-connector no-regression test) -> WC4 (read-ACID query-finish commit wiring, the hard flip prerequisite), with the flip-time residuals (R1-R4) and the post- write-chain candidates (hudi write-reject, getNewestUpdateVersionOrTime[blocked on D2]) spelled out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ation (no-regression); hudi now gates the flip Code-grounded recon (4 readers + completeness/ordering critics, all HEAD-verified) turning the three signed decisions (sibling delegation like iceberg / no incremental+time-travel+MVCC regression / plan-first) into an ordered, HEAD-grounded plan (HD-A0..HD-D1, groups A-D). Key results: - The flip is atomic: adding "hms" to SPI_READY_TYPES converts hive + iceberg-on-HMS + hudi-on-HMS at once, so hudi's no-regression build-out is a HARD flip-blocker (the iceberg spine alone is not enough). - PhysicalHudiScan wrinkle RESOLVED: a flipped hudi table is a generic PluginDrivenExternalTable -> LogicalFileScan -> PluginDrivenScanNode (same path as iceberg); incremental/time-travel re-home connector-side via the EXISTING resolveTimeTravel/applySnapshot/INCREMENTAL seam (paimon template) -- NO new fe-core SPI, no source-specific BindRelation/CheckPolicy arm (iron rule). - DV-005 model mismatch resolved: getType()="hudi" is a sibling-only type string (createSiblingConnector bypasses SPI_READY_TYPES); never promote hudi into SPI_READY_TYPES. - ONE new design decision surfaced for sign-off (HD-B1): with two siblings the binary handle discriminator breaks; recommend a neutral Connector.ownsHandle(handle) SPI predicate (dormant-unit-testable) over classloader-identity routing. It mutates the already-landed iceberg spine -- must stay byte-behavior-identical on the iceberg arm. - BIGGEST risk (HD-C3, possibly unfixable under the iron rules): COW-incremental row-level _hoodie_commit_time window filter is skipped on the generic path; needs BE-reader verification (BE tree absent here) + a straddling-base-file e2e fixture. Also surfaced a real snapshot-path bug (HD-A3 partition-value parse infidelity -> silent NULL partition columns on non-hive-style partitioned reads). No code (plan-first per sign-off). HANDOFF updated: hudi elevated from a write-reject footnote to a first-class flip-blocking work-stream. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ATE + display (dormant) Adds three switch cases so a flipped HMS external catalog (a PluginDrivenExternalCatalog with getType()=="hms") resolves the right engine names. All strictly unreachable pre-flip (Class A): "hms" is not in CatalogFactory.SPI_READY_TYPES, so a type=hms catalog is still a legacy HMSExternalCatalog / HMSExternalTable and none of these cases is entered until the flip. - CreateTableInfo.pluginCatalogTypeToEngine: case "hms" -> ENGINE_HIVE (CREATE engine; mirrors legacy paddingEngineName/checkEngineWithCatalog instanceof HMSExternalCatalog arm). hive != iceberg, so the getEffectiveIcebergFormatVersion arm stays inert for hms. - PluginDrivenExternalTable.getEngine(): case "hms" -> HMS_EXTERNAL_TABLE.toEngineName() == "hms" (DISPLAY engine; legacy HMSExternalTable showed "hms", NOT "hive" -- distinct from the CREATE engine). Falling through to "Plugin" would regress SHOW TABLE STATUS / information_schema.tables. - PluginDrivenExternalTable.getEngineTableTypeName(): case "hms" -> HMS_EXTERNAL_TABLE.name() (keeps the pluginCatalogTypeToEngine sync invariant documented at CreateTableInfo.java). Dormant unit tests (mock getType()=="hms", no actual flip): CreateTableInfoEngineCatalogTest +5 (no-ENGINE pads hive, CTAS pads hive, wrong explicit engine rejected, hive accepted, iceberg-format non-interference lock); PluginDrivenExternalTableEngineTest +2 (engine "hms" and type name HMS_EXTERNAL_TABLE). fe-core BUILD SUCCESS; 31 tests pass (0 fail); checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… WC3 (explicit-partition INSERT reject) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…n-spec INSERT via neutral SPI (dormant) Ports the legacy hive reject of INSERT INTO hive_tbl PARTITION(p1, p2) (the dynamic partition-NAME list form) into the connector-delegated write path. Legacy fe-core threw "Not support insert with partition spec in hive catalog." in BindSink.bindHiveTableSink; post-flip a hive write binds through bindConnectorTableSink, which never read sink.getPartitions() -- so the reject would be silently lost (a fail-loud regression). This wires it through a neutral SPI so the rejection + message stay in the connector. - fe-connector-api ConnectorWriteOps: new default no-op SPI validateWritePartitionNames(session, handle, partitionNames), mirroring the validateStaticPartitionColumns permissive-default pattern. - fe-connector-hive HiveConnectorMetadata: @OverRide -- foreign (iceberg-on-HMS) handle forwards to the sibling REGARDLESS of emptiness (accepts PARTITION(names) as a standalone type=iceberg catalog does, no heterogeneous-vs-standalone divergence); a hive handle THROWS the exact legacy message on a non-empty list, returns silently on empty. Foreign handle never cast. - fe-core BindSink: new private checkConnectorWritePartitionNames mirroring checkConnectorStaticPartitions (empty-guard -> resolve handle -> SPI -> DorisConnectorException mapped to AnalysisException), called in bindConnectorTableSink with sink.getPartitions(). Keyed on the dynamic partition-NAME list, NOT staticPartitionKeyValues -- a static PARTITION(col='val') INSERT and a plain INSERT ... SELECT are unaffected. Class C (live wiring, permissive no-op default): the fe-core call runs today on live iceberg/paimon/max_compute/jdbc/es, but the empty-guard makes plain INSERT byte-unchanged and no live connector overrides the SPI (all inherit the no-op), so a non-empty PARTITION(names) on them stays silently accepted as before. The hive throw is dormant until hms enters SPI_READY_TYPES. The flip-time delete of BindSink:667-669 (with the rest of bindHiveTableSink) is a residual, not part of this step. Message byte-parity with the pre-existing e2e test_hive_write_type.groovy:250, which runs on both the legacy and the flipped connector path. Coverage mirrors the peer static-partition seam (connector unit test + e2e; no fe-core unit test precedent for bindConnectorTableSink): HiveConnectorMetadataSiblingDelegationTest +1 (hive rejects non-empty with the legacy message, empty is silent, never consults the sibling) and the EXPECTED_WRITE_METHODS Rule-9 completeness lock extended with validateWritePartitionNames (foreign forward proven regardless of emptiness). Adversarial 4-lens review (routing / message-parity / live-no-regression + dormancy / field-keying + binding-completeness) with a refute pass: 0 confirmed findings. api+hive+core BUILD SUCCESS; sibling test 8/8; checkstyle 0 (api/hive/core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… WC4 (read-ACID query-finish commit) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ish commit via neutral SPI (dormant) A transactional (full-ACID / insert-only) hive read opens a metastore read transaction + shared read lock during planScan (HiveScanPlanProvider.planAcidScan -> readTxnManager.register -> openTxn + acquireSharedLock). The matching release (HiveReadTransactionManager.deregister -> commitTxn) had ZERO callers, so a flipped ACID read would leak the metastore shared read lock for the metastore's lifetime. WC4 wires the release through the generic query-finish spine, so fe-core never names hive. - fe-connector-api ConnectorScanPlanProvider: new default no-op releaseReadTransaction(queryId), matching the getDeleteFiles/classifyColumn no-op-default precedent. - fe-connector-hive HiveScanPlanProvider: @OverRide -> readTxnManager .deregister(queryId). The manager is the SAME per-connector instance HiveConnector injects into register and every provider, so the keys match; deregister commits exactly once, is idempotent, and swallows a commit failure (best-effort). - fe-core PluginDrivenScanNode.getSplits(): UNCONDITIONALLY register a query-finish callback (QeProcessorImpl.registerQueryFinishCallback) right after the scan-provider null-guard -- BEFORE the pruned-to-zero short-circuit and planScan -- so a planScan that opens the txn then throws still has its release in place. queryId is computed ONCE (connectorSession.getQueryId()) == the runAndClear key (DebugUtil.printId) == the connector txnMap key. - fe-core buildReadTransactionReleaseCallback: extracted public-static helper returning the Runnable that pins the provider's plugin classloader (onPluginClassLoader) around the release -- load-bearing, since the callback runs on the StmtExecutor thread (app-loader TCCL) and commitTxn resolves metastore/thrift classes by name. Extracted so the release + TCCL-pin is unit-testable without driving getSplits (mirrors applyMvccSnapshotPin). Class C (live wiring, no-op default): the registration runs today on every live plugin scan (es/jdbc/paimon/max_compute/iceberg); only hive overrides the SPI, so the callback is inert for them (pins TCCL, calls the no-op default, never throws) and runAndClear clears it at query finish. Hive's release is dormant until hms enters SPI_READY_TYPES; the legacy HiveScanNode + Env.hiveTransactionMgr path is untouched (its delete + a live-ACID e2e read are flip-time residuals). Foreign (iceberg-on-HMS) handles route to the iceberg sibling provider whose releaseReadTransaction is the no-op default (no hive read txn opened). Tests: HiveReadTransactionTest +1 (provider.releaseReadTransaction commits via the shared manager exactly once, idempotent repeat, no exception on commit -- swallowed); new PluginDrivenScanNodeReadTxnReleaseTest (callback releases for the query; pins the provider classloader DURING the release against a distinct sentinel TCCL and restores it after). Existing scan-node suites unaffected (none drive getSplits end-to-end). Adversarial 5-lens review (placement/dominance, TCCL pin, queryId-identity + registry, live-no-regression + dormancy, connector-release/idempotency) with a refute pass: 0 confirmed findings. api+hive+core BUILD SUCCESS; HiveReadTransactionTest 4/4; all 16 fe-core scan-node classes 78/78; checkstyle 0 (api/hive/core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ormant WC1-WC4 all complete; next thread = hudi flip-blocker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ling-only, never a standalone catalog type Correct the misleading "dedicated Hudi catalogs" framing that invites a future maintainer to "fix" the standalone-type model mismatch by promoting "hudi" into SPI_READY_TYPES: - HudiConnectorProvider javadoc + getType(): "hudi" is a SIBLING-ONLY type string resolved via createSiblingConnector; not a user-facing catalog type; never add to SPI_READY_TYPES / a factory case. - HudiConnector javadoc: built only as an embedded sibling of the hive hms gateway. - CatalogFactory.SPI_READY_TYPES: replace the stale "hms, hudi still use built-in ExternalCatalog" comment (there is NO HudiExternalCatalog) with a guard — hms uses built-in HMSExternalCatalog until the cutover, hudi is parasitic on HMS and served as an hms-gateway sibling, never added here. Comment/javadoc only; zero logic change. Dormant (hms not in SPI_READY_TYPES). checkstyle 0 (hudi + fe-core). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…y synthesis (mirror iceberg S1/S2, dormant)
Give the hive hms gateway a second embedded SIBLING connector for hudi-on-HMS, mirroring the reviewed iceberg
sibling holder. Dormant: no production path references it until the getTableHandle HUDI divert lands (a later
substep).
- New HudiSiblingProperties.synthesize(gatewayProps): a verbatim defensive copy of the gateway catalog's whole
property map. Unlike IcebergSiblingProperties it injects NO flavor key — hudi has no iceberg.catalog.type
analogue; HudiConnector.createClient reads hive.metastore.uris/uri + raw hadoop.*/fs.*/dfs.*/hive.*/s3.*
straight from the map. Whole-map copy avoids silently dropping a connectivity key.
- HiveConnector: add HUDI_CONNECTOR_TYPE="hudi" literal + volatile hudiSibling field + getOrCreateHudiSibling()
(lazy volatile double-checked build via context.createSiblingConnector("hudi", synthesize(props)); fail-loud
naming the catalog when the plugin is absent; failure NOT memoized; held ONLY as the parent-first Connector
interface, never cast — its concrete type is invisible across the loader split). close() forwards to the hudi
sibling too (gateway owns both siblings' lifecycle; each null-guarded, cleared, no-op when never built).
Tests: HudiSiblingPropertiesTest (3: verbatim carry, no-flavor-injected, defensive copy) + HiveConnectorSiblingTest
hudi cases (build/memoize/fail-loud-not-memoized/close-forward) + a closeForwardsToBothSiblingsIndependently
regression guard (adding the hudi arm must not drop/double the iceberg arm). fe-connector-hive 230 tests green,
checkstyle 0, import gate net. 4-dimension adversarial review (holder-parity/close-lifecycle/dormancy-synthesis/
test-intent) + verify: 0 findings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…2-A5) + pending HD-B1 sign-off Record the first two hudi flip-blocker substeps as complete (37b85348667 DV-005 guardrail; 28d4781ccac hudi sibling holder + property synthesis). Next thread = rest of Group A (Kerberos, partition-value fidelity FIX, force-JNI, BE descriptor), plus HD-B1 (3-way foreign-handle routing) as the one decision needing user sign-off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…nsHandle neutral SPI) Record the user's 2026-07-08 sign-off on the one new design decision in the hudi work line: the 3-way foreign-handle routing uses a neutral parent-first Connector.ownsHandle(handle) SPI default (each sibling self-identifies its own handle type in its own classloader), chosen over classloader-identity routing because the routing logic is same-loader unit-testable. Implementation must keep the already-landed iceberg delegation spine byte-behavior-identical. Marked in HANDOFF + the authoritative hudi plan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… (snapshot-path no-regression FIX, dormant) Replace HudiScanPlanProvider.parsePartitionValues (naive split-on-'=') with a byte-faithful port of legacy HudiPartitionUtils.parsePartitionValues, fixing a real snapshot-path correctness regression: Hudi's DEFAULT non-hive-style layout yields relative paths like "2024/01" (no "col=" prefix); the old logic silently DROPPED every prefix-less fragment, so a partitioned hudi table read NULL partition columns on a plain snapshot read, and escaped values (%20, %2F) arrived un-decoded. - Positional mapping: a prefix-less fragment maps to the i-th partition column; a "col="-prefixed fragment contributes its suffix (legacy decides per fragment). - Single-column whole-path fallback when the fragment count mismatches; > 1 column with a mismatch fails loud with the exact legacy message. - Every value URL-unescaped via an inlined unescapePathName (byte-faithful port of Hive FileUtils.unescapePathName, identical to the fe-connector-hive HiveWriteUtils copy) so the connector needs no hive-common dependency. - Method made static + package-private for direct unit testing (no live HoodieTableMetaClient needed); produces a column->value map (the existing consumer shape), not legacy's positional list. Scope: this closes the value-parse regression on the UNPRUNED / most-basic read (getAllPartitionPaths returns Hudi's own relative path = the FileSystemView shape). The PRUNED path (applyFilter) still feeds HMS hive-style partition NAMES, correct for hive-sync'd tables; making the pruned partition SOURCE useHiveSyncPartition-aware for non-hive-style tables belongs to the partition-listing step (which ports that source once) and is likewise closed before the catalog flip — recorded, not silently folded in. Dormant (hms not in SPI_READY_TYPES). New HudiPartitionValuesTest (8: positional / hive-style / mixed / unescape / single-col fallback / prefix-strip / multi-col fail-loud / unpartitioned). fe-connector-hudi 41 + hms 56 + api 61 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… the hudi sibling (dormant)
The hudi connector built its ThriftHmsClient AuthAction from context::executeAuthenticated, which post-flip (the
hudi sibling shares the hms gateway's FE-injected context) resolves to NOOP/SIMPLE — silently downgrading a
Kerberized HMS. Port HiveConnector's plugin-owned authenticator so the metastore RPC runs under the PLUGIN's own
UGI doAs. A hudi sibling runs in its OWN classloader, so it must own its authenticator (sharing the gateway's
hive-loader authenticator would split the UGI copy across loaders); hadoop + fe-kerberos are bundled child-first
in the hudi plugin zip, so the plugin authenticator and the plugin RPC share one UGI copy (same reasoning as the
hive gateway).
- pom: add fe-connector-metastore-hms (HMS Kerberos parser via MetaStoreProviders.bindForType("hms") +
transitively fe-kerberos), auto-bundled child-first by the existing plugin-zip runtime dependencySet.
- HudiConnector: byte-faithful mirror of buildPluginAuthenticator / buildHadoopConf / pluginAuthenticator +
createClient auth-action selection. Two Kerberos sources in precedence order (raw storage kerberos, then
HMS-metastore kerberos with simple storage via HmsMetaStoreProperties.kerberos()); null (FE-injected path
unchanged) for a non-Kerberos catalog.
Dormant (hms not in SPI_READY_TYPES); Kerberos-only, provable end-to-end only at flip-time e2e against a
Kerberized HMS. New HudiConnectorPluginAuthenticatorTest (5, mirror of the hive test, KDC-free — lazy login).
fe-connector-hudi 46 + hms 56 green, checkstyle 0, import gate net.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…MOR detection hardening (dormant) Honor the force_jni_scanner escape hatch (legacy HudiScanNode parity) and harden the scan-path COW/MOR decision. - force_jni_scanner: read via ConnectorSession.getSessionProperties() (same key + VariableMgr.toMap channel as the paimon connector's FORCE_JNI_SCANNER; default false). Threaded through planScan in BOTH directions: (a) canUseNativeReader = isCow && !forceJni -> a COW table under force_jni takes the merged-file-slice (JNI) path (legacy HudiScanNode.canUseNativeReader); (b) collectMorSplits' no-delta-log native downgrade gains a !forceJni guard, and the flag is BAKED onto HudiScanRange so populateRangeParams (which has no session) suppresses its own no-log downgrade too -- keeping the plan-time native/JNI decision and the range-time downgrade CONSISTENT (legacy setScanParams' !isForceJniScanner() guard). - Detection hardening: planScan now derives isCow from the authoritative Hudi table config (metaClient.getTableType() == COPY_ON_WRITE) instead of the substring-detected handle type, so an UNKNOWN detection cannot silently pick the wrong read path for a COW table. (getScanNodeProperties' file_format_type default still uses the handle type -- best-effort, per-split overridden.) Dormant (hms not in SPI_READY_TYPES); end-to-end only provable at flip-time e2e with force_jni on COW/MOR. New HudiForceJniTest (6: session-flag read x4 incl. null/default-false + populateRangeParams true/false contrast pair pinning the downgrade suppression). fe-connector-hudi 52 + hms 56 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…-canonical storage props (dormant) Give hudi tables the correct BE descriptor and BE-facing storage, matching legacy (HudiScanNode extends HiveScanNode -> HIVE_TABLE; getLocationProperties dual merge). - HudiConnectorMetadata.buildTableDescriptor override -> TTableType.HIVE_TABLE + THiveTable (port of legacy HMSExternalTable.toThrift, mirror of HiveConnectorMetadata). Without it the SPI default null makes fe-core build a generic SCHEMA_TABLE descriptor -> BE SchemaTableDescriptor instead of HiveTableDescriptor. No handle in the SPI signature, so the one override serves base + system tables. - HudiScanPlanProvider.getScanNodeProperties storage: now emits BE-canonical static creds from ConnectorContext.getBackendStorageProperties() (AWS_* for object stores / resolved hadoop.dfs.* for HDFS) under location.* for the native (FILE_S3) reader -- BE's native reader understands ONLY canonical keys, so the old raw-alias-only copy 403'd a private bucket. The hadoop-format passthrough (fs.s3a.* etc) is kept AFTER the canonical set (legacy putAll order) for the Hudi JNI reader. Threads the ConnectorContext into the provider (new 2-arg ctor; HudiConnector.getScanPlanProvider passes it). Residual (recorded): a JNI (MOR) read over S3 configured with ONLY Doris aliases (s3.access_key, no fs.s3a.*) still lacks hadoop-format creds -- getBackendStorageProperties gives AWS_* (native), not fs.s3a.*; a hadoop-format normalization hook is a later refinement. Verify the hudi plugin zip carries the JNI reader's transitive deps (be-java-ext shared classpath rule) at flip-time. Dormant (hms not in SPI_READY_TYPES); storage correctness provable only at flip-time e2e over a private bucket. New HudiBackendDescriptorTest (3: HIVE_TABLE descriptor + canonical+passthrough merge + no-context passthrough). fe-connector-hudi 55 + hms 56 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…st set the per-range format explicitly
Adversarial review of Group A caught a real defect (masked by the existing tests): populateRangeParams never
called rangeDesc.setFormatType() for the native path, relying on the SINGLE node-level file_format_type default,
which is wrong whenever a slice's actual format differs from that default:
- MOR read-optimized / no-delta-log slice (insert-only or post-compaction): collectMorSplits stamps
fileFormat="parquet"/"orc" and skips the JNI metadata, but the MOR node default is "jni" -> the range reached
BE as FORMAT_JNI with an all-empty THudiFileDesc -> JNI reader errors/garbage (BLOCKER).
- COW under force_jni_scanner: planScan correctly routes to the JNI path, but the COW node default is "parquet"
-> the range reached BE as FORMAT_PARQUET, silently defeating the escape hatch (native reader used).
- COW ORC table: node default parquet -> an ORC base file read as parquet.
Fix (mirrors PaimonScanRange.populateRangeParams): after the no-log downgrade resolution, set the format type
EXPLICITLY per range -- FORMAT_JNI on the JNI path, else FORMAT_ORC/FORMAT_PARQUET from the range's own file
format (or, for a downgraded "jni" range, the base file suffix). New nativeFormatType helper.
Root cause the old tests missed: HudiScanRangeTest/HudiForceJniTest built ranges with .fileFormat("jni") + a
data_file_path, a shape collectMorSplits NEVER produces for a native slice (it stamps "parquet" directly), so
they exercised the dead downgrade block and passed while production was broken (Rule 9 masking). Added
nativeParquet/nativeOrc slice tests that build the range as the planner actually does, and strengthened the
force_jni test to assert the range reaches BE as FORMAT_JNI.
Dormant (hms not in SPI_READY_TYPES). fe-connector-hudi 57 green (HudiScanRangeTest 4, HudiForceJniTest 6),
checkstyle 0, import gate net.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… (signed Option 1) + Group C/D Record all of Group A complete: A0 guardrail, A1 sibling holder, A2 Kerberos, A3 partition-value fidelity FIX, A4 force-JNI + detection hardening, A5 BE HIVE_TABLE descriptor + BE-canonical storage. Includes the adversarial review catch (ce700db35e0): populateRangeParams must set the per-range format explicitly (MOR no-log native slice + COW-force_jni + COW-ORC were mis-formatted via the single node default). Next = HD-B1 three-way routing (signed Option 1 ownsHandle, must keep the iceberg spine byte-identical), then Group C/D, then HD-B2 pivot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ing via Connector.ownsHandle (dormant) Generalize the flipped hms gateway's binary per-handle discriminator (hive-else-iceberg) to a 3-way router now that a second sibling (hudi) exists: - New neutral SPI Connector.ownsHandle(handle) (default false); IcebergConnector and HudiConnector override it with their OWN in-loader instanceof. fe-core never calls it, so the engine stays format-agnostic. - HiveConnector.resolveSiblingOwner PEEKS the already-built sibling fields (never force-builds) and routes a foreign handle to the sibling that owns it, else fails loud naming the catalog. The 3 connector-level get*Provider(handle) seams + the ~38 HiveConnectorMetadata guard-and-forward sites route through it; the getTableHandle ICEBERG by-type divert keeps the force-build path (renamed icebergSiblingMetadata). - PEEK, not force-build: a hudi-only hms catalog without the iceberg plugin still routes its hudi handles, and the iceberg arm stays byte-behaviour-identical (the owning sibling is always built by getTableHandle before it produces the handle). Dormant until hms enters SPI_READY_TYPES. Foreign handles are never cast. Tests: ownsHandle default (api) + overrides (iceberg/hudi); new HiveConnectorThreeWayRoutingTest (iceberg/hudi/hive routing, unknown fail-loud, peek no-force-build in both directions); updated scan/write/procedure/metadata divert suites for peek + ownsHandle (delegation suite fail-louds the by-type supplier so a site regressing off the peek resolver is caught). 4-lens adversarial review: 0 blocker/major/minor, 2 nits (test tightening) fixed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…DONE; next = Group C (no-regression) + Group D, then HD-B2 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…Group C step 1) authoritative design + HANDOFF Code-grounded design (recon wf_1a09236d-ee0, 5 readers + synthesis, HEAD-verified). Follow the paimon template (override only beginQuerySnapshot + listPartitions/Names/Values; NOT getMvccPartitionView/getTableFreshness/getPartitionFreshnessMillis = dead code under lastModifiedFreshness=false). Pin the latest completed instant as snapshotId; per-partition lastModifiedMillis = the instant; render hive-style partition names (checkState landmine); partition-source consistency (prune-to-zero) + auth/TCCL wiring flagged HIGH. One pending user decision: hudi MTMV freshness scope (legacy is a 0-const stub; recommend real instant freshness). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…al instant freshness, paimon model) hudi MTMV freshness scope signed 2026-07-09 = implement real instant freshness now (legacy was a 0-const stub; new path pins the latest completed instant, an intentional improvement — hudi MVs will auto-refresh on a new commit). Recorded in the design doc + HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…urface (Group C step 1, dormant) Adds the connector-side MVCC/partition surface a partitioned hudi-on-HMS table needs once served post-flip through the generic PluginDrivenScanNode path (like paimon): partition pruning / selectedPartitionNum / SHOW PARTITIONS / TVFs / MTMV freshness. Dormant (hms not in SPI_READY_TYPES; HD-B2 getTableHandle divert not yet armed). Zero fe-core changes; mirrors the paimon template. HudiConnectorMetadata: - beginQuerySnapshot: pin the latest completed instant as the MVCC snapshot (snapshot-id freshness), lastModifiedFreshness LEFT false so fe-core serves MTMVSnapshotIdSnapshot(instant) for the table and MTMVTimestampSnapshot(instant) per partition. INTENTIONAL improvement over legacy HudiDlaTable (which pinned constant 0L and never auto-refreshed) — a hudi MV now refreshes on a new base commit and stays stable otherwise. Do NOT override getMvccPartitionView/getTableFreshness/getPartitionFreshnessMillis (dead code under flag=false). - listPartitions/listPartitionNames/listPartitionValues via a shared collectPartitions: partition-name SOURCE is use_hive_sync_partition-aware (HMS names, else the hudi metadata listing; empty-HMS falls back), mirroring legacy HudiExternalMetaCache.loadPartitionNames. Per partition: hive-style name, unescaped value map, lastModifiedMillis = the instant (stable non-negative marker). Partition NAME rendering (R1): render col=escape(value) via a Hive makePartName port so the generic model's HiveUtil.toPartitionValues re-parse recovers exactly the connector's values with the right arity. Escaping is load-bearing for a single partition column whose value spans '/' (e.g. TimestampBasedKeyGenerator yyyy/MM/dd -> path "2024/01/02"): without escaping the '/' the re-parse would truncate/collide it. Added escapePathName (inverse of the existing unescapePathName). Auth/TCCL (R4): HudiConnector injects a HudiMetaClientExecutor that runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + a TCCL pin to the hudi plugin classloader (the metadata/MTMV thread is not the TCCL-pinned scan thread; post-flip the shared gateway context is NOOP). Shared statics lifted into HudiScanPlanProvider (buildMetaClient / latestCompletedInstant / listAllPartitionPaths) so the MVCC pin and the scan take the identical instant and one copy of the getAllPartitionPaths dance; resolvePartitions refactored to reuse listAllPartitionPaths (byte-identical). Tests: HudiConnectorPartitionListingTest (18) covers instant->long, hive-style rendering + arity + escaped/slash round-trips (incl. the single-column slash no-collision regression), buildPartitionInfos, listPartitions/Names/Values, use_hive_sync_partition source selection + empty-HMS fallback, beginQuerySnapshot pin (static + instance override), and the dead-code SPI-default guards. Reviewed by a 6-dimension adversarial pass with refute-verification (1 major + 2 test-coverage findings, all fixed here; the R2 applyFilter/prune-to-zero residual is pre-existing dormant scan-path code deferred to the flip per the signed design). fe-connector-hudi 76 tests + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ing + freshness) DONE; next = Group C余 (time travel / incremental / schema-evolution) + Group D, then HD-B2 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ormant)
Serve hudi-on-HMS FOR TIME AS OF connector-side (mirroring paimon; byte-faithful
to legacy HudiScanNode), so a hudi table read through the generic PluginDrivenScanNode
path after the catalog flip honors an explicit instant with no regression. Establishes
the query-instant pin spine on HudiTableHandle that incremental read (next step) reuses.
- HudiTableHandle: add nullable String queryInstant (+ getter, Builder setter, copies in
ctor and toBuilder so it preserves prunedPartitionPaths). No equals/hashCode (reference
identity kept; pin excluded from identity, mirroring paimon).
- HudiConnectorMetadata.resolveTimeTravel: TIMESTAMP -> strip [-: ] (legacy parity, no
session-TZ, no epoch-millis, no timeline validation) and pin it via an internal MVCC
property; SNAPSHOT_ID/VERSION_REF -> throw the byte-for-byte legacy reject message
("Hudi does not support `FOR VERSION AS OF`, please use `FOR TIME AS OF`"); other kinds
-> Optional.empty() (SPI default; INCREMENTAL is a later step).
- HudiConnectorMetadata.applySnapshot: property present -> stamp queryInstant via toBuilder
(preserves prunedPartitionPaths, which applyFilter set earlier at scan time); empty
properties (query-begin latest pin) or null -> handle UNCHANGED, so every non-time-travel
read stays byte-identical (planScan falls back to timeline.lastInstant()).
- HudiScanPlanProvider.planScan: honor handle.getQueryInstant() at the single instant
chokepoint; drives COW/MOR before-or-on file selection AND the MOR-JNI THudiFileDesc
instantTime consistently.
Zero fe-core changes (pin flows via the generic MVCC seam: loadSnapshot -> resolveTimeTravel
-> ConnectorMvccSnapshot property -> applyMvccSnapshotPin -> applySnapshot -> handle ->
planScan). Dormant until "hms" enters SPI_READY_TYPES.
Design: plan-doc/tasks/hudi-time-travel-step-design-2026-07-09.md. 5-reader recon +
reconciliation critic + 4-lens adversarial review (no-regression / legacy-parity /
pin-plumbing / test-strength) + refutation verify: 0 confirmed defects.
Deferrals (documented, e2e-owed): schema-at-instant on schema-on-read evolved tables and
the partition-SET-at-instant (dropped-partition edge) both read LATEST -> later steps;
planScan honoring is only e2e-provable (offline unit tests cover routing/normalization/
handle-threading).
Tests: HudiTimeTravelTest (9). fe-connector-hudi 85/85 green, checkstyle 0, import gate net.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…NE; next = Group C余 (HD-C3 增量读) + HD-C4/C5 + Group D, then HD-B2 Design doc for the time-travel step + HANDOFF roll-forward. HD-C2 committed at cf4f0e1b756 (dormant). Two plan corrections recorded in the design (both surfaced by the 5-reader recon + reconciliation critic and confirmed by the 4-lens adversarial review, 0 confirmed defects): FOR TIME AS OF is byte-faithful-permissive (no timeline validation, never errors — the written plan's "validate + notFound" was a paimon-ism that would regress); FOR VERSION AS OF is rejected by THROWING the byte-for-byte legacy message (empty-return would surface fe-core's wrong-domain "can't find snapshot"). Documented deferrals (e2e-owed, not silent): schema-at-instant (HD-C4), partition-SET- at-instant dropped-partition edge, and planScan-honoring (only e2e-provable). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…only neutral-SPI design + sign-off Two code-grounded recon workflows (BE row-filter verdict + FE-only feasibility), both HEAD/BE-source verified. Signed decisions (2026-07-09): architecture = FE-only neutral SPI (connector supplies a synthetic _hoodie_commit_time window predicate via a new connector-agnostic ConnectorMetadata.getSyntheticScanPredicates default-empty SPI; fe-core injects it as an analysis-time LogicalFilter; BE applies it with its existing scan-conjunct machinery) = NO BE change, NO source-specific fe-core branch; D-C3-1 = expose all 5 _hoodie_* meta columns VISIBLE (legacy SELECT * parity). Corrects the stale 'no BE tree in checkout' claim. Planning only; next = implement INC-1..INC-5 (design doc §4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… spine (dormant) Connector-only, dormant (hms not in SPI_READY_TYPES). First of five INC steps for hudi @incr incremental read; extends the HD-C2 FOR TIME AS OF spine. - resolveTimeTravel(INCREMENTAL): resolve the (begin, end] completed-timeline window in ONE connector locus (consolidating legacy's per-relation COW/MOR resolution). begin required (byte-faithful fail-loud message), "earliest"->"000", end default-to-latest, "latest"->latest completed instant tested on the RESOLVED end value (legacy COW form, inherently avoiding the dead-code MOR:92 bug); empty timeline -> (000,000] without the begin-required check. Returns a NON-EMPTY property-only pin (snapshotId/schemaId inert: fe-core's INCREMENTAL loadSnapshot branch lists latest partitions + latest schema and reads only the window props). - applySnapshot: stamp begin/endInstant onto HudiTableHandle (mutually exclusive with the FOR TIME AS OF queryInstant carrier), preserving prunedPartitionPaths. - HudiTableHandle: new beginInstant/endInstant fields + Builder/toBuilder. - HudiScanPlanProvider: extract latestCompletedInstantTime(String) from latestCompletedInstant (byte-identical delegate). The single getCommitsAndCompactionTimeline resolves per table type to exactly the timeline legacy uses per type (verified vs hudi-common 1.0.2 bytecode: COW = getCommitAndReplaceTimeline = legacy COW's metaClient.getCommitTimeline, incl. replacecommit/clustering; MOR = getWriteTimeline) => byte-parity for both. Deferred by design (documented, fail-loud, not silent): populateMetaFields fail-loud + hollow-commit USE_TRANSITION_TIME variant -> INC-2 (relation-family port); raw hoodie.datasource.read.*.instanttime window keys -> INC-4. Adversarial review (4 dims, each finding refuted) + hudi-1.0.2 bytecode verification: 0 confirmed defects (blocker on COW timeline parity refuted). fe-connector-hudi 96 tests (HudiIncrementalTest +11) + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…4), mark done Land the three paimon subgroup fix designs (JNI/COUNT file_format from data-file suffix; nested nullability in Doris->Paimon mapping; honor ignore_split_type), each with its folded-in red-team resolution (workflow wf_05574ccb-bd2, 3 designs x 3 lenses, no UNSOUND). Mark L11/L13/L14 done in the tracking table and roll the HANDOFF: batch 4 paimon subgroup complete, next = iceberg/misc L15-L19. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…already done) The detailed L-list already recorded L3-L9 as DONE with commits, but the top progress table still showed them todo. Confirmed all 7 commits exist in the log (trino 4cd63c6/e27602d4ab6/be96adf76ba/18048f7f217, kerberos 9e4f299/59697ce3fc7, maxcompute 017d1af); mark them done in the table. L2 and L10 remain decision-type (no commit yet). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…apability) L2 (flipped-hive SQL result-cache eligibility + COUNTER) was already resolved by c9a8633 on a separate workstream: CacheAnalyzer/BindRelation/ SqlCacheContext/NereidsSqlCacheManager now recognize plugin tables via the connector-agnostic MTMVRelatedTableIf capability + its data-tied version token (the "add capability / restore cache" decision, not the register-deviation alternative), and the dead source-specific COUNTER_QUERY_HIVE_TABLE bump was intentionally removed. Mark done in the tracking table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…ation (DV-050) Per user signoff (2026-07-12), accept the flipped-external EXPLAIN scan-node name VPluginDrivenScanNode as a display-only deviation (DV-050) rather than add a Connector.getLegacyEngineName SPI to restore the per-connector VHIVE_/ VICEBERG_SCAN_NODE names: the getNodeExplainString CONNECTOR: line already discloses the catalog type, the regression goldens already expect the generic name (only one reference left, already the new name), and this matches Trino (one generic TableScan node with the connector as an attribute). No code change. Also roll the L2-L10 reconciliation: L3-L9 confirmed done in the log, L2 done by c9a8633 (query-cache SPI capability). L2-L10 are now all closed; only L12/L20 remain as decision-type items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
TPC-H: Total hot run time: 29603 ms |
Contributor
TPC-DS: Total hot run time: 180412 ms |
Contributor
ClickBench: Total hot run time: 24.77 s |
The plugin-driven HMS connector wrapped client-creation failures with only the
immediate cause's getMessage(). For a kerberos/SASL misconfig that immediate
cause is Hive's opaque RuntimeException("Unable to instantiate
...HiveMetaStoreClient"), so the real reason (thrift TTransportException: GSS
initiate failed) was dropped from the user-facing message. FE surfaces only the
top exception's message, unlike the legacy ThriftHMSCachedClient/HMSClientException
which appended Util.getRootCauseMessage(cause).
Add ThriftHmsClient.withRootCause() to append the deepest cause (className:
message form, matching the legacy format) at the createFreshClient/borrowClient
throw sites, with a guard so the pool re-wrap does not duplicate the reason.
This restores the thrift/GSS reason that
external_table_p0/kerberos/test_single_hive_kerberos asserts (both err1 and
err2 cases), and keeps the actionable cause visible to users debugging HMS
connection failures. Message-only change; no connector logic altered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
Author
|
run buildall |
… rows The connector-supplied per-value NULL partition flag (58f3e36) plus the pn_NULL naming (a634c5c) changed the MV refresh predicate for paimon's genuine-NULL region partition from region IN ('__HIVE_DEFAULT_PARTITION__') -- which matched no rows and silently dropped the null rows -- to region IS NULL, which correctly materializes them. test_paimon_mtmv order_qt_null_partition now returns 5 rows: the two genuine-NULL region rows (id=2, id=3 -> \N) join the existing 'bj'/'null'/'NULL' rows. The golden still encoded the stale 3-row result and failed with "line 2, BIGINT result mismatch" (TeamCity #992648). Update the golden to the corrected baseline. Values taken verbatim from the CI run's realResults; the partition set itself (p_bj/p_null/p_NULL/pn_NULL) was already correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
Author
|
run buildall |
- root cause: P5 (dbc38a2) dropped the paimon FE subsystem and its scan metrics, but left three dead references to PAIMON_SCAN_METRICS in SummaryProfile (constant + EXECUTION_SUMMARY_KEYS + indentation map). Nothing populates the "Paimon Scan Metrics" profile group anymore (no PaimonMetricsReporter), unlike the still-live ICEBERG_SCAN_METRICS which IcebergMetricsReporter creates/fills. - solution: delete the three dead references; leave ICEBERG_SCAN_METRICS untouched. Zero behavior change (the column never appeared). - tests: none added (dead-code removal, nothing behavioral to assert); fe-core test-compile BUILD SUCCESS, 0 Checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…ision) - L15: mark done (b2cdf97); add FIX-L15 design + summary. - L12: record the 4-agent recon (wf_6c516483-c34) — iceberg is affected (routes through PluginDrivenScanNode), old=SDK-distinct vs new=Nereids- pruned count, divergence only under hidden-partition/data-skipping, 0 golden EXPLAIN blast radius, 3 sql_block_rule tests unaffected by either option, Trino leans Option B. Still awaiting user decision. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
TPC-H: Total hot run time: 29179 ms |
Contributor
TPC-DS: Total hot run time: 180483 ms |
Contributor
ClickBench: Total hot run time: 25.12 s |
…ctedPartitionNum - root cause: migrating to the generic plugin scan node changed selectedPartitionNum (EXPLAIN partition=N/M + sql_block_rule partition_num) from the connector SDK's distinct scanned-partition count to the Nereids declared-partition-column prune count (always >= real). For predicate-driven connectors (iceberg hidden/transform partitioning, paimon non-partition-column manifest pruning) this over-reports the partitions actually scanned and can over-block a governed query that really touches one partition. - solution: opt-in SPI ConnectorScanPlanProvider.scannedPartitionCount (default empty, mirroring supportsTableSample). A pure fe-core helper resolveSelectedPartitionNum prefers the connector's count when present and not under COUNT(*) pushdown (collapsed ranges lose per-partition info), else keeps the conservative Nereids count. paimon counts distinct getPartitionValues(); iceberg counts distinct specId|partitionDataJson (new IcebergScanRange.getScannedPartitionKey). hive/MaxCompute don't override -> keep the Nereids count (they coincide). Generic node stays connector-agnostic (no source branch). - tests: fe-core resolveSelectedPartitionNum helper (8/8), paimon scannedPartitionCount (5/5), iceberg scannedPartitionCount + getScannedPartitionKey (4/4) - all with RED-able distinct-count assertions; full paimon 356/356 + iceberg 978/978 green, 0 Checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…itionCount SPI) - mark L12 done (e5de7ae); add FIX-L12 design + summary. - design red-teamed (wf_f1524868-4b8, 3 lenses all SOUND_WITH_CHANGES); all major/minor findings folded in (iceberg streaming inert = legacy parity, cross-spec key collision fixed via specId, query-cache third consumer registered benign, unregistered under-count edges logged). - update HANDOFF: this session = L15 + L12 done; next = L16-L19, decision-class remaining only L20. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
FE UT Coverage ReportIncrement line coverage |
…neutral SPI - problem: the plugin migration dropped the paimon AND iceberg SDK scan metrics (manifest cache hit/miss, scan/planning durations, files and manifests scanned vs skipped) from the query profile. The connectors cannot write fe-core's SummaryProfile (iron rule), so the legacy reporters were left behind; iceberg's constant only looked live via the dead legacy IcebergScanNode. - solution: a connector-neutral opt-in SPI. New ConnectorScanProfile value type + ConnectorScanPlanProvider.collectScanProfiles (default empty, mirroring scannedPartitionCount). Connectors harvest their SDK metrics during planScan, stash keyed by queryId, and drain in collectScanProfiles; fe-core transcribes them into the profile via the pure static writeScanProfilesInto (no source branch). paimon PULLS from a ported PaimonMetricRegistry (InnerTableScan.withMetricRegistry); iceberg PUSHES via IcebergScanProfileReporter attached on the synchronous data/count path only (not shared buildScan -> no streaming stash leak). DebugUtil's time/byte formatters are self-ported (connectors can't import fe-core). Revives SummaryProfile.PAIMON_SCAN_METRICS (supersedes the L15 deletion). - tests: fe-core writeScanProfilesInto (4/4), paimon harvest via real ScanMetrics/ScanStats (4/4), iceberg report via real ImmutableScanReport (4/4) - all RED-able; group-name mirror checks. Full paimon 360/360 + iceberg 982/982 + fe-core plugin-scan 94/94 green, 0 Checkstyle, import-gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…+ tracking - add FIX-scan-metrics-spi design + summary (superseding the L15 deletion: the migration dropped a real feature for BOTH connectors, so it is restored via a connector-neutral SPI rather than deleted). - record the 3-lens design red-team (wf_0f803c49-7bb) and folded fixes. - update task-list + HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
… properties
The SPI carries partition info to fe-core through the reserved key "partition_columns"
inside the same map that also carries the source table's pass-through properties. fe-core
(PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty "partition_columns" as
the partition-column CSV. iceberg/hive/paimon all putAll(sourceProperties) and then stamp
the key only for genuinely partitioned tables, never removing a pre-existing one — so a
NON-partitioned table whose source properties literally contain "partition_columns" (e.g.
ALTER TABLE ... SET TBLPROPERTIES('partition_columns'='id')) leaks that value into fe-core
and is misdetected as partitioned (wrong pruning / row-count / EXPLAIN partition=N/M).
Fix (producer-side, the only workable locus — fe-core cannot distinguish connector-emitted
from user-passthrough values without parsing connector key semantics it must not know):
each connector removes the reserved key(s) right after the pass-through putAll, before the
conditional stamp, so only its own determination reaches fe-core.
- iceberg: remove "partition_columns" after putAll(table.properties())
- hive: remove PARTITION_COLUMNS_PROPERTY after copying HMS parameters
- paimon: remove "partition_columns"/"primary_keys" after putAll(coreOptions().toMap())
Partitioned tables re-stamp their own value below, so the connector value always wins.
No user-facing regression: getTableProperties() already strips these keys from SHOW CREATE.
Tests (RED-able, recording fakes): iceberg + hive assert an unpartitioned table with a
colliding source property emits no "partition_columns", and a partitioned table still emits
its spec/key-derived value (guard against over-strip). paimon's DataTable coreOptions path
is not reachable via the module's non-DataTable fake -> covered by inspection + e2e-gated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
… (DV-051) IcebergTypeMapping's read direction maps iceberg types Doris cannot represent — the v3 primitives TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN and the non-primitive VARIANT — to UNSUPPORTED, so the table LOADS with only the exotic column present-but-unqueryable. This diverges from legacy fe-core (IcebergUtils threw IllegalArgumentException at schema-load, failing the whole table) and from Trino (throws NOT_SUPPORTED). Per user decision 2026-07-13 the looser behavior is ACCEPTED (registered DV-051): map every unrepresentable column uniformly to UNSUPPORTED so one exotic column does not make a wide table unloadable. No functional change — the two default arms already return UNSUPPORTED; this adds clarifying comments documenting the intentional divergence, and a guard test (unknownAndV3TypesDegradeToUnsupportedByDesign) that pins the choice so a future mutation to throw turns RED. The write direction (toIcebergPrimitive) still throws — CREATE TABLE must not silently accept a type it cannot round-trip. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…etry (L16) Reverify apache#65185 L16 (snapshot/schema cache skew) is VERIFIED BENIGN at HEAD: the field-id dict's snapshot-pin arm already passes the FULL pinned schema (Collections.emptyList(), not requestedLowerNames), and the snapshot-vs-schema selection is skew-free by construction — the (snapshotId, schemaId) pin is atomic, iceberg schemas() is append-only, and the dict-schema selector (IcebergScanPlanProvider.pinnedSchema) and slot-schema selector (IcebergConnectorMetadata.getTableSchema) use the SAME schemaId lookup + the SAME silent fallback to table.schema(), so their top-level names always agree. That agreement is a load-bearing invariant: a divergence would make BE's unconditional children.at(name) std::out_of_range-SIGABRT the whole BE on a schema-evolved time-travel read. Add cross-referencing comments on both fallback sites stating they MUST stay identical, so a future "defensive" edit hardening one side (e.g. throw-loud on a missing schemaId) does not silently reintroduce the asymmetry it looks like it prevents. Comments only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Schema binding for a PluginDrivenMvccExternalTable is version-BLIND (getSchemaCacheValue has no per-reference selector) while the data path is version-AWARE (PluginDrivenScanNode.pinMvccSnapshot). For a self-join across a schema change (t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b, or v1 joined with a latest reference), a reference's FE tuple can be bound at a different schema than the version it scans, so BE field-id/name-mismatches file columns to tuple slots -> crash / wrong NULLs. Newly possible in this branch (the data path became per-reference version-aware while schema binding stayed version-blind). Per user decision 2026-07-13: fail loud now; the per-reference version-aware schema-binding refactor is tracked as design debt D-MVCC-VERSION-SCHEMA. A 3-lens red-team overturned an analysis-time getSchemaCacheValue guard: it is order-dependent and silently masked by a plain/latest reference's default pin, by @incr, and by MTMV refresh's pre-seeded default (the same query throws or silently skews by binding order). Instead guard at the scan node where each reference's OWN version-aware pinned schema is resolved: verify every bound tuple column resolves in that pinned schema (by iceberg field-id when present, else by name), throw UserException otherwise. Deterministic + per-reference: catches the self-join, latest-masked, @incr, and MTMV-refresh cases, with no false positive on t@old JOIN t@latest (each reference's tuple matches its own version). The check is a static, directly-unit-testable helper. No-op for a latest / @incr / sys-table / hive scan (null pinnedSchema). Tests: PluginDrivenScanNodeMvccSchemaGuardTest 7/7 (RED on field-id renumber / added column / paimon name-miss; GREEN on id-stable rename / subset projection / name match / null schema); PluginDrivenScanNodeMvccPinTest 3/3 + PluginDrivenMvccExternalTableTest 59/59 unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…HANDOFF, D-MVCC-VERSION-SCHEMA) Land the L16-L19 fix designs and tracking: - FIX-L16-design.md: verified REFUTED/benign (guard-comment only) - FIX-L17-design.md: scan-node fail-loud guard (red-team overturned the analysis-time placement); registers D-MVCC-VERSION-SCHEMA design debt (user's "refactor later" TODO) - FIX-L18-design.md: accept UNSUPPORTED graceful degradation (user decision), DV-051 - FIX-L19-design.md: producer-side reserved-key strip across iceberg/hive/paimon - deviations-log.md: DV-051 (iceberg unknown/v3 types -> UNSUPPORTED, accepted) - task-list: mark L16-L19 done, add D-MVCC-VERSION-SCHEMA, sync progress table + rolling log - HANDOFF.md: L16-L19 summary + remaining = L20 decision + live-gated e2e Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…rnal. (supersedes silent strip) The SPI passes structural info from connector -> fe-core through reserved control keys inside the same property map that also carries the source table's user pass-through properties. Two of the seven keys were bare, un-namespaced literals scattered across connectors (partition_columns, primary_keys), so a source table whose own properties literally contained such a key collided -- a non-partitioned table could be misdetected as partitioned. The prior fix silently remove()'d the colliding key in three connectors, which discards user data with no signal and gives a future developer no feedback when adding a new reserved keyword. Instead, namespace EVERY reserved control key under a distinctive __internal. prefix (the same mechanism the already-namespaced show.* / connector.* keys use to avoid collision) and centralize them as constants in ConnectorTableSchema: __internal.partition_columns / __internal.primary_keys / __internal.show.location / __internal.show.partition-clause / __internal.show.sort-clause / __internal.connector.per-table-capabilities / __internal.connector.distribution-columns Collision is now impossible by construction: a user's own bare partition_columns property flows through as a normal user property (preserved, rendered in SHOW CREATE), never mistaken for the control key -- no silent strip, no failure. No validation is needed (consistent with the five keys that already relied on their namespace). All reserved keys are FE-only (none is emitted to BE via getScanNodeProperties; BE gets partition columns via the separate path_partition_keys), not GSON/editlog-persisted (schema cache rebuilt on demand), and stripped from SHOW CREATE -- so the rename has zero BE / serialization / golden impact. - ConnectorTableSchema: INTERNAL_KEY_PREFIX, the two new constants, RESERVED_CONTROL_KEYS set. - Producers (iceberg/hive/hudi/maxcompute/paimon) reference the constants; the L19 per-connector remove() strips are deleted. - fe-core: toSchemaCacheValue reads PARTITION_COLUMNS_KEY; getTableProperties strips RESERVED_CONTROL_KEYS (a future reserved key is stripped automatically). - Tests: connectors assert the constant key; new tests assert a bare user partition_columns coexists / flows through (no collision). iceberg 50, hive 23, paimon 12+19+43, fe-core 8+36 green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
- DESIGN-reserved-connector-keys-framework.md: the __internal.-prefix rename (implemented) - FIX-L19-design.md: mark the producer-side silent-strip approach SUPERSEDED - task-list / HANDOFF: L19 done via the rename (supersedes 0166877) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…cription (was '==')
MaxComputePredicateConverter hand-wrote the ODPS operator symbols and the EQ arm
emitted Java's '==', producing pushdown strings like `id == 5`. MaxCompute (like
SQL) has no '==' operator, so ODPS rejects it and the equality predicate silently
fails to push down (full scan). Legacy fe-core MaxComputeScanNode never had this:
it mapped the operator to the ODPS SDK BinaryPredicate.Operator enum and took the
symbol from odpsOp.getDescription() (EQUALS -> "="). SDK bytecode confirms the
canonical descriptions ("=", "!=", "<", "<=", ">", ">="); the connector-api
ConnectorComparison.Operator.EQ also declares "=".
Restore the legacy pattern: map ConnectorComparison.Operator to the SDK
BinaryPredicate.Operator and emit getDescription(), instead of patching one string.
This removes the hand-typed-symbol drift class entirely (single authority = the
SDK). EQ_FOR_NULL ("<=>") has no ODPS equivalent and still falls to default ->
throw -> NO_PREDICATE (BE re-filters), matching legacy's skip. RawPredicate kept
(custom datetime/string value formatting unchanged); only EQ output changes
(id == 5 -> id = 5), the other five operators are byte-identical.
Add guard tests: EQ emits single '=' never '==' (RED-able), full operator set maps
to SDK symbols, EQ_FOR_NULL not pushed, and IN/NOT IN direction (col IN (values)).
Design: plan-doc/tasks/designs/FIX-L20-design.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…scription; series closed Mark L20 done in the reverify tracking table + rolling log and HANDOFF: the last decision-class item is resolved. The whole apache#65185 reverify fix series (H1-H4, M1-M8, L1-L20) is now closed (done or user sign-off accept/skip); only the D-series design debt remains (deferred to P8). All series e2e remain live-gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
only for testing #65473