Skip to content

feat(datafusion): support REST Catalog CREATE VIEW#497

Merged
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/rest-catalog-create-view
Jul 11, 2026
Merged

feat(datafusion): support REST Catalog CREATE VIEW#497
JingsongLi merged 2 commits into
apache:mainfrom
JingsongLi:codex/rest-catalog-create-view

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Summary

Add end-to-end persistent CREATE VIEW support for Paimon REST Catalogs through SQLContext. The defining query is planned in the view's owning namespace, its schema is inferred by DataFusion, and the Java-compatible view request is persisted through the catalog API.

Changes

  • add CreateViewRequest, ViewSchema::new, Catalog::create_view, and ViewAlreadyExist
  • implement REST create-view transport, HTTP 409 mapping, IF NOT EXISTS, and HTTP 501 fallback
  • support CREATE VIEW [IF NOT EXISTS] name [(columns)] AS query in SQLContext
  • infer field types/nullability, preserve the canonical DataFusion query, and bind bare relations/functions to the target catalog and database
  • validate explicit column lists and reject unsupported persistent-view modifiers instead of silently ignoring them
  • document the supported syntax and current limitations in the DataFusion README and SQL guide

Testing

  • cargo test -p paimon --test rest_object_models_test --test rest_api_test --test rest_catalog_test
  • cargo test -p paimon-datafusion --lib sql_context::tests -- --test-threads=1
  • cargo test -p paimon-datafusion --lib persistent_rest_catalog_view_ -- --test-threads=1
  • cargo clippy -p paimon -p paimon-datafusion --lib --tests -- -D warnings
  • cargo fmt --all -- --check
  • mkdocs build --strict

Notes

  • Persistent CREATE VIEW is implemented by REST Catalog; other catalog implementations keep the default Unsupported behavior.
  • CREATE OR REPLACE VIEW, materialized/secure views, comments/options, persistent ALTER VIEW / DROP VIEW, and persistent function DDL remain out of scope.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding persistent REST view creation. The main flow and the existing targeted tests look good, but I found two correctness issues that should be addressed before merge. I reproduced both with focused regression cases locally.

Comment thread crates/integrations/datafusion/src/sql_context.rs
Comment thread crates/paimon/src/catalog/rest/rest_catalog.rs Outdated

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the latest changes. Both previous issues are fixed correctly: COPY GRANTS is explicitly rejected with regression coverage, and CREATE VIEW now maps a missing parent database to DatabaseNotExist. The focused DataFusion and REST Catalog tests, formatting check, and diff check all pass locally. LGTM.

@JingsongLi JingsongLi merged commit 4e67c89 into apache:main Jul 11, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants