fix(datafusion): snapshot metadata for synchronous catalog callbacks

[shyjsarah]

Purpose

Refs #872. Supersedes #873 for the metadata-discovery path by fixing the catalog boundary instead of changing Tokio blocking behavior.

DataFusion's metadata discovery APIs are synchronous, while Paimon catalog and external table-engine resolution are asynchronous. Calling those async paths from schema_names, schema, table_names, or table_exist can block a DataFusion worker that the query itself needs, which is exposed by filtered information_schema queries with repartitioning.

This PR intentionally does not close #872. Synchronous mutation callbacks (register_schema, deregister_schema, and deregister_table) still use the existing blocking bridge; removing that bridge requires a separate lifecycle/API change.

Brief change log

• Build database, table, view, and system-table capability snapshots asynchronously when catalog/schema providers are initialized or explicitly refreshed.
• Make synchronous catalog discovery callbacks read only the in-memory snapshot; concrete table loading remains asynchronous.
• Bound automatic information_schema refreshes with a freshness TTL, failure backoff, per-target single-flight, global concurrency limit, and timeout. A failed or timed-out catalog remains best-effort and cannot block healthy catalogs indefinitely.
• Merge concurrent refreshes per database. If a DDL delta rejects a full-refresh publication, re-read that database with bounded retry instead of silently discarding either side.
• Serialize the remote mutation and local snapshot delta for same-database DDL, including database/table DDL interactions, so response ordering cannot invert backend commit order.
• Apply rename deltas from the authoritative mutation outcome. ALTER TABLE IF EXISTS uses a strict backend rename; a missing source succeeds without creating a phantom target and removes a stale-positive source from the snapshot.
• Remove the misleading synchronous provider new(...) shims. try_new(...) is the initialized constructor; new_uninitialized(...) is the explicit deferred path used by Python construction.
• Separate base-object existence from Paimon system-table capability. Native object tables and routed tables no longer advertise $snapshots; classification is bounded and cached so unchanged refreshes do not reload table metadata.
• Add Catalog::list_table_types with a Paimon-only compatibility default and exact bounded implementations for filesystem and REST catalogs.
• Preserve resolver Ok(None) as not found in table(), table_exist(), and table_type().
• Bound retained database-generation tombstones while protecting every tombstone still needed by an active older refresh.
• Keep metadata initialization deferred in Python PaimonCatalog; DataFusion registration initializes the provider and refresh_metadata() remains an explicit strict refresh.

Tests

• cargo fmt --all -- --check
• cargo clippy --locked -p paimon -p paimon-datafusion --all-targets -- -D warnings
• cargo test -p paimon-datafusion --test sql_context_tests (94 passed)
• cargo test -p paimon-datafusion --test table_type_routing (41 passed)
• cargo test -p paimon-datafusion --lib 'sql_context::tests' (176 passed)
• cargo test -p paimon-datafusion --doc (2 passed, 3 ignored)
• cargo test -p paimon --test rest_catalog_test test_catalog_lists_declared_table_types (passed)
• PYO3_PYTHON=/opt/homebrew/bin/python3.11 cargo check --locked -p pypaimon_rust (passed)
• Python suite after rebuilding the extension: 185 passed; 4 shared-warehouse fixture tests unavailable locally

API and Format

PaimonCatalogProvider::new and PaimonSchemaProvider::new are removed because a synchronous constructor cannot preserve the previous ready discovery behavior without restoring the blocking bridge. Migrate initialized callers to try_new(...).await?; use new_uninitialized(...) only when initialization is deliberately deferred and complete initialize_metadata().await? before exposing discovery callbacks.

Catalog::list_table_types is a new defaulted method. Existing Paimon-only catalog implementations remain source compatible; catalogs containing object or external table types should override it.

No storage format changes.

Documentation

Updated provider API documentation to describe the explicit initialized/deferred lifecycle and compile-check the removed synchronous constructor path.

[shyjsarah]

Closing this PR because #891 has landed and addresses the original Python/process-runtime information_schema deadlock.

The snapshot approach here adds a much broader metadata caching, freshness, DDL-coherence, and lifecycle subsystem that is no longer justified for the original incident. #891 does not claim to cover every caller-owned Tokio runtime topology; if that remaining boundary can be reproduced, it should be tracked separately with a focused regression test and a narrower fix.