Add OFFLINE mode for ivy_mooncake#4
Open
NotHimmel wants to merge 8 commits into
Open
Conversation
mooncake.create_table opens a loopback connection on the Postgres-mode port, so the new session defaults to ivorysql.compatible_mode = pg even on an Oracle-mode cluster. Oracle types emitted by format_type (number, varchar2, clob) then fail to resolve in the generated CREATE TABLE, producing "error creating table". Propagate the calling session's compatible_mode to the loopback connection(s) so the DDL is parsed in the same mode the source table was defined in. No-op on native PostgreSQL (current_setting returns NULL) and when the caller is already in pg mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mooncake.create_table created the Postgres-side mooncake table before the moonlink RPC and did not clean it up if the RPC failed (e.g. a column type moonlink cannot store in Iceberg). That left a mooncake table moonlink had no knowledge of, so a same-name retry then failed with "relation already exists". Bind the RPC result before inspecting it so the moonlink stream mutex guard is released first: panicking while the guard was held poisoned the mutex and broke every later mooncake call in the backend. On error, drop the just-created mirror table over the loopback before re-raising. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The main manifest referenced Mooncake-Labs/rust-postgres without a rev while ivy_moonlink pins 14c8e599. Same commit, but the differing source strings make `cargo vendor --sync` see two copies of the same package and refuse to vendor. Pinning also stops the dependency from floating to whatever the branch head happens to be. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For fully offline ("完全内网") deployments. On a connected machine,
`make offline-bundle` produces a self-contained tarball:
- source tree + recursive submodules, real .git dirs replaced by the
marker stubs the Makefiles check, so no git is needed offline
- vendor/ with every Rust crate from both workspaces (cargo vendor
--sync ivy_moonlink), activated by .cargo/config.offline.toml
- the pinned duckdb-httpfs source for the FetchContent gate in
ivy_duckdb (FETCHCONTENT_FULLY_DISCONNECTED + SOURCE_DIR)
- the signed community mooncake.duckdb_extension; pre-placed in the
extension_directory it turns the runtime INSTALL into a no-network
cache hit
On the air-gapped host `make OFFLINE=1 install` swaps the vendored
cargo config in for the duration of the build (trap restores it) and
fails loudly if vendor/ is missing instead of silently reaching for
the network. cargo-pgrx spawns its own cargo and forwards no
--offline/--config flags, hence the config-swap approach plus
CARGO_NET_OFFLINE. Online builds are unaffected.
make install/package/run/test now also pass the pgrx feature matching
PG_VERSION; previously the feature came from the Cargo default and
could target a Postgres version pgrx had not initialized.
Verified end-to-end under `unshare -rn` (no network namespace): full
DuckDB rebuild from a clean build dir plus dependency-tree resolution
with an empty CARGO_HOME, followed by a working mooncake mirror query.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The submodule pointer still referenced the pre-fix upstream revision, so fresh clones (and `git submodule update`, which make offline-bundle runs) checked out ivy_duckdb without the Oracle-mode cast rewrite or the OFFLINE FetchContent gate -- the update silently reset a working tree that had them. Those commits live on the NotHimmel fork, so the .gitmodules URL moves there too; the upstream URL could not serve the pinned revision. ivy_duckdb now pins: 110be79 Gate the httpfs FetchContent fetch for air-gapped builds 9d12304 Rewrite IvorySQL Oracle-mode cast type names for DuckDB Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The air-gapped runtime step previously required hand-copying mooncake.duckdb_extension into every data directory. Fold it into the build instead: `make OFFLINE=1 install` now places the bundled file under <sharedir>/pg_duckdb/extensions, and one postgresql.conf line points DuckDB's extension cache there: duckdb.extension_directory = '<sharedir>/pg_duckdb/extensions' Tying the cache to the installation rather than a data directory means re-initdb needs no re-copying, and no PGDATA has to exist at install time. DuckDB resolves <version>/<platform>/ under that directory, which is exactly the layout the bundle ships; with the file pre-placed the runtime `INSTALL mooncake FROM community` is a zero-network cache hit. Verified live: extension_directory switched to the sharedir path, cluster restarted, mirror-table queries load the extension from there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Everything in the bundle except the mooncake DuckDB extension is source and builds natively on whatever architecture the air-gapped host runs; that one binary was pinned to the packaging machine's uname -m. Accept a platform list instead: BUNDLE_PLATFORMS="linux_amd64 linux_arm64" make offline-bundle Default stays the packaging machine's platform. install-duckdb-extensions already copies the whole tree and DuckDB resolves its own <platform>/ subdirectory at runtime, so one bundle can serve mixed-arch fleets. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
IvorySQL/ivy_duckdb merged the Oracle-mode cast rewrite and the OFFLINE FetchContent gate (upstream PR IvorySQL#4). Move the submodule URL back from the NotHimmel fork to upstream and pin the merge commit, so fresh clones and offline-bundle's `git submodule update` track upstream directly. ivy_duckdb now pins: 505507c Merge pull request IvorySQL#4 from NotHimmel/main (contains 110be79 gate + 9d12304 cast rewrite) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.