Conversation
b2aed36 to
2ac61fe
Compare
lfrancke
moved this from Development: Waiting for Review
to Development: In Progress
in Stackable Engineering
Hive 4.0.1 removed get_table and get_table_objects_by_name from the metastore Thrift interface. Any client built against a Hive 2.3 metastore client still calls them, and that is what Spark bundles: 2.3.9 in Spark 3.5, 2.3.10 in Spark 4.0 through 4.2. Iceberg's HiveCatalog reaches the metastore through IMetaStoreClient and therefore uses whichever client Spark supplies, so an Iceberg job against a Hive 4.0.1 or newer metastore fails. This restores the methods as they were in Hive 4.0.0.
2ac61fe to
339a2e8
Compare
lfrancke
moved this from Development: In Progress
to Development: Waiting for Review
in Stackable Engineering
fix(opa,statsd-exporter): Detect the Go dependency licenses and report modules instead of packages
* fix(patchable): Fail when the mirror rejects the pushed base ref libgit2 reports a reference the remote refused through the push_update_reference callback rather than by failing the push, so `init version --mirror` logged success and wrote a patchable.toml with a mirror that never received anything. --ssh selects a transport for a single invocation, but its rewritten URL was stored in patchable.toml, which forces SSH on everyone who checks that file out and makes `checkout --ssh` fail because the stored URL has no https scheme. An existing configuration is now detected before the fetch and the push, so a re-run fails immediately instead of mirroring and then refusing to overwrite. The fmt layer wrote to stderr directly instead of through the IndicatifLayer, so log lines were printed on top of a progress bar that was never cleared. * chore: Bump libssh2-sys to 0.3.3 0.3.2 is yanked, which fails the cargo-deny advisories check. It is reached through git2, so both boil and patchable depend on it.
* chore(boil): Remove some dead code We ideally want dead_code_pub_in_binary = "warn", but that is currently nightly only and produces noise when running unit tests. For now, we could replace all pub items with pub(crate) to get proper dead code warnings again. * fix: Bump rustls to 0.23.45 to negate RUSTSEC-2026-0285
* feat(shared/statsd-exporter): Bump to 0.31.0 * changelog
NickLarsenNZ
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
There was a problem hiding this comment.
Hive has - in some places - hardcoded the wrong version which means that Hive would not actually use the things we build but instead downloads the dependencies from Maven Central.
Wow!
Is this something we can/should upstream?
There was a problem hiding this comment.
Part of it is already upstreamed.
https://issues.apache.org/jira/browse/HIVE-29827 and some other related work is ongoing https://issues.apache.org/jira/browse/HIVE-30010 which is why I didn't want to open another PR while that second one seems to be in progress. But yes, in theory we should upstream all of this, I had a reason not to though in this case.... should those upstream issues not move we should revisit that.
| mvn \ | ||
| clean package \ | ||
| -Dhadoop.version=${HADOOP_VERSION}-stackable${RELEASE_VERSION} \ | ||
| -Dstandalone-metastore.version=${NEW_VERSION} \ |
There was a problem hiding this comment.
How did you find this? Did you just see mvn output showing dependencies without -stackable?
There was a problem hiding this comment.
Is this something we need to do across all other products?
There was a problem hiding this comment.
Claude found it.
It was relatively easy though: We looked at the resulting folder with the jars and found Hive jars that didn't have our stackable suffix. All of those should come from "within" the build so we hunted it down.
I had Claude look briefly at our other Dockerfiles and it said they are looking good but I don't think I let it look at the results in the actual images, so it might still be worth it.
#1638, #1641 and #1664 landed on main and appended to the same Fixed section and link-definition block this branch adds to. Merged rather than rebased so the reviewed commit stays untouched and the PR gains a commit rather than a rewrite. The resolution is mechanical: both sides' entries kept, link definitions sorted.
|
Had to fix the changelog conflict. |
5 participants
Description
Hive 4.0.1 removed get_table and get_table_objects_by_name from the metastore Thrift interface. Any client built against a Hive 2.3 metastore client still calls them, and that is what Spark bundles: 2.3.9 in Spark 3.5, 2.3.10 in Spark 4.0 through 4.2. Iceberg's HiveCatalog reaches the metastore through IMetaStoreClient and therefore uses whichever client Spark supplies, so an Iceberg job against a Hive 4.0.1 or newer metastore fails.
This restores the methods as they were in Hive 4.0.0.
This also fixes a related problem: We set the version to build to include the
-stackablepostfix. And we'd expect Maven to use those artifacts it builds in the same build process. Well...it didn't because Hive has - in some places - hardcoded the wrong version which means that Hive would not actually use the things we build but instead downloads the dependencies from Maven Central. This fixes that as well.I ran the Hive kuttl test against this and they all succeeded. That doesn't say much because they don't test the Iceberg behavior which is why there is another PR for hive-operator upcoming that adds a test there.
Definition of Done Checklist
Note
Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.
Please make sure all these things are done and tick the boxes
TIP: Running integration tests with a new product image
The image can be built and uploaded to the kind cluster with the following commands:
See the output of
boilto retrieve the image manifest URI for<MANIFEST_URI>.