Cherry-pick commits from main to REL_2_STABLE since 03/23 - #142
Merged
Conversation
For RC tags like X.Y.Z-incubating-rcN, generate the source tarball filename and top-level directory using VERSION_FILE (without -rcN). $VERSION_FILE is input from the file version. This keeps the voted bits ready for promotion without rebuilding and avoids -rcN showing up in the extracted source directory.
* Brand: update pxf_fdw Makefile description * update pxf_fdw.control
* Upgrade Gradle from 6.8.2 to 7.6.6 * Replace gradle-lombok plugin with io.freefair.lombok 6.6.3 * Upgrade io.spring.dependency-management to 1.0.15.RELEASE * Add -Xlint:-removal to suppress Security Manager deprecation warnings on Java 17+ * Add --add-opens JVM flags for JEP 486 module access (build and runtime) * Add CI matrix job for Java 8, 11, 17 compatibility testing * Remove server test group from automation (covered by java-compatibility-test) * Refactor build_pxf.sh to use per-component install targets
* fix: add Parquet UUID type support for read/write - Add UUID (OID 2950) handling by mapping UUID to FIXED_LEN_BYTE_ARRAY primitive type. - UUIDARRAY is also supported. - Add BYTEA fallback for unknown FIXED_LEN_BYTE_ARRAY logical types - Add uuid_types.parquet test fixture and corresponding unit tests
- Unify CI scripts from ubuntu/ to common/, supporting both Ubuntu and Rocky/RHEL - via auto-detection of package manager (apt-get vs dnf) - Add Rocky 9 Docker Compose and RPM build script (rocky9/) - Parameterize singlecluster Dockerfile with ARG BASE_IMAGE for multi-distro builds - Add Rocky 9 singlecluster image build job to GitHub Actions workflow - Cross-platform JAVA_HOME auto-detection in build_pxf.sh
* Partially revert 4830298 * Add `@Test(enabled=false)` to failed tests
Update release script to append `-incubating` suffix to the extracted directory name, ensuring consistency with Apache release naming conventions. Example output: - Tarball: apache-cloudberry-pxf-2.1.0-incubating-rc2-src.tar.gz - Extracted: apache-cloudberry-pxf-2.1.0-incubating/
Add NOTICE, DISCLAIMER, and LICENSE files to the stage target in Makefile to ensure they are included in all package formats (tar, rpm, and deb). This ensures proper license attribution and compliance with Apache Software Foundation requirements for all distribution methods. Previously, these files were not copied during the packaging process, which could lead to incomplete license information in distributed packages.
* Run JDBC test in Testcontainers * both Ubuntu 22.04 and Rocky Linux 9
The pxf script uses the 'ps' command in the isRunning() function to check if the PXF process is running. However, minimal container images like Rocky Linux 9 do not include the 'ps' command by default. This causes the 'pxf stop' command to fail with: /usr/local/cloudberry-pxf/bin/pxf: line 162: ps: command not found Add explicit package dependencies to ensure the 'ps' command is available when PXF is installed: - RPM packages: procps-ng - DEB packages: procps The 'ps' command is used in: - server/pxf-service/src/scripts/pxf:162 (isRunning function)
**Cache CPU-heavy CI steps:** * DEB build * RPM build This should speed up our CI pipelines and reduce github actions quota usage. **Implementations details:** * Github cache has 7 days TTL. Every build will reset expiration timeout. In order to force CI to build cloudberry from time to time - I am explicitly specifying current month in a cache key. * keep both: `actions/cache` and `actions/actions/upload-artifact`. In case cache eviction happens during build (e.g. hit 10Gb limit) we will still have running builds. * cache reused between `main` and other PRs (but not between PRs). So, cache will be filled only during builds in `main`.
### Add ClickHouse JDBC tests Add new tests to cover Cloudberry -> PXF -> JDBC path. In this test we are verifying that main data Cloudberry and ClickHouse types can be converted back and forth. Test covers ClickHouse 24.x and 26.x versions. Open-source ClickHouse has short support lifetime. I have tested following clickhouse-jdbc drivers: * 0.6.x (`jdbc-v1`) - works well with old ClickHouse versions. In ClickHouse 25.10 `jdbc-v1` got broken ClickHouse/clickhouse-java#2636 and all queries raised an "Magic is not correct" error message. * 0.9.4 - fixes "Magic" issues, however has issue with String <-> bytea conversion `ERROR: PXF server error : Method: getBytes("bin") encountered an exception.` * 0.9.7+ - works well ### Add jdbc-pxf-drivers project Add new `jdbc-pxf-drivers` project to server. It excluded from default DEB-package build. Explicit actions required to install jdbc-drivers. ### Side quest Cloudberry FDW serializes rows as Greenplum CSV => PXF `TextRecordReader` parses the CSV stream via univocity CSV parser => for `BYTEA` columns it calls `pgUtilities.parseByteaLiteral()` => which returns a `ByteBuffer`.
Speed up our CI pipeline: * cache testcontainer images * do not rebuild singlecluster image on every test run * allow gradle parallel builds It was disabled for gradle 4.x in apache@124115d
- CI workflow: GO_VERSION 1.21 -> 1.24 - Dockerfile: GO_VERSION 1.21.13 -> 1.24.0 - go.mod: go 1.21.3 -> go 1.24 - Documentation: Update minimum Go version requirement * Refactor ci/singlecluster/Dockerfile to use GO_VERSION variable See: apache#96
Update cloudberry-pxf-release.sh to generate artifact filenames
without RC suffix (e.g., apache-cloudberry-pxf-2.1.0-incubating-src.tar.gz)
while keeping RC identifier in directory names only.
Benefits:
- Simplifies promotion process after vote passes
- No need to rename files for checksums/signatures
- Consistent with best practices from other ASF Incubator projects
- Files are ready for final release from RC stage, can use `svn mv` to
promote the RC artifacts to the final version.
Example structure:
dev/incubator/cloudberry/2.1.0-incubating-rc1/
apache-cloudberry-pxf-2.1.0-incubating-src.tar.gz
Use pinned commit hash (4d9f0ba0025fe599b4ebab900eb7f3a1d93ef4c2) instead of version tag @v5 to comply with Apache GitHub Actions policy. Fixes apache#101
Add new tests to cover Cloudberry -> PXF -> JDBC path. In this test we are verifying that main data Cloudberry and Oracle types can be converted back and forth. Test covers Oracle 23 (only supported by Testcontainers version).
… in /server (apache#105) * Bump org.apache.tomcat.embed:tomcat-embed-core in /server Bumps org.apache.tomcat.embed:tomcat-embed-core from 9.0.72 to 9.0.117. --- updated-dependencies: - dependency-name: org.apache.tomcat.embed:tomcat-embed-core dependency-version: 9.0.117 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update server/build.gradle --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dianjin Wang <wangdianjin@gmail.com>
Start HBase only for test groups that require it (hbase, proxy) instead of launching it as part of the default Hadoop stack. This frees RAM for all other test groups. Additionally, run HBase in standalone mode: * No separate RegionServer process * No separate ZooKeeper process * No dependency on HDFS
* Docs: update JDBC-related docs * review
### Add Microsoft SQL JDBC tests Add new tests to cover Cloudberry -> PXF -> JDBC path. In this test we are verifying that main data Cloudberry and MS SQL types can be converted back and forth. Test covers MS SQL 2019 and 2022 versions.
When a numeric column is compared with an integer constant (e.g.,
biz_dt = 20260331), PostgreSQL wraps the constant in an implicit
int4 -> numeric type cast FuncExpr. The filter extraction code in
both the FDW path (fdw/pxf_filter.c) and the external table path
(external-table/src/pxffilters.c) only recognized plain Const
nodes, so these predicates were silently dropped and the query
fell back to a full scan on the remote source.
Before matching the Var+Const pattern, run eval_const_expressions()
on any operand that is neither a Var nor a Const. This folds the
implicit cast into a plain Const, after which the existing filter
serialization path can push the predicate down unchanged.
Fix is applied symmetrically in OpExprToPxfFilter (FDW) and
opexpr_to_pxffilter (external table). Neither function mutates
the original plan tree; only the local leftop/rightop pointers
are redirected at the simplified Const.
Test coverage:
* Extend FilterPushDownTest and FDW_FilterPushDownTest regression
suites with c3 = 5, c3 < 5, c3 > 1, c3 <= 2, c3 >= 5, c3 <> 5,
and c3 BETWEEN 1 AND 5, covering the full set of supported
scalar operators when the constant type differs from the column
type.
* Synchronize the int-const block across both test segments of
FilterPushDownTest.sql and its expected output.
In this PR following issues addressed: * Docker image size reduced from 4.52G to 3.34G (github provides 10G cache) * Runs MS SQL tests with FDW plugin * Always specify username/password for PostgreSQL JDBC tests (required for hardened PXF versions) * minor cleanups
- CI workflow: GO_VERSION 1.24 -> 1.25 - Dockerfile: GO_VERSION 1.24.0 -> 1.25.0 - go.mod: go 1.24 -> go 1.25 - Documentation: Update minimum Go version requirement
1. Use cloudberry 2.x in CI 2. trying to workaround Rocky Linux issues with modular filtering
* chore(docs): remove Bookbinder build stack The PXF documentation is moving to the Apache Cloudberry website at cloudberry.apache.org/docs (built by apache/cloudberry-site, Docusaurus). This repository keeps the Markdown sources under docs/content/ but no longer ships its own build pipeline. Drop the Bookbinder/Middleman scaffolding under docs/book/ that would otherwise become unmaintained. Subsequent commits convert the Markdown sources to a Docusaurus-friendly layout, refresh docs/README.md, and add a docs lint workflow. * docs: convert PXF docs to Docusaurus-compatible Markdown Convert the upstream Bookbinder-flavoured PXF documentation under docs/content/ into a layout that the Apache Cloudberry website (apache/cloudberry-site, Docusaurus) can consume directly. Changes per file: * Rename .html.md.erb -> .md. The .erb sources contained no ERB template code, so removing both suffixes is a no-op for content while letting Markdown tooling pick the files up. * Rewrite intra-doc links from `(foo.html)` and `(foo.html#anchor)` to Docusaurus-style relative paths like `(./foo.md)` or `(../administering/cfg_server.md#about-the-pxf-fs-basepath-property)`. Same-page bare anchors `(#foo)` are also remapped, including a fix for the upstream typo `(#procedure.html)` in pxf_kerbhdfs.md. * Replace heading anchor blocks of the form `## <a id="suppplat"></a> Supported Platforms` with plain headings, since Docusaurus auto-generates slug-based anchors. Cross-file references that used the old IDs are rewritten to the new slug. Stand-alone `<a id>` tags (table captions, mid-section deep links) are preserved as MDX-friendly invisible anchors. * Add `description` and `sidebar_position` frontmatter to every page so the Docusaurus sidebar can be auto-generated and pages get sensible meta tags. Page titles that referenced "Greenplum® Platform Extension Framework" are rebranded to "Apache Cloudberry Platform Extension Framework". * Reorganise the previously-flat `docs/content/` into category sub-directories matching the legacy subnav: `intro/`, `administering/`, `access-hadoop/`, `access-objectstores/`, `access-jdbc/`, `access-nfs/`, `troubleshooting/`, `upgrade/`, plus the existing `ref/`. Each carries a `_category_.json` for the Docusaurus sidebar. * Rebrand prose: "Greenplum Database" -> "Apache Cloudberry" and bare "Greenplum" (where it refers to the deployment, not a specific Greenplum release/version) -> "Apache Cloudberry". Compatibility tables, transition notes, and other historical references that need to keep the original wording are preserved via a small set of guard patterns. * Tweak raw HTML so MDX v3 can render it: `class="..."` -> `className="..."`, and `<a href="foo.html">` rewritten the same way as Markdown links. A few warnings remain that reflect pre-existing dead links upstream (e.g. `#s3_override_ext_ext`, `init_pxf.html`, `#topic1` in ref pages); these are flagged for follow-up but kept as-is so the diff is mechanical. * docs: refresh docs/README.md for the Docusaurus-based publishing flow The previous README described the now-deleted Bookbinder build under docs/book/. Replace it with a brief authoring guide that points at apache/cloudberry-site as the source of the rendered documentation and documents the conventions used by the Markdown sources (frontmatter, relative links, image placement, category metadata). * Mv docs/content/ to docs/ * Fix broken links
For Apache license files, we need to keep the original NOTICE file content if there is one. Seem as apache/cloudberry-backup#102
Fix S3-related tests from apache#86: - Add `MinIOContainer` and `S3Application` to manage MinIO - Generate fixtures via `SmallDataFactory` (prefer composition over inheritance) - Migrate `CloudAccessTest` and `S3SelectTest` to TestContainers suite. - Pre-bake test `s3`/`s3-invalid` PXF server configs into image. - Move S3 tests from the singlecluster to the testcontainers pipeline (`pxf-s3`) and delete the unused legacy MinIO scripting. - Replace s3a client with aws S3 client in tests data management.
v5.0.0 is no longer in the Apache infrastructure-actions allowlist (actions.yml). Bump to v6.1.0 which is already trusted. See: - https://github.com/apache/infrastructure-actions/blob/2b6ec5f38ac73c7c5970f3b4f863e8d15bf12d7d/actions.yml#L424-L438
Update `hbase-client` from `2.3.7` to `2.5.15` (to version with hadoop3 support): * Still use hadoop 2 version. * Update dependencies, where it changed * Automation tests: upgrade hbase to 2.5.15 as well This should prepare PXF migration to Hadoop 3.x
Bump log4j from 2.17.2 to 2.25.4. Closes CVE-2026-34480 Signed-off-by: Sanikadze <kavun.alexsander@gmail.com> Co-authored-by: Sanikadze <kavun.alexsander@gmail.com>
* Fix ZSTD compression support with Parquet * add snappy * aircompressor is a runtime dependency
Use TestContainers for "pxf_extension" test group: 1. use TestContainers 2. explicitly specify extension version - so, there is no need in SQL file manipulation 3. remove 'SELECT' from test - we don't care about external-table 2.0 (pre PXF 6.7).
I am not sure that I understand Gradle Compatibility Matrix[1]. However it seems that Gradle 8.x should work with java ranging from Java 8 to Java 21 well. [1] https://docs.gradle.org/current/userguide/compatibility.html
… timeout) (apache#130) * ci: harden Rocky 9 baseline (TZ pin + HDFS write tuning) Rocky 9 CI has been consistently red since apache#78 introduced Rocky 9 support on 2026-03-26. This commit addresses two orthogonal Rocky 9 stability issues in one topic, so the reproducibility story stays coherent. TZ pin (Etc/GMT-1) ------------------ ParquetTypeConverter.java encodes/decodes INT96 timestamps using ZoneId.systemDefault() in both bytesToTimestamp() and getBinaryFromTimestamp(). Ubuntu 22.04 inherits Europe/London (BST = UTC+1 in summer) from the base image; Rocky 9 defaults to UTC. Same Parquet input, different timestamp strings, and the .ans expected values (calibrated to Ubuntu output) diff on Rocky 9. Pinning TZ=Etc/GMT-1 (fixed UTC+1, no DST) in both distros' compose files makes JVM systemDefault() identical everywhere, matches the PGOPTIONS timezone='GMT-1' already set by run_tests.sh, and preserves existing .ans values (Ubuntu summer already emits UTC+1). HDFS write tuning ----------------- HdfsWritableTextTest multi-block variants timed out or silently dropped writes on shared GHA free runners. Root cause is a combination of slow DataNode ACK under noisy-neighbor I/O contention (default 60s socket timeout kicks in) and unnecessarily large test payloads (1.2 GB) for a 'multi-block' assertion that only needs to cross a block boundary a handful of times. * dfs.client.socket-timeout: 60s -> 180s. Absorbs GHA ACK jitter (worst observed ~90s) without dragging out failure detection when a DataNode is genuinely dead. * dfs.blocksize: 128 MB -> 16 MB. Multi-block tests hit boundaries at 1/8 the data volume; the boundary code paths (LineRecordReader cross-block splits) are exercised the same way. 16 MB (not 4 MB) leaves headroom for textFormatWideRowsInsert's 10 MiB rows to fit in one block, avoiding a Rocky 9-only write/read comparison bug at cross-block row boundaries. * HdfsWritableTextTest data volume: 15000x1000 rows (~1.2 GB) -> 1000x1000 rows (~80 MB). 5 blocks at 16 MB each still exercises multi-block reader/writer paths. All changes are in CI configuration and test fixtures. No PXF Java source, no shipped artifacts, zero customer-facing risk. Related to apache#17. * ci: cache & retry external downloads CI jobs fail intermittently due to transient upstream unavailability of network dependencies pulled at build/test time. Symptoms observed on apache#130 across four rounds of retries: * Rocky HPC SIG dnf mirrorlist returning empty (spurious dnf install failures during container bring-up) * Maven Central returning 5xx on artifactregistry-maven-wagon-*.pom * maven.top-q.co.il (single-source repo for JSystem test framework) returning 504 for jsystemCore-*.pom * mcr.microsoft.com returning transient 5xx during testcontainers' runtime pull of mssql/server:2019-latest for JdbcMssqlTest None of these are our upstreams, but blaming them isn't a fix; the purpose of apache#130 (and apache#17) is to make CI green regardless of external service jitter. Two complementary strategies: 1. Retry package-manager installs (shell layer) -------------------------------------------------- Add a shared retry() helper in utils.sh (3 attempts, 15s delay) and wrap the network-touching install/update commands in entrypoint.sh, build_pxf.sh, and build_cloudberrry.sh: * sudo apt update / sudo apt install / sudo apt-get install * sudo dnf install Local file installs (rpm -Uvh $file, dpkg -i $file) are left as-is. 2. Cache external artifacts across CI runs (workflow layer) ------------------------------------------------------------- * Maven local repository (~/.m2/repository) — cached with a key derived from automation/pom.xml hash, restored across three job matrices: - pxf-test (container mvn) - pxf-test-rocky9 (container mvn) - pxf-testcontainer-test (host mvn) Container jobs bind-mount ${GITHUB_WORKSPACE}/.m2 into the pxf-cbdb-dev container as /home/gpadmin/.m2 so both mvn invocations share one cache. Once warm, Maven Central + top-q.co.il are never contacted on reruns. * MSSQL testcontainer image (mcr.microsoft.com/mssql/server:2019-latest) — saved as a tarball and cached. On cache miss, retries the docker pull 3 times with backoff before failing. Only enabled for the pxf-jdbc TC group (the only consumer). All changes are scoped to CI infrastructure. No non-official Maven mirrors are configured (Apache release policy). Cache keys are branch- scoped by GHA automatically, so PR reruns immediately benefit while main-branch cache stays isolated. Related to apache#17. * ci: raise HDFS DataNode startup wait to 180s Test PXF Rocky9 - smoke intermittently fails with 'DataNode not available after 90s' in entrypoint.sh's wait_for_datanode helper. The smoke test is the first test group run against a freshly booted pxf-cbdb-dev container and starts the entire Hadoop/Hive/HBase stack cold — NameNode, DataNode, Hive metastore, HiveServer2, HBase master/regionserver — all competing for CPU and I/O on a shared GHA runner. DataNode registration path breakdown: 1. JVM cold start (class load + JIT + GC warmup) 2. DataNode scans /data/dn/dfs/data for existing block metadata 3. DataNode registers with NameNode (sends block report) 4. NameNode ACKs, exits safe mode, marks DataNode Live 5. 'hdfs dfsadmin -report' shows 'Live datanodes: 1' On healthy GHA runners this cycle completes in 20-40s. Under I/O contention (overlay2 file I/O slowness + noisy-neighbor CPU starvation) it can stretch to 90-180s. The prior 90s/attempt * 2 attempts = 180s total budget is on the edge and intermittently insufficient. Raise per-attempt wait from 90s to 180s (max_attempts unchanged at 2, total budget 360s). Healthy runs are unaffected — the loop exits as soon as DataNode reports Live. Only the tail latency budget grows. Related to apache#17.
* CI: add PXF convenience package build workflow Add a new GitHub Actions workflow to build DEB/RPM convenience packages from ASF-approved Apache Cloudberry PXF source release tarballs, and test them against Cloudberry built from its official source release. * Bring Rocky10 / Ubuntu 24.04 support back
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.21.0 to 0.52.0. - [Commits](golang/crypto@v0.21.0...v0.52.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.52.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…8 in /server (apache#138) * Bump org.apache.tomcat.embed:tomcat-embed-core in /server Bumps org.apache.tomcat.embed:tomcat-embed-core from 9.0.117 to 9.0.118. --- updated-dependencies: - dependency-name: org.apache.tomcat.embed:tomcat-embed-core dependency-version: 9.0.118 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update server/build.gradle --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dianjin Wang <wangdianjin@gmail.com>
In apache#136 it is planned to add 20+ Github Action Steps. And we agreed (apache#113) that we should keep number of tests we run under control. This PR reduces amount of test steps we run: 1. Remove `sanity` tests group (few tests here) 2. Remove `jdbc` test group (no tests in the group) 3. Run 'features' and 'gpdb' tests together - test groups shares 80% of the tests: | Group | Tests count | |--------|--------| | `features` total | 329 | | `gpdb` total | 321 | | `features` & `gpdb` | 264 | | `features` only | 65 | | `gpdb` only | 57 |
* CI: introduce maximum length for test report messages * truncate in table reporting
### Add `pxf_stat_activity` Add `pxf_stat_activity` to help DBAs find out what happening inside PXF server. ``` postgres=# select * from pxf_stat_activity ; -[ RECORD 1 ]-+--------------------------------------------------------- segment_id | 1 session_id | 57997 command_count | 8 xid | 57997-0000000008 operation | READ_BRIDGE usename | gpadmin server | ostinru-test profile | s3:parquet schema_name | public table_name | test_blob_read data_source | ostinru-test/topics/cuscom.email.created.v1/2026-05-29/* query_start | 2026-06-23 16:16:48.318+03 pxf_host | example.com ``` ### Add `pxf_cancel_backend` and `pxf_interrupt_backend` Add `pxf_cancel_backend` and `pxf_interrupt_backend` to allow killing running queries. ### Update tests 1. Update pxfExtension tests - use testconatiners. 2. Explicitly define extension version. So, there is no need in sql file manipulations.
Member
Author
|
Once #136 is merged, we need also cherry-pick it to this PR. |
* CI: add Rocky 10 support
Mirror the existing Rocky 9 lane for Rocky 10 so the two distros run the
same jobs with the same test scope:
- build-cloudberry-rpm-rocky10 builds the Cloudberry RPM on the Rocky 10
build image and publishes it as its own artifact/cache.
- build-docker-images-rocky10 builds the singlecluster image.
- pxf-test-rocky10 runs the same 16 test groups as pxf-test-rocky9.
- rocky10 is added to the testcontainer image matrix and to the
testcontainer test matrix.
- test-summary gains a Rocky 10 section.
The Rocky 10 jobs are derived from the Rocky 9 ones by substituting the
distro identifiers only, so the two matrices cannot drift apart.
RHEL/Rocky 10 dropped OpenJDK 8 and 11 from its repositories (only 21 and
25 remain), but Hadoop 3.1.2 still needs JDK 8 and the PXF server builds
with JDK 11. Both images therefore pull Temurin 8/11 from the Adoptium
repository on 10+ and expose them under the /usr/lib/jvm/java-1.8.0-openjdk
and /usr/lib/jvm/java-11-openjdk names the CI scripts already resolve
JAVA_HADOOP and JAVA_BUILD to, so no script needs a Rocky 10 special case.
The Adoptium package layout has changed names across releases, so the JDK
home is discovered with rpm -ql rather than hardcoded, and the resulting
symlinks are verified at image build time. Older releases keep using the
distro OpenJDK packages unchanged.
For the same reason the shared entrypoint no longer asks dnf for the
OpenJDK 8/11 packages on 10+ -- the image already provides them.
* CI: disable rpath QA when building the Rocky 10 Cloudberry RPM
RHEL/Rocky 10 runs check-rpaths as part of the default %install
post-processing chain, which RHEL 9 never invokes. Cloudberry links its
libraries with a RUNPATH into $INSTALL_PREFIX/lib on purpose, and the
plpython3 modules additionally carry /usr/lib64 and /ext/python/lib, so
the check aborts the build:
ERROR 0002: file '.../lib/postgresql/hstore.so' contains an invalid
runpath '/usr/local/cloudberry-db/lib'
ERROR 0001: file '.../lib/postgresql/plpython3.so' contains a standard
runpath '/usr/lib64'
error: Bad exit status from /var/tmp/rpm-tmp.X7bvEp (%install)
This RPM only ever installs Cloudberry into a CI container and is never
published, so set QA_RPATHS to switch the rpath QA off entirely, which
leaves the Rocky 10 lane behaving like the Rocky 9 one.
Verified against the apache/incubator-cloudberry:cbdb-build-rocky10-latest
image with a REL_2_STABLE checkout: the build script fails at check-rpaths
without QA_RPATHS and produces apache-cloudberry-db-99.0.0-1.el10.x86_64.rpm
with it, check-rpaths still running but reporting no errors. The same probe
on the Rocky 9 image shows check-rpaths is never invoked there.
* CI: stop naming ssh-dss in the sshd config on Rocky 10
OpenSSH 9.8 removed DSA support entirely and RHEL/Rocky 10 ships 9.9, so
the key types the automation sshd config asks for no longer parse there:
/etc/ssh/sshd_config.d/pxf-automation.conf line 2: Bad key types
'+ssh-rsa,ssh-dss'.
sshd rejects the whole file and exits, which took down every Rocky 10 job
in both lanes -- the 16 pxf-test-rocky10 jobs failed in "Start Services",
and the 6 rocky10 testcontainer jobs failed with "entrypoint.sh failed
(exit 1)" out of PXFCloudberryContainer, since both entrypoints write the
same file.
Ask for ssh-dss only when the local sshd still knows it, so Ubuntu and
Rocky 9 keep the exact configuration they have today while Rocky 10 gets
ssh-rsa alone. ssh-keygen -A does not generate DSA host keys on either
release, so nothing is lost. The KEX line is unchanged: Rocky 10 still
offers the group1-sha1 / group-exchange-sha1 / group14-sha1 algorithms
the Ganymed SSH-2 client needs.
Also validate the configuration with sshd -t before starting and echo the
log on failure. This failure produced nothing but "Failed to start sshd,
check /tmp/sshd.log" in CI, with the log never printed and the parse error
never captured.
Verified on apache/incubator-cloudberry:cbdb-build-rocky10-latest: sshd -t
rejects '+ssh-rsa,ssh-dss' and accepts '+ssh-rsa', and a full run of the
shared entrypoint against that image now reports "sshd key types: ssh-rsa"
followed by "sshd is running on port 22", then installs Cloudberry and
brings up the demo cluster. The same probe on the Rocky 9 image shows
ssh-dss is still supported and the config is unchanged there.
* CI: make the automation sshd drop-in win over the crypto policy
The automation framework talks SSH through Ganymed SSH-2 (ch.ethz.ssh2),
which only knows the ssh-rsa / ssh-dss host key algorithms. On Rocky 10
every Hadoop-dependent test group failed to reach the cluster:
java.io.IOException: There was a problem while connecting to localhost:22
at ch.ethz.ssh2.Connection.connect(Connection.java:805)
at ...cluster.SingleCluster.init(SingleCluster.java:37)
sshd was running and the KEX algorithms negotiated fine; the handshake
died on the host key:
debug1: kex: algorithm: diffie-hellman-group14-sha1
debug1: kex: host key algorithm: (no match)
Unable to negotiate: no matching host key type found.
Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
ssh-rsa was missing from the offer even though pxf-automation.conf asks
for it, because sshd keeps the FIRST value it obtains for a keyword and
the drop-ins are read in name order:
40-redhat-crypto-policies.conf -> Include .../opensshserver.config
50-redhat.conf
pxf-automation.conf -> HostKeyAlgorithms +ssh-rsa (ignored)
So the crypto policy always won and our directive never applied. Rocky 9
did not notice because its image has no 40-redhat-crypto-policies.conf and
its LEGACY policy still carries ssh-rsa; Rocky 10 removed SHA-1 from every
policy, leaving nothing to enable it.
Write the file as 01-pxf-automation.conf so it is read before the crypto
policy include, and drop any file left under the old name.
Verified on both build images by replaying setup_ssh and connecting with
exactly the algorithm set Ganymed offers: ssh-rsa appears in sshd -T on
both, and the connection succeeds on Rocky 9 (ssh-rsa,ssh-dss) as well as
Rocky 10 (ssh-rsa). Confirmed in the live rocky10 container that the
rename turns the failing handshake into a successful gpadmin login.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #ISSUE_Number
Change logs
Contributor's checklist
Here are some reminders before you submit your pull request: