diff --git a/.github/workflows/pd-store-ci.yml b/.github/workflows/pd-store-ci.yml index 1a6825e7e4..fce7168e05 100644 --- a/.github/workflows/pd-store-ci.yml +++ b/.github/workflows/pd-store-ci.yml @@ -134,6 +134,14 @@ jobs: done echo "can_run=true" >> "$GITHUB_OUTPUT" + - name: Run PD docker entrypoint secret override tests + run: | + $TRAVIS_DIR/test-pd-docker-entrypoint.sh + + - name: Check every shipped PD config carries the REST hardening + run: | + $TRAVIS_DIR/test-pd-shipped-config.sh + - name: Run start-hugegraph-pd.sh foreground mode tests if: steps.pd-preflight.outputs.can_run == 'true' run: | diff --git a/.gitignore b/.gitignore index e1f546810f..edfe89f7d2 100644 --- a/.gitignore +++ b/.gitignore @@ -44,6 +44,8 @@ build/ .env.test.local .env.production.local docker/.env +# generated by docker/set-hubble-pd-password.sh, carries the PD REST secret +docker/conf/hubble/*.local.properties* *.orig *.rej diff --git a/docker/README.md b/docker/README.md index 6fcfdee425..081fa36dc8 100644 --- a/docker/README.md +++ b/docker/README.md @@ -39,33 +39,42 @@ contains a single quote or newline. echo ".env already exists; edit it instead of overwriting it" >&2 exit 1 } - printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\n" \ - 'replace-with-your-password' "${jwt_secret}" > .env + pd_secret="$(openssl rand -hex 24)" + printf "HUGEGRAPH_ADMIN_PASSWORD='%s'\nHUGEGRAPH_AUTH_TOKEN_SECRET='%s'\nHG_PD_AUTH_SECRET_KEY='%s'\n" \ + 'replace-with-your-password' "${jwt_secret}" "${pd_secret}" > .env + # Hubble reads the PD secret from a file, not from .env: generate the untracked properties files the HStore topologies mount. + HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore + HG_PD_AUTH_SECRET_KEY="${pd_secret}" ./set-hubble-pd-password.sh hstore-ha ) ``` -Do not commit `.env`. Keeping the same JWT secret preserves authentication -tokens when containers are recreated. For authenticated topologies with -multiple Server replicas, all replicas receive this same secret. The HA -topology fails fast if authentication is enabled without this shared secret. +Do not commit `.env` or `conf/hubble/*.local.properties`; both are in `.gitignore`. Keeping the same JWT secret preserves authentication tokens when containers are recreated. For authenticated topologies with multiple Server replicas, all replicas receive this same secret. The HA topology fails fast if authentication is enabled without this shared secret. -A non-empty `HUGEGRAPH_ADMIN_PASSWORD` enables Server authentication, and -Hubble detects that mode automatically. Omitting the variable or setting it to -an empty value disables authentication. Auth-off is only suitable for a -trusted local environment; never expose it to a public or untrusted network. -Hubble listens on host loopback by default. Set `HUBBLE_PUBLISH_HOST` only -behind an HTTPS reverse proxy and trusted network controls. +A non-empty `HUGEGRAPH_ADMIN_PASSWORD` enables Server authentication, and Hubble detects that mode automatically. Omitting the variable or setting it to an empty value disables authentication. Auth-off is only suitable for a trusted local environment; never expose it to a public or untrusted network. Hubble listens on host loopback by default. Set `HUBBLE_PUBLISH_HOST` only behind an HTTPS reverse proxy and trusted network controls. -`HUGEGRAPH_ADMIN_PASSWORD` initializes the built-in `admin` account on its -first authenticated startup. Changing `.env` does not rotate an existing -administrator password; use the HugeGraph user API for credential changes. +`HUGEGRAPH_ADMIN_PASSWORD` initializes the built-in `admin` account on its first authenticated startup. Changing `.env` does not rotate an existing administrator password; use the HugeGraph user API for credential changes. -For the verification commands below, set the password in your current shell: +For the verification commands below, load `.env` into your current shell and set the password: ```bash +set -a; . ./.env; set +a ADMIN_PASSWORD='the-same-password-used-in-.env' ``` +The PD REST API (port 8620, HStore topologies only) requires HTTP Basic auth (`hg:${HG_PD_AUTH_SECRET_KEY}`) for all endpoints except health/readiness probes (`/v1/health`, `/v1/ready`). `HG_PD_AUTH_SECRET_KEY` is shared across PD, Server (`bin/wait-storage.sh`), and Hubble (`conf/hubble/*.local.properties` generated by `./set-hubble-pd-password.sh`). + +Verify registered stores: + +```bash +curl -u "hg:${HG_PD_AUTH_SECRET_KEY}" http://localhost:8620/v1/stores +``` + +To regenerate Hubble configuration after modifying `.env`: + +```bash +./set-hubble-pd-password.sh hstore # or hstore-ha +``` + ### Standalone This is the recommended quickstart. @@ -181,8 +190,7 @@ Status: docker compose -f docker-compose-3pd-3store-3server.yml ps ``` -Verify all published PD, Store, and Server endpoints, Server authentication, -and Hubble: +Verify all published PD, Store, and Server endpoints, Server authentication, and Hubble: ```bash for port in 8620 8621 8622; do @@ -202,34 +210,11 @@ done curl -fsS http://localhost:8088/about ``` -PD answers two unauthenticated probe endpoints. `/v1/health` is liveness only: -it returns `200` as soon as the REST listener is up, even when the PD has no -raft leader. `/v1/ready` returns `200` only while the PD sees a raft leader, -and `503` otherwise. Each PD answers for itself: a single PD elects itself, and -in a three-PD group the two that can reach each other elect a leader and turn -ready, while a partitioned third keeps answering `503` until it sees that -leader. - -The healthchecks in these files still gate on `/v1/health`, because -`/v1/ready` ships from the next release onwards while the files run published -images. Two things to know before pointing them at readiness: - -- Match on the body, not the status code. As of 1.7.0 PD answers `200` with - `{"status":-1,"error":"Unauthorized!"}` on every path its auth interceptor - does not exclude, a path that does not exist included, so a status-only - probe reads a PD too old to have `/v1/ready` as ready. The body match holds - whichever status a refusal carries. Gate with - `curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'` instead. -- Pin `HUGEGRAPH_VERSION` to a release that carries the endpoint, or build the - images from source with `docker-compose.dev.yml`. - -The `HEALTHCHECK` baked into `hugegraph-pd/Dockerfile` is `/v1/health` as well. -Both compose files override it, so it governs `docker run` and anything else -inheriting the image probe, and those keep reading a PD without a quorum as -healthy. +PD answers two unauthenticated probe endpoints: `/v1/health` for liveness (returns 200 once the REST listener is up, regardless of raft state), and `/v1/ready` for readiness (returns 200 only when PD sees a raft leader, 503 otherwise). -Open `http://localhost:8088` and sign in as `admin` with the password from -`.env`. +Compose healthchecks currently gate on `/v1/health` for compatibility with published images. When targeting readiness on newer releases or source builds (`docker-compose.dev.yml`), match on the response body (`curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'`). + +Open `http://localhost:8088` and sign in as `admin` with the password from `.env`. Stop containers while keeping them: @@ -265,9 +250,7 @@ HUBBLE_IMAGE=hugegraph/hubble:latest \ docker compose -f docker-compose.yml up -d ``` -The Hubble `latest` image is expected to work with HugeGraph Server 1.7 and -Server `latest`; compatibility with versions older than 1.7 is not promised. -Pin immutable image references when reproducibility is required. +The Hubble `latest` image is expected to work with HugeGraph Server 1.7 and Server `latest`; compatibility with versions older than 1.7 is not promised. Pin immutable image references when reproducibility is required. ### Server startup timeout @@ -348,16 +331,15 @@ docker compose -f docker-compose-hstore.yml up -d --wait ### Hubble configuration -The three small files under `conf/hubble/` contain only topology-specific -discovery settings and container paths: +The three small files under `conf/hubble/` contain only topology-specific discovery settings, the PD REST credential (`operations.pd.username` and `operations.pd.password`, which must match PD's `auth.secret-key`), and container paths: - `conf/hubble/standalone.properties` uses direct Server mode. -- `conf/hubble/hstore.properties` uses one PD and one Store REST target. -- `conf/hubble/hstore-ha.properties` uses all three PD peers and all three - allowed Store REST targets. +- `conf/hubble/hstore.properties.example` uses one PD and one Store REST target. +- `conf/hubble/hstore-ha.properties.example` uses all three PD peers and all three allowed Store REST targets. + +The two HStore topologies mount the generated `*.local.properties` next to these examples (see `set-hubble-pd-password.sh`), never the examples themselves, so the PD secret stays out of tracked files. -Hubble detects Server authentication through the Server API. Do not add an -`auth.enabled` property or duplicate auth-on/auth-off configurations. +Hubble detects Server authentication through the Server API. Do not add an `auth.enabled` property or duplicate auth-on/auth-off configurations. ### Render and smoke checks @@ -367,12 +349,9 @@ Render every topology with auth-on inputs before submitting a change: bash test-compose.sh render ``` -The HA render is mandatory even when local resources are insufficient to start -its ten containers. +The HA render is mandatory even when local resources are insufficient to start its ten containers. -Run focused auth-on smoke checks for standalone and minimal HStore with the -corresponding `up -d --wait`, status, authentication, Hubble `/about`, and -`down -v` commands from the Users section: +Run focused auth-on smoke checks for standalone and minimal HStore with the corresponding `up -d --wait`, status, authentication, Hubble `/about`, and `down -v` commands from the Users section: ```bash bash test-compose.sh smoke @@ -384,6 +363,4 @@ Run the required local auth-off checks separately: bash test-compose.sh smoke-auth-off ``` -The auth-off mode is intentionally excluded from the default CI matrix and must -remain on a trusted local machine. Both smoke modes remove only the isolated -Compose projects and volumes that they create. +The auth-off mode is intentionally excluded from the default CI matrix and must remain on a trusted local machine. Both smoke modes remove only the isolated Compose projects and volumes that they create. diff --git a/docker/conf/hubble/hstore-ha.properties b/docker/conf/hubble/hstore-ha.properties.example similarity index 76% rename from docker/conf/hubble/hstore-ha.properties rename to docker/conf/hubble/hstore-ha.properties.example index a50c52c96f..82c32248d9 100644 --- a/docker/conf/hubble/hstore-ha.properties +++ b/docker/conf/hubble/hstore-ha.properties.example @@ -20,6 +20,12 @@ pd.enabled=true server.direct_url=http://server0:8080 pd.peers=pd0:8686,pd1:8686,pd2:8686 pd.server=pd0:8620 +# PD REST credential. The password must equal PD's auth.secret-key, which has +# no default. Do not edit this tracked example: docker/set-hubble-pd-password.sh +# generates the untracked .local.properties that Compose mounts, with the value +# from HG_PD_AUTH_SECRET_KEY in .env. +operations.pd.username=hubble +operations.pd.password= operations.store.allowed_targets=[http://store0:8520,http://store1:8520,http://store2:8520] upload_file.location=/hubble/data/upload-files dashboard.address= diff --git a/docker/conf/hubble/hstore.properties b/docker/conf/hubble/hstore.properties.example similarity index 75% rename from docker/conf/hubble/hstore.properties rename to docker/conf/hubble/hstore.properties.example index 5de43c212f..271ef9039f 100644 --- a/docker/conf/hubble/hstore.properties +++ b/docker/conf/hubble/hstore.properties.example @@ -20,6 +20,12 @@ pd.enabled=true server.direct_url=http://server:8080 pd.peers=pd:8686 pd.server=pd:8620 +# PD REST credential. The password must equal PD's auth.secret-key, which has +# no default. Do not edit this tracked example: docker/set-hubble-pd-password.sh +# generates the untracked .local.properties that Compose mounts, with the value +# from HG_PD_AUTH_SECRET_KEY in .env. +operations.pd.username=hubble +operations.pd.password= operations.store.allowed_targets=[http://store:8520] upload_file.location=/hubble/data/upload-files dashboard.address= diff --git a/docker/docker-compose-3pd-3store-3server.yml b/docker/docker-compose-3pd-3store-3server.yml index 9c066242e4..999f2126fb 100644 --- a/docker/docker-compose-3pd-3store-3server.yml +++ b/docker/docker-compose-3pd-3store-3server.yml @@ -73,6 +73,8 @@ x-server-environment: &server-environment HG_SERVER_STARTUP_TIMEOUT_S: ${HG_SERVER_STARTUP_TIMEOUT_S-120} HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-} PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-} + # bin/wait-storage.sh polls the PD REST API, so it needs the same secret + PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} x-server-common: &server-common image: hugegraph/server:${HUGEGRAPH_VERSION:-latest} @@ -110,6 +112,7 @@ services: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data HG_PD_INITIAL_STORE_COUNT: 3 + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: ["8620:8620", "8686:8686"] volumes: - hg-pd0-data:/hugegraph-pd/pd_data @@ -128,6 +131,7 @@ services: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data HG_PD_INITIAL_STORE_COUNT: 3 + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: ["8621:8620", "8687:8686"] volumes: - hg-pd1-data:/hugegraph-pd/pd_data @@ -146,6 +150,7 @@ services: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data HG_PD_INITIAL_STORE_COUNT: 3 + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: ["8622:8620", "8688:8686"] volumes: - hg-pd2-data:/hugegraph-pd/pd_data @@ -239,7 +244,15 @@ services: - "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088" volumes: - hubble-data:/hubble/data - - ./conf/hubble/hstore-ha.properties:/hubble/conf/hugegraph-hubble.properties:ro + - type: bind + source: ./conf/hubble/hstore-ha.local.properties + target: /hubble/conf/hugegraph-hubble.properties + read_only: true + bind: + # The file is generated by set-hubble-pd-password.sh and is + # gitignored. Without this, Docker would create an empty + # directory at that path and Hubble would boot unconfigured. + create_host_path: false healthcheck: test: - CMD-SHELL diff --git a/docker/docker-compose-hstore.yml b/docker/docker-compose-hstore.yml index f6f20bd896..0fcc3697f6 100644 --- a/docker/docker-compose-hstore.yml +++ b/docker/docker-compose-hstore.yml @@ -39,6 +39,7 @@ services: HG_PD_RAFT_PEERS_LIST: pd:8610 HG_PD_INITIAL_STORE_LIST: store:8500 HG_PD_DATA_PATH: /hugegraph-pd/pd_data + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: - "8620:8620" volumes: @@ -97,6 +98,8 @@ services: HG_SERVER_STARTUP_TIMEOUT_S: ${HG_SERVER_STARTUP_TIMEOUT_S-120} HG_SERVER_AUTH_TOKEN_SECRET: ${HUGEGRAPH_AUTH_TOKEN_SECRET:-} PASSWORD: ${HUGEGRAPH_ADMIN_PASSWORD:-} + # bin/wait-storage.sh polls the PD REST API, so it needs the same secret + PD_AUTH_PASSWORD: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md} ports: - "8080:8080" healthcheck: @@ -120,7 +123,15 @@ services: - "${HUBBLE_PUBLISH_HOST:-127.0.0.1}:8088:8088" volumes: - hubble-data:/hubble/data - - ./conf/hubble/hstore.properties:/hubble/conf/hugegraph-hubble.properties:ro + - type: bind + source: ./conf/hubble/hstore.local.properties + target: /hubble/conf/hugegraph-hubble.properties + read_only: true + bind: + # The file is generated by set-hubble-pd-password.sh and is + # gitignored. Without this, Docker would create an empty + # directory at that path and Hubble would boot unconfigured. + create_host_path: false healthcheck: test: - CMD-SHELL diff --git a/docker/set-hubble-pd-password.sh b/docker/set-hubble-pd-password.sh new file mode 100755 index 0000000000..73d3f18a63 --- /dev/null +++ b/docker/set-hubble-pd-password.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Generate the Hubble properties file a Compose topology mounts, with PD's +# REST secret written in as operations.pd.password. +# +# usage: set-hubble-pd-password.sh [secret] +# +# Reads conf/hubble/.properties.example (tracked) and writes +# conf/hubble/.local.properties (ignored by git), so the secret never +# lands in a tracked file. The secret defaults to $HG_PD_AUTH_SECRET_KEY. The +# value never goes through a sed replacement, where & # and backslash are +# special, and backslashes are doubled for the .properties format. Run this +# before `docker compose up`: the bind pins create_host_path: false, so a +# missing target makes Compose refuse to start. +# +# The secret must be printable ASCII. PD compares it as UTF-8 bytes +# (Authentication.verifySecret), while Hubble reads this file through +# commons-configuration2, whose DEFAULT_ENCODING is ISO-8859-1, so a non-ASCII +# secret decodes to different bytes on the two sides and gives a permanent 401 +# with no diagnostic anywhere. The README recipe generates hex, which is safe. +set -euo pipefail + +name=${1:?usage: $0 [secret]} +secret=${2:-${HG_PD_AUTH_SECRET_KEY:-}} +dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/conf/hubble" +example="${dir}/${name}.properties.example" +out="${dir}/${name}.local.properties" + +[[ -f "$example" ]] || { echo "no such topology: ${name} (expected ${example})" >&2; exit 1; } +[[ -n "$secret" ]] || { echo "secret is empty; load .env first (set -a; . ./.env; set +a)" >&2; exit 1; } +case "$secret" in + *$'\n'*|*$'\r'*) echo "secret contains a line break, which a .properties value cannot hold" >&2; exit 1 ;; +esac +# LC_ALL=C so the range is ordinal and the walk byte-wise: under the caller's +# collation a non-ASCII character can sort inside \x20-\x7e and slip through. +is_printable_ascii() { + local LC_ALL=C + case "$1" in + *[!$'\x20'-$'\x7e']*) return 1 ;; + esac +} +is_printable_ascii "$secret" || { + echo "secret must be printable ASCII: Hubble reads .properties as ISO-8859-1, PD compares as UTF-8" >&2 + exit 1 +} + +escaped=${secret//\\/\\\\} +# java.util.Properties skips whitespace between the separator and the value, so +# a secret that starts with a space would reach Hubble shortened while PD and +# the Server kept the original. A backslash before it keeps that first byte. +case "$escaped" in + [$' \t']*) escaped="\\${escaped}" ;; +esac +tmp=$(mktemp "${out}.XXXXXX") +trap 'rm -f "$tmp"' EXIT +{ + printf '# Generated from %s by set-hubble-pd-password.sh; not tracked by git.\n' "$(basename "$example")" + grep -v '^operations\.pd\.password=' "$example" || true + printf 'operations.pd.password=%s\n' "$escaped" +} > "$tmp" +# Hubble runs unprivileged and the mount is read-only, so the file must be world-readable +chmod 644 "$tmp" +mv "$tmp" "$out" +trap - EXIT +echo "wrote ${out}" diff --git a/docker/test-compose.sh b/docker/test-compose.sh index 7673036481..54418d413a 100644 --- a/docker/test-compose.sh +++ b/docker/test-compose.sh @@ -18,9 +18,16 @@ set -Eeuo pipefail +# Most assertions below are a bare `jq -e ... >/dev/null`, so `set -e` used to +# end the run with no output at all and a CI log that said only "exit code 1". +# Name the command and line that failed instead; the trap fires once per frame, +# so the innermost assertion comes first and its callers follow. +trap 'echo "FAILED at ${BASH_SOURCE[0]}:${LINENO}: ${BASH_COMMAND}" >&2' ERR + DOCKER_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" PASSWORD="ci-compose-password" SECRET="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef" +PD_SECRET="ci-compose-pd-secret" VERSION="ci-version" RENDER_HUBBLE_IMAGE="example.invalid/hugegraph/hubble:ci" DATASOURCE="jdbc:h2:file:/hubble/data/hubble;DB_CLOSE_ON_EXIT=FALSE" @@ -41,6 +48,7 @@ compose_auth() { HUBBLE_IMAGE="${RENDER_HUBBLE_IMAGE}" \ HUGEGRAPH_ADMIN_PASSWORD="${PASSWORD}" \ HUGEGRAPH_AUTH_TOKEN_SECRET="${SECRET}" \ + HG_PD_AUTH_SECRET_KEY="${PD_SECRET}" \ docker compose "$@" } @@ -144,6 +152,36 @@ assert_hubble() { ' "${rendered}" >/dev/null } +# The two *.local.properties files are generated by set-hubble-pd-password.sh +# and gitignored. Their binds must disable host-path creation, so a first +# `docker compose up` that skipped the generator fails instead of getting an +# empty directory mounted over Hubble's config. +# +# This reads the Compose file rather than the render on purpose. compose-go +# tags ServiceVolumeBind.CreateHostPath `json:"create_host_path,omitempty"` +# (checked in compose-go v2.7.0, the version inside the Compose 2.38.2 that +# ubuntu-24.04 runners ship), and omitempty drops a false bool, so +# `config --format json` emits the same `"bind": {}` for the long syntax with +# create_host_path disabled and for the short syntax this pins away from. The +# render simply cannot carry this contract on that version. +assert_hubble_bind_pinned() { + local file="$1" properties="$2" + grep -Fq "source: ./conf/hubble/${properties}" "${file}" || { + echo "${file}: the ${properties} mount is not in long bind syntax" >&2 + return 1 + } + grep -Fq "create_host_path: false" "${file}" || { + echo "${file}: the ${properties} bind does not disable" \ + "create_host_path" >&2 + return 1 + } + if grep -Fq "./conf/hubble/${properties}:/hubble/conf" "${file}"; then + echo "${file}: the ${properties} mount is back on the short bind" \ + "syntax, which lets Docker create an empty directory there" >&2 + return 1 + fi +} + assert_standalone() { local rendered="$1" assert_common "${rendered}" \ @@ -170,7 +208,9 @@ assert_hstore() { assert_common "${rendered}" \ '["hubble","pd","server","store"]' \ '["hubble-data","pd-data","store-data"]' - assert_hubble "${rendered}" "hstore.properties" server + assert_hubble "${rendered}" "hstore.local.properties" server + assert_hubble_bind_pinned "${DOCKER_DIR}/docker-compose-hstore.yml" \ + "hstore.local.properties" jq -e ' .services.pd.image == "hugegraph/pd:ci-version" and .services.store.image == "hugegraph/store:ci-version" and @@ -192,9 +232,9 @@ assert_hstore() { .source == "store-data" and .target == "/hugegraph-store/storage") ' "${rendered}" >/dev/null - assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.properties" \ + assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.local.properties" \ "pd.peers=pd:8686" - assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.properties" \ + assert_file_property "${DOCKER_DIR}/conf/hubble/hstore.local.properties" \ "operations.store.allowed_targets=[http://store:8520]" } @@ -203,8 +243,11 @@ assert_ha() { assert_common "${rendered}" \ '["hubble","pd0","pd1","pd2","server0","server1","server2","store0","store1","store2"]' \ '["hg-pd0-data","hg-pd1-data","hg-pd2-data","hg-store0-data","hg-store1-data","hg-store2-data","hubble-data"]' - assert_hubble "${rendered}" "hstore-ha.properties" \ + assert_hubble "${rendered}" "hstore-ha.local.properties" \ server0 server1 server2 + assert_hubble_bind_pinned \ + "${DOCKER_DIR}/docker-compose-3pd-3store-3server.yml" \ + "hstore-ha.local.properties" jq -e ' all([.services.pd0, .services.pd1, .services.pd2][]; .image == "hugegraph/pd:ci-version" and @@ -237,9 +280,9 @@ assert_ha() { .healthcheck.retries == 30 and .healthcheck.start_period == "1m0s") ' "${rendered}" >/dev/null - assert_file_property "${DOCKER_DIR}/conf/hubble/hstore-ha.properties" \ + assert_file_property "${DOCKER_DIR}/conf/hubble/hstore-ha.local.properties" \ "pd.peers=pd0:8686,pd1:8686,pd2:8686" - assert_file_property "${DOCKER_DIR}/conf/hubble/hstore-ha.properties" \ + assert_file_property "${DOCKER_DIR}/conf/hubble/hstore-ha.local.properties" \ "operations.store.allowed_targets=[http://store0:8520,http://store1:8520,http://store2:8520]" } @@ -274,12 +317,78 @@ cleanup() { if [[ -n "${ACTIVE_PROJECT}" ]]; then compose_active down -v --remove-orphans >/dev/null 2>&1 || true fi + restore_hubble_configs [[ -z "${RENDER_DIR}" ]] || rm -rf "${RENDER_DIR}" } +# The HStore topologies mount conf/hubble/.local.properties, which is +# generated and untracked. Generate both with the CI secret before any render +# or `up`; the binds pin create_host_path: false, so a missing file makes +# Compose refuse to start. A developer's own local files are put back +# afterwards. +HUBBLE_BACKUP_DIR="" +prepare_hubble_configs() { + local name + HUBBLE_BACKUP_DIR="$(mktemp -d)" + for name in hstore hstore-ha; do + local f="${DOCKER_DIR}/conf/hubble/${name}.local.properties" + [[ ! -f "${f}" ]] || cp -p "${f}" "${HUBBLE_BACKUP_DIR}/${name}.local.properties" + "${DOCKER_DIR}/set-hubble-pd-password.sh" "${name}" "${PD_SECRET}" >/dev/null + done +} +restore_hubble_configs() { + [[ -n "${HUBBLE_BACKUP_DIR}" ]] || return 0 + local name + for name in hstore hstore-ha; do + local f="${DOCKER_DIR}/conf/hubble/${name}.local.properties" + if [[ -f "${HUBBLE_BACKUP_DIR}/${name}.local.properties" ]]; then + cp -p "${HUBBLE_BACKUP_DIR}/${name}.local.properties" "${f}" + else + rm -f "${f}" + fi + done + rm -rf "${HUBBLE_BACKUP_DIR}" + HUBBLE_BACKUP_DIR="" +} + +# set-hubble-pd-password.sh must survive the characters a sed replacement +# would mangle, keep the rest of the example, produce a world-readable file +# for the read-only mount, and refuse an empty secret or unknown topology. +hubble_password_helper_check() { + local f="${DOCKER_DIR}/conf/hubble/hstore.local.properties" + "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore 'a&b#c\d' >/dev/null + local line mode + line=$(grep '^operations\.pd\.password=' "${f}") + mode=$(stat -c '%a' "${f}" 2>/dev/null || stat -f '%Lp' "${f}") + [[ "${line}" == 'operations.pd.password=a&b#c\\d' ]] || { + echo "set-hubble-pd-password.sh mangled the secret: ${line}" >&2; exit 1; } + [[ "${mode}" == "644" ]] || { + echo "set-hubble-pd-password.sh wrote mode ${mode}, Hubble could not read it" >&2; exit 1; } + grep -q '^pd.server=pd:8620$' "${f}" || { + echo "set-hubble-pd-password.sh dropped the example's other properties" >&2; exit 1; } + ! "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore '' 2>/dev/null || { + echo "set-hubble-pd-password.sh accepted an empty secret" >&2; exit 1; } + ! "${DOCKER_DIR}/set-hubble-pd-password.sh" nope 'x' 2>/dev/null || { + echo "set-hubble-pd-password.sh accepted an unknown topology" >&2; exit 1; } + # Hubble decodes this file as ISO-8859-1 and PD compares UTF-8 bytes, so a + # non-ASCII secret would be a permanent 401 with nothing in any log. + ! "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore 'pässwörd' 2>/dev/null || { + echo "set-hubble-pd-password.sh accepted a non-ASCII secret" >&2; exit 1; } + # java.util.Properties drops whitespace after the separator, so a leading + # space has to survive as an escape or Hubble reads a shortened secret. + "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore ' lead' >/dev/null + line=$(grep '^operations\.pd\.password=' "${f}") + [[ "${line}" == 'operations.pd.password=\ lead' ]] || { + echo "set-hubble-pd-password.sh left a leading space unescaped: ${line}" >&2 + exit 1; } + # put the CI value back for the render/smoke that follows + "${DOCKER_DIR}/set-hubble-pd-password.sh" hstore "${PD_SECRET}" >/dev/null +} + run_render() { RENDER_DIR="$(mktemp -d)" trap cleanup EXIT INT TERM + prepare_hubble_configs render "${RENDER_DIR}/standalone.json" \ -f "${DOCKER_DIR}/docker-compose.yml" render "${RENDER_DIR}/hstore.json" \ @@ -331,6 +440,8 @@ run_render() { assert_startup_timeout "${RENDER_DIR}/hstore-empty.json" "" server assert_startup_timeout "${RENDER_DIR}/ha-empty.json" "" \ server0 server1 server2 + + hubble_password_helper_check echo "Compose render contracts passed" } @@ -340,6 +451,7 @@ compose_active() { HUBBLE_IMAGE="${HUBBLE_IMAGE:-hugegraph/hubble:latest}" \ HUGEGRAPH_ADMIN_PASSWORD="${PASSWORD}" \ HUGEGRAPH_AUTH_TOKEN_SECRET="${SECRET}" \ + HG_PD_AUTH_SECRET_KEY="${PD_SECRET}" \ COMPOSE_PROGRESS=plain \ docker compose -p "${ACTIVE_PROJECT}" "${ACTIVE_FILES[@]}" "$@" } @@ -442,6 +554,7 @@ smoke() { run_smoke() { trap cleanup EXIT INT TERM + prepare_hubble_configs smoke standalone false true "${DOCKER_DIR}/docker-compose.yml" smoke hstore true true "${DOCKER_DIR}/docker-compose-hstore.yml" } @@ -449,6 +562,7 @@ run_smoke() { run_smoke_auth_off() { PASSWORD="" trap cleanup EXIT INT TERM + prepare_hubble_configs smoke standalone-anon false false "${DOCKER_DIR}/docker-compose.yml" smoke hstore-anon true false "${DOCKER_DIR}/docker-compose-hstore.yml" } diff --git a/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template b/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template index 87229aabcf..55a1afcbdb 100644 --- a/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template +++ b/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template @@ -27,7 +27,10 @@ management: endpoints: web: exposure: - include: "*" + # Allowlist, not "*": actuator has its own handler mapping, which + # the REST auth interceptor is not attached to, so anything exposed + # here is anonymous on this port. This list is what bounds it. + include: "health,metrics,prometheus" logging: config: 'file:./conf/log4j2.xml' @@ -43,6 +46,14 @@ server: # REST service port number port : $REST_PORT$ +auth: + # Shared secret checked against the password of the Basic credential on every + # authenticated REST request. Required and deliberately empty: a secret in the + # source tree is published to everyone. Generate one per deployment and give + # every REST client the same value. While empty, PD refuses every + # authenticated REST request. + secret-key: + pd: # Storage path data-path: ./pd_data diff --git a/hugegraph-pd/README.md b/hugegraph-pd/README.md index c940dae4c9..5050025565 100644 --- a/hugegraph-pd/README.md +++ b/hugegraph-pd/README.md @@ -100,6 +100,7 @@ Key configuration file: `conf/application.yml` | `raft.address` | `127.0.0.1:8610` | Raft service address for this PD node | | `raft.peers-list` | `127.0.0.1:8610` | Comma-separated list of all PD nodes in the Raft cluster | | `pd.data-path` | `./pd_data` | Directory for storing PD metadata and Raft logs | +| `auth.secret-key` | none (required) | Password required by the REST API with an internal service name (`hg`, `store`, `hubble`, `vermeer`) via HTTP Basic auth. No default is shipped; generate one per deployment and configure every REST client (e.g. Hubble's `operations.pd.password`) with the same value | #### Single-Node Example @@ -157,11 +158,12 @@ For detailed configuration options and production tuning, see [Configuration Gui #### Docker Bridge Network Example -When running PD in Docker with bridge networking (e.g., `docker/docker-compose-3pd-3store-3server.yml`), configuration is injected via environment variables instead of editing `application.yml` directly. Container hostnames are used instead of IP addresses: +When running PD in Docker with bridge networking (e.g., `docker/docker-compose-3pd-3store-3server.yml`), configuration is injected via environment variables instead of editing `application.yml` directly. Container hostnames are used instead of IP addresses. `HG_PD_AUTH_SECRET_KEY` is required by the image and must be the same value on every PD node and every PD REST client; generate it once (`openssl rand -hex 24`) and keep it: **pd0** container: ```bash HG_PD_GRPC_HOST=pd0 +HG_PD_AUTH_SECRET_KEY= HG_PD_RAFT_ADDRESS=pd0:8610 HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610 HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500 @@ -170,6 +172,7 @@ HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500 **pd1** container: ```bash HG_PD_GRPC_HOST=pd1 +HG_PD_AUTH_SECRET_KEY= HG_PD_RAFT_ADDRESS=pd1:8610 HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610 HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500 @@ -178,6 +181,7 @@ HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500 **pd2** container: ```bash HG_PD_GRPC_HOST=pd2 +HG_PD_AUTH_SECRET_KEY= HG_PD_RAFT_ADDRESS=pd2:8610 HG_PD_RAFT_PEERS_LIST=pd0:8610,pd1:8610,pd2:8610 HG_PD_INITIAL_STORE_LIST=store0:8500,store1:8500,store2:8500 @@ -236,11 +240,15 @@ Build PD Docker image: # From project root docker build -f hugegraph-pd/Dockerfile -t hugegraph/pd:latest . +# Generate the REST secret once and keep it: every PD REST client needs this same value, and a new one silently breaks the clients already using the old one. Store it somewhere durable rather than only in this shell. +export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)" + # Run container docker run -d \ -p 8620:8620 \ -p 8686:8686 \ -p 8610:8610 \ + -e HG_PD_AUTH_SECRET_KEY="${HG_PD_AUTH_SECRET_KEY}" \ -e HG_PD_GRPC_HOST= \ -e HG_PD_RAFT_ADDRESS=:8610 \ -e HG_PD_RAFT_PEERS_LIST=:8610 \ @@ -280,6 +288,13 @@ docker/docker-compose-3pd-3store-3server.yml - Ensure low latency (<5ms) between PD nodes for Raft consensus - Open required ports: `8620` (REST), `8686` (gRPC), `8610` (Raft) +### Security + +- Keep all three ports on a trusted network. The REST API on `8620` includes management endpoints that mutate the cluster (peer changes, store removal, data movement), and the gRPC and Raft ports carry no authentication. +- REST requests need HTTP Basic auth: one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) with the `auth.secret-key` value as the password. Health probes (`/v1/health`, `/v1/ready`, `/actuator/**`, `/v1/prom/targets/*`) stay unauthenticated. +- `auth.secret-key` has no shipped default, because a secret in the source tree is published to everyone. Generate one per deployment (`openssl rand -hex 24`) and set it in the config file, or through `HG_PD_AUTH_SECRET_KEY`, which the Docker image requires. Give every REST client the same value: the Server's `bin/wait-storage.sh` reads `PD_AUTH_PASSWORD` (and `PD_AUTH_USER`, default `store`), and Hubble reads `operations.pd.password`. A client left on a stale secret gets 401, and `wait-storage.sh` aborts the Server's startup on the first one rather than waiting out `WAIT_STORAGE_TIMEOUT_S`. +- An existing `conf/application.yml` carried over from an earlier release has no `auth` block, and still carries `management.endpoints.web.exposure.include: "*"`. PD then starts with an empty secret and refuses every authenticated REST request, logging an error that names `auth.secret-key`, while `/actuator/env`, `/actuator/configprops` and `/actuator/beans` stay anonymously readable on `8620`. Before upgrading, add `auth.secret-key` and narrow that exposure to `health,metrics,prometheus`. PD refuses to start if the key is set to the placeholder value that earlier revisions of this repository carried. + ### Monitoring PD exposes metrics via REST API at: diff --git a/hugegraph-pd/docs/api-reference.md b/hugegraph-pd/docs/api-reference.md index 58b9698862..f7c37be9d5 100644 --- a/hugegraph-pd/docs/api-reference.md +++ b/hugegraph-pd/docs/api-reference.md @@ -760,6 +760,16 @@ for (Map.Entry entry : results.entrySet()) { PD exposes a REST API for management and monitoring (default port: 8620). +### Authentication + +Every endpoint below except the probes needs HTTP Basic auth: one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) as the user, and the `auth.secret-key` value from PD's `conf/application.yml` as the password. A missing or wrong credential gets HTTP 401. The `curl` examples that follow omit `-u` for readability; add it to every call except `/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, which stay unauthenticated for probes. + +```bash +curl -u hg: http://localhost:8620/v1/stores +``` + +Endpoints under `/v1` mutate the cluster (peer list changes, store removal, partition balancing), so keep port 8620 on a trusted network regardless. + ### Health Check ```bash @@ -796,33 +806,13 @@ curl -i http://localhost:8620/v1/ready } ``` -A follower reports `"state": "STATE_FOLLOWER"` with `"isLeader": false`. When -the quorum is lost the PD keeps answering `/v1/health` with `200` but -`/v1/ready` turns into `503` with `"ready": false`. Being unauthenticated, the -body carries no cluster addresses; the leader's address stays on `/v1/members`. - -The answer is served from state the raft callbacks maintain rather than from -the raft node, so it stays prompt while an election is running and never waits -on the node lock. `state` is therefore the last change raft announced. A PD -reports `STATE_UNINITIALIZED` with `"ready": false` from process start until -its first raft callback, which is the ordinary startup window before a quorum -first forms, and jraft emits no callback for candidacy or leadership transfer, -so a candidate reports `STATE_FOLLOWER` with `"ready": false`. - -Point Kubernetes readiness probes, `depends_on` healthchecks and any -"wait for PD" script at `/v1/ready`; keep liveness probes on `/v1/health` -so a PD that merely lost its leader is not restarted. - -Match on the body rather than on the status code alone. A PD that predates this -endpoint does not reliably answer `404` for it: `RestAuthentication` refuses a -request it does not exclude by writing an error envelope, and as of 1.7.0 it -does so without setting a status, so an unknown path answers `200` with -`{"status":-1,"error":"Unauthorized!"}`. A status-only probe therefore reads -such a PD as ready. The body match holds whichever status a refusal carries: a -shell gate should use -`curl -fsS http://:8620/v1/ready | grep -q '"ready":true'`, and a -Kubernetes `httpGet` probe should be paired with a PD image that carries the -endpoint. +A follower reports `"state": "STATE_FOLLOWER"` with `"isLeader": false`. When the quorum is lost the PD keeps answering `/v1/health` with `200` but `/v1/ready` turns into `503` with `"ready": false`. Being unauthenticated, the body carries no cluster addresses; the leader's address stays on `/v1/members`. + +The answer is served from state the raft callbacks maintain rather than from the raft node, so it stays prompt while an election is running and never waits on the node lock. `state` is therefore the last change raft announced. A PD reports `STATE_UNINITIALIZED` with `"ready": false` from process start until its first raft callback, which is the ordinary startup window before a quorum first forms, and jraft emits no callback for candidacy or leadership transfer, so a candidate reports `STATE_FOLLOWER` with `"ready": false`. + +Point Kubernetes readiness probes, `depends_on` healthchecks and any "wait for PD" script at `/v1/ready`; keep liveness probes on `/v1/health` so a PD that merely lost its leader is not restarted. + +Match on the body rather than on the status code alone. A PD that predates this endpoint does not reliably answer `404` for it: `RestAuthentication` refuses a request it does not exclude by writing an error envelope, and as of 1.7.0 it does so without setting a status, so an unknown path answers `200` with `{"status":-1,"error":"Unauthorized!"}`. A status-only probe therefore reads such a PD as ready. The body match holds whichever status a refusal carries: a shell gate should use `curl -fsS http://:8620/v1/ready | grep -q '"ready":true'`, and a Kubernetes `httpGet` probe should be paired with a PD image that carries the endpoint. ### Metrics @@ -856,19 +846,11 @@ Exported on `/actuator/prometheus` for alerting on quorum loss: | `hg_raft_has_leader` | `1` while this PD sees a leader (is inside a quorum), `0` otherwise | | `hg_raft_alive_peers` | Number of alive peers on the leader, itself included; `NaN` elsewhere | -`hg_raft_alive_peers` counts the peers the leader has heard from within the -leader lease timeout, which jraft derives as 90% of the election timeout by -default. +`hg_raft_alive_peers` counts the peers the leader has heard from within the leader lease timeout, which jraft derives as 90% of the election timeout by default. -A cluster has lost its quorum when `sum(hg_raft_leader) == 0` or when -`hg_raft_has_leader == 0` on every member. Both are briefly true during a -normal election, so alert on them with a `for:` clause longer than the -election timeout rather than on the instantaneous value. +A cluster has lost its quorum when `sum(hg_raft_leader) == 0` or when `hg_raft_has_leader == 0` on every member. Both are briefly true during a normal election, so alert on them with a `for:` clause longer than the election timeout rather than on the instantaneous value. -Do not aggregate `hg_raft_alive_peers` across instances: it is `NaN` on every -node but the leader, and one `NaN` sample turns the result of `sum()` or -`avg()` into `NaN` as well. Select the leader's series instead, for example -`hg_raft_alive_peers and on(instance) (hg_raft_leader == 1)`. +Do not aggregate `hg_raft_alive_peers` across instances: it is `NaN` on every node but the leader, and one `NaN` sample turns the result of `sum()` or `avg()` into `NaN` as well. Select the leader's series instead, for example `hg_raft_alive_peers and on(instance) (hg_raft_leader == 1)`. ### Partition API diff --git a/hugegraph-pd/docs/configuration.md b/hugegraph-pd/docs/configuration.md index e3ae4f6f25..4e60d8e8be 100644 --- a/hugegraph-pd/docs/configuration.md +++ b/hugegraph-pd/docs/configuration.md @@ -79,6 +79,32 @@ server: - Metrics: `http://:8620/actuator/metrics` - Prometheus: `http://:8620/actuator/prometheus` +### REST Authentication Settings + +Every REST request except the probes below must carry HTTP Basic auth: one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) as the user, and the shared secret as the password. A missing or wrong credential gets HTTP 401. Unauthenticated paths: `/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`. + +```yaml +auth: + secret-key: +``` + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `auth.secret-key` | String | none (required) | Password checked against the Basic credential. There is no default: a secret shipped in the source tree would be published to everyone. While it is empty PD refuses every authenticated REST request and logs an error naming this parameter, and PD refuses to start at all if it is set to the value that earlier revisions carried as a placeholder. | + +Every REST client needs the same value: the Server's `bin/wait-storage.sh` reads it from `PD_AUTH_PASSWORD`, Hubble from `operations.pd.password`, and the Docker image takes `HG_PD_AUTH_SECRET_KEY`. + +```bash +curl -u hg: http://:8620/v1/stores +``` + +`-u` puts the secret in curl's process arguments, where any local account can read it while the call runs, and PD REST is plain HTTP. On a shared host, or across a network you do not control, keep the secret out of `argv` by reading it from a file mode 0600: + +```bash +printf 'user = "hg:%s"\n' "${PD_SECRET}" > pd.curlrc && chmod 600 pd.curlrc +curl -K pd.curlrc http://:8620/v1/stores +``` + ### Raft Consensus Settings Controls Raft consensus for PD cluster coordination. @@ -253,13 +279,13 @@ management: endpoints: web: exposure: - include: "*" # Expose all actuator endpoints + include: "health,metrics,prometheus" # Allowlist; see note below ``` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `management.metrics.export.prometheus.enabled` | Boolean | `true` | Enable Prometheus-compatible metrics at `/actuator/prometheus`. | -| `management.endpoints.web.exposure.include` | String | `"*"` | Actuator endpoints to expose. `"*"` = all, or specify comma-separated list (e.g., `"health,metrics"`). | +| `management.endpoints.web.exposure.include` | String | `"health,metrics,prometheus"` | Actuator endpoints to expose. Actuator is served by its own handler mapping, which the REST authentication interceptor is not attached to, so every endpoint listed here is reachable without a credential on port 8620. This allowlist is what bounds which endpoints exist there, so prefer it over `"*"`. The interceptor's `/actuator/**` exclusion records the same intent but is not what makes these paths anonymous. In the PD Docker image the entrypoint emits this key in `SPRING_APPLICATION_JSON`, which outranks a mounted `conf/application.yml`, so editing it there has no effect; set `HG_PD_ACTUATOR_EXPOSURE` on the container instead. That variable defaults to the same allowlist and refuses a value containing `*`, because `/actuator/env` returns the `SPRING_APPLICATION_JSON` entry verbatim, PD's REST secret included. | ## Deployment Scenarios diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java index acfc2ec290..56bd58b34d 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java @@ -25,6 +25,7 @@ import org.apache.hugegraph.pd.ConfigService; import org.apache.hugegraph.pd.IdService; +import org.springframework.beans.factory.InitializingBean; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Configuration; @@ -32,13 +33,23 @@ import lombok.Data; import lombok.ToString; +import lombok.extern.slf4j.Slf4j; /** * PD profile */ +@Slf4j @Data @Component -public class PDConfig { +public class PDConfig implements InitializingBean { + + /** + * The secret that earlier revisions carried as the placeholder default for + * `auth.secret-key`. It is published in this repository, so a deployment + * still using it authenticates anyone who can read the source. Refuse to + * start rather than let a well-known string look like authentication. + */ + private static final String PUBLISHED_SECRET_KEY = "FXQXbJtbCLxODc6tGci732pkH1cyf8Qg"; // cluster ID @Value("${pd.cluster_id:1}") @@ -69,7 +80,11 @@ public class PDConfig { @Autowired private ThreadPoolGrpc threadPoolGrpc; - @Value("${auth.secret-key: 'FXQXbJtbCLxODc6tGci732pkH1cyf8Qg'}") + // No default: Spring takes the text after the first ':' literally, so a + // quoted default would resolve to a value including the quotes and the + // leading space, and no client would ever match it. An absent key must + // yield "" so the REST interceptor can refuse every request and say why. + @Value("${auth.secret-key:}") @ToString.Exclude private String secretKey; @@ -85,6 +100,25 @@ public class PDConfig { private ConfigService configService; private IdService idService; + @Override + public void afterPropertiesSet() { + if (PUBLISHED_SECRET_KEY.equals(this.secretKey)) { + throw new IllegalStateException( + "auth.secret-key is set to the value published in the HugeGraph source " + + "tree, which authenticates anyone who can read it. Set a " + + "deployment-specific secret in conf/application.yml, or through the " + + "HG_PD_AUTH_SECRET_KEY environment variable for the Docker image."); + } + // The shipped configs leave this empty on purpose. Say so in the boot log: + // the REST interceptor also logs it, but only on the first refused request, + // and /v1/health keeps answering 200 in the meantime. + if (this.secretKey == null || this.secretKey.isEmpty()) { + log.error("auth.secret-key is not configured, so every authenticated REST " + + "request will be refused. Add it to conf/application.yml (or set " + + "HG_PD_AUTH_SECRET_KEY) and give every REST client the same value."); + } + } + public Map getInitialStoreMap() { if (initialStoreMap == null) { initialStoreMap = new HashMap<>(); diff --git a/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh b/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh index 529936d06a..77af2af89d 100755 --- a/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh +++ b/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh @@ -26,10 +26,40 @@ require_env() { fi } +# Escape a value for use inside a JSON string: backslash and quote, then every +# remaining C0 control character as \uXXXX. Dropping only LF, as an earlier +# version did, left CR and TAB to produce invalid JSON and a container that +# failed before startup. +# +# LC_ALL=C is what makes the walk deterministic. eclipse-temurin:11-jre-jammy +# sets LC_ALL=en_US.UTF-8, and under that locale `${#s}`/`${s:i:1}` iterate +# characters rather than bytes and `<` sorts by collation rather than by code +# point. Pinned to C, iteration is byte-wise, the comparison is ordinal, and +# the bytes of a non-ASCII character (0x80 and up) pass through untouched, +# which is still valid JSON. The escape branch then only sees single-byte +# ASCII, so this pin is the only behavioral change. json_escape() { - local s="$1" - s=${s//\\/\\\\}; s=${s//\"/\\\"}; s=${s//$'\n'/} - printf "%s" "$s" + local LC_ALL=C + local s="$1" out="" i c + s=${s//\\/\\\\} + s=${s//\"/\\\"} + for (( i = 0; i < ${#s}; i++ )); do + c=${s:i:1} + case "$c" in + $'\n') out+='\n' ;; + $'\r') out+='\r' ;; + $'\t') out+='\t' ;; + $'\b') out+='\b' ;; + $'\f') out+='\f' ;; + *) + if [[ "$c" < $'\x20' || "$c" == $'\x7f' ]]; then + printf -v c '\\u%04x' "'$c" + fi + out+="$c" + ;; + esac + done + printf "%s" "$out" } migrate_env() { @@ -51,14 +81,44 @@ require_env "HG_PD_GRPC_HOST" require_env "HG_PD_RAFT_ADDRESS" require_env "HG_PD_RAFT_PEERS_LIST" require_env "HG_PD_INITIAL_STORE_LIST" +# The REST API refuses every authenticated request without this, and the image +# ships no default because a published secret is not a secret. +require_env "HG_PD_AUTH_SECRET_KEY" : "${HG_PD_GRPC_PORT:=8686}" : "${HG_PD_REST_PORT:=8620}" : "${HG_PD_DATA_PATH:=/hugegraph-pd/pd_data}" : "${HG_PD_INITIAL_STORE_COUNT:=1}" +# Actuator endpoints reachable without a credential. Hardened by default; an +# operator who needs /actuator/info or /actuator/loggers from this image opts +# in deliberately instead of losing the endpoint. "*" is refused: /actuator/env +# returns the SPRING_APPLICATION_JSON entry below verbatim, secret included. +: "${HG_PD_ACTUATOR_EXPOSURE:=health,metrics,prometheus}" +if [[ "${HG_PD_ACTUATOR_EXPOSURE}" == *'*'* ]]; then + echo "ERROR: HG_PD_ACTUATOR_EXPOSURE must not be a wildcard;" \ + "/actuator/env would publish the PD REST secret" >&2 + exit 2 +fi + +# Secret for REST Basic authentication (auth.secret-key). Required above and +# never logged. +AUTH_JSON="\"auth\": { \"secret-key\": \"$(json_escape "${HG_PD_AUTH_SECRET_KEY}")\" }," + +# The secret above lands in SPRING_APPLICATION_JSON, and actuator's /env +# sanitizer keys off the property name: it redacts auth.secret-key but returns +# the SPRING_APPLICATION_JSON environment entry verbatim, secret included. The +# image's own conf/application.yml already narrows the exposure, but a +# bind-mounted pre-1.8 config brings back include: "*". SPRING_APPLICATION_JSON +# outranks the config file, so pin the allowlist here too, from the env-driven +# default above rather than from a literal: every other PD setting in this +# entrypoint is env-driven, and pinning a literal took away the operator's +# only way to expose another endpoint from this image. +MANAGEMENT_JSON="\"management\": { \"endpoints\": { \"web\": { \"exposure\": { \"include\": \"$(json_escape "${HG_PD_ACTUATOR_EXPOSURE}")\" } } } }," SPRING_APPLICATION_JSON="$(cat < DEFAULT_HANDLE = () -> true; @Override @@ -49,19 +50,31 @@ public boolean preHandle(HttpServletRequest request, HttpServletResponse respons String authority = request.getHeader("Authorization"); if (authority == null) { - throw new Exception("Unauthorized!"); + throw new BadCredentialsException("Unauthorized!"); + } + if (!authority.regionMatches(true, 0, "Basic ", 0, 6)) { + throw new BadCredentialsException("invalid basic authentication info"); } Function tokenCall = t -> { response.addHeader(TOKEN_KEY, t); return true; }; - authority = authority.replace("Basic ", ""); + authority = authority.substring(6); return authenticate(authority, token, tokenCall, DEFAULT_HANDLE); } catch (Exception e) { + response.setStatus(HttpServletResponse.SC_UNAUTHORIZED); + // RFC 7235 requires a challenge on a 401; without it clients that + // authenticate reactively never retry with credentials + response.setHeader("WWW-Authenticate", "Basic realm=\"hugegraph-pd\""); response.setContentType("application/json"); - response.getWriter().println(new API().toJSON(e)); + // Constant body: the exception text named internal classes and told an + // unauthenticated caller whether the name or the password was wrong. + response.getWriter().println(UNAUTHORIZED_BODY); response.getWriter().flush(); + Throwable reason = e.getCause() != null ? e.getCause() : e; + log.debug("REST authentication refused for {} {}: {}", request.getMethod(), + request.getRequestURI(), reason.getMessage()); return false; } } diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java index 48bcf38683..4a8d73fa7e 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/service/interceptor/Authentication.java @@ -18,22 +18,29 @@ package org.apache.hugegraph.pd.service.interceptor; import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; import java.util.Base64; import java.util.Set; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.function.Function; import java.util.function.Supplier; import org.apache.commons.lang3.StringUtils; +import org.apache.hugegraph.pd.config.PDConfig; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.security.access.AccessDeniedException; import org.springframework.security.authentication.BadCredentialsException; import org.springframework.stereotype.Component; +import lombok.extern.slf4j.Slf4j; + /** * Simple internal authentication component for PD service. *

- * WARNING: This class currently implements only basic internal authentication - * validation for internal modules (hg, store, hubble, vermeer). The authentication mechanism - * is designed for internal service-to-service communication only. + * WARNING: This class validates a Basic credential for internal modules + * (hg, store, hubble, vermeer): the service name must be one of the four and the + * password must match the shared secret configured via `auth.secret-key`. The + * mechanism is designed for internal service-to-service communication only. *

* *

Important SEC Considerations:

@@ -57,10 +64,16 @@ * and regular security audits. *

*/ +@Slf4j @Component public class Authentication { private static final Set innerModules = Set.of("hg", "store", "hubble", "vermeer"); + private static final AtomicBoolean missingSecretLogged = new AtomicBoolean(); + + @Autowired + private PDConfig pdConfig; + protected T authenticate(String authority, String token, Function tokenCall, Supplier call) { try { @@ -70,26 +83,49 @@ protected T authenticate(String authority, String token, Function } byte[] bytes = authority.getBytes(StandardCharsets.UTF_8); byte[] decode = Base64.getDecoder().decode(bytes); - String info = new String(decode); + // RFC 7617: Basic credentials are UTF-8. Decoding with the platform + // default would compare against a UTF-8 secret only when the host + // locale happens to agree. + String info = new String(decode, StandardCharsets.UTF_8); int delim = info.indexOf(':'); if (delim == -1) { throw new BadCredentialsException(invalidBasicInfo); } String name = info.substring(0, delim); - // TODO: password validation is skipped — only service name is checked against - // innerModules. Full credential validation should be added as part of the auth refactor. - //String pwd = info.substring(delim + 1); - if (innerModules.contains(name)) { - return call.get(); - } else { + String pwd = info.substring(delim + 1); + if (!innerModules.contains(name)) { throw new AccessDeniedException("invalid service name"); } + if (!verifySecret(pwd)) { + throw new BadCredentialsException("invalid credential"); + } + return call.get(); } catch (Exception e) { throw new RuntimeException(e); } } + /** + * Compare the password of the Basic credential with the shared secret + * configured via `auth.secret-key`. A missing or empty secret refuses every + * request instead of falling back to name-only authentication. + */ + private boolean verifySecret(String pwd) { + String secret = this.pdConfig == null ? null : this.pdConfig.getSecretKey(); + if (StringUtils.isEmpty(secret)) { + // Logged once: this path is reachable by unauthenticated callers + if (missingSecretLogged.compareAndSet(false, true)) { + log.error("auth.secret-key is not configured, so every authenticated REST " + + "request is refused. Add it to conf/application.yml (or set " + + "HG_PD_AUTH_SECRET_KEY) and give every REST client the same value."); + } + return false; + } + return MessageDigest.isEqual(pwd.getBytes(StandardCharsets.UTF_8), + secret.getBytes(StandardCharsets.UTF_8)); + } + public static String getTokenKey(String name) { return "PD/TOKEN/" + name; } diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java index 2b1103739b..224d417cc4 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/util/grpc/GRpcServerConfig.java @@ -41,7 +41,12 @@ public void configure(ServerBuilder serverBuilder) { poolGrpc.getQueue())); serverBuilder.maxInboundMessageSize(MAX_INBOUND_MESSAGE_SIZE); // TODO: GrpcAuthentication is instantiated as a Spring bean but never registered - // here — add serverBuilder.intercept(grpcAuthentication) once auth is refactored. + // here - add serverBuilder.intercept(grpcAuthentication) once auth is refactored. + // It extends Authentication, which now also checks the Basic password against + // auth.secret-key. Registering it therefore requires giving that value to every + // gRPC client first: ServiceConstant.AUTHORITY (Server) and DefaultPdProvider + // .authority (Store) are "" and "default" today, and hg-pd-cli sends "". + // Otherwise no store can register once the interceptor is enabled. } } diff --git a/hugegraph-pd/hg-pd-service/src/main/resources/application.yml b/hugegraph-pd/hg-pd-service/src/main/resources/application.yml index 5a03595f7b..f8c1ecea84 100644 --- a/hugegraph-pd/hg-pd-service/src/main/resources/application.yml +++ b/hugegraph-pd/hg-pd-service/src/main/resources/application.yml @@ -27,7 +27,10 @@ management: endpoints: web: exposure: - include: "*" + # Allowlist, not "*": actuator has its own handler mapping, which + # the REST auth interceptor is not attached to, so anything exposed + # here is anonymous on this port. This list is what bounds it. + include: "health,metrics,prometheus" grpc: port: 8686 @@ -43,6 +46,14 @@ license: server: port: 8620 +auth: + # Shared secret checked against the password of the Basic credential on every + # authenticated REST request. Required and deliberately empty: a secret in + # the source tree is published to everyone. Set a deployment-specific value + # here (or via HG_PD_AUTH_SECRET_KEY) and give every REST client the same + # value. While empty, PD refuses every authenticated REST request. + secret-key: + pd: # Periodically check whether the cluster is healthy at intervals, in seconds patrol-interval: 300 diff --git a/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java new file mode 100644 index 0000000000..8793642d35 --- /dev/null +++ b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/service/interceptor/AuthenticationTest.java @@ -0,0 +1,220 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.pd.service.interceptor; + +import java.io.PrintWriter; +import java.io.StringWriter; +import java.lang.reflect.Field; +import java.lang.reflect.Proxy; +import java.nio.charset.StandardCharsets; +import java.util.Base64; + +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.apache.hugegraph.pd.config.PDConfig; +import org.apache.hugegraph.pd.rest.interceptor.RestAuthentication; +import org.junit.Assert; +import org.junit.Test; + +/** + * In-process cover for the REST credential check. The suites that exercise it + * over HTTP talk to a PD in another JVM, so nothing here is covered by them. + */ +public class AuthenticationTest { + + private static final String SECRET = "unit-test-secret"; + + private static Authentication authWithSecret(String secret) throws Exception { + Authentication auth = new Authentication(); + PDConfig config = new PDConfig(); + config.setSecretKey(secret); + Field field = Authentication.class.getDeclaredField("pdConfig"); + field.setAccessible(true); + field.set(auth, config); + return auth; + } + + private static RestAuthentication restAuthWithSecret(String secret) throws Exception { + RestAuthentication auth = new RestAuthentication(); + PDConfig config = new PDConfig(); + config.setSecretKey(secret); + Field field = Authentication.class.getDeclaredField("pdConfig"); + field.setAccessible(true); + field.set(auth, config); + return auth; + } + + private static boolean acceptsRest(RestAuthentication auth, String authHeader) { + try { + HttpServletRequest req = (HttpServletRequest) Proxy.newProxyInstance( + HttpServletRequest.class.getClassLoader(), + new Class[]{HttpServletRequest.class}, + (proxy, method, args) -> { + if ("getHeader".equals(method.getName())) { + return "Authorization".equalsIgnoreCase((String) args[0]) ? authHeader : null; + } + if ("getMethod".equals(method.getName())) { + return "GET"; + } + if ("getRequestURI".equals(method.getName())) { + return "/v1/stores"; + } + return null; + }); + StringWriter sw = new StringWriter(); + PrintWriter pw = new PrintWriter(sw); + HttpServletResponse resp = (HttpServletResponse) Proxy.newProxyInstance( + HttpServletResponse.class.getClassLoader(), + new Class[]{HttpServletResponse.class}, + (proxy, method, args) -> { + if ("getWriter".equals(method.getName())) { + return pw; + } + return null; + }); + return auth.preHandle(req, resp, null); + } catch (Exception e) { + return false; + } + } + + private static String credential(String name, String pwd) { + return Base64.getEncoder().encodeToString( + (name + ":" + pwd).getBytes(StandardCharsets.UTF_8)); + } + + private static boolean accepts(Authentication auth, String authority) { + try { + return auth.authenticate(authority, null, t -> Boolean.TRUE, () -> Boolean.TRUE); + } catch (RuntimeException e) { + return false; + } + } + + @Test + public void testEveryInnerModuleIsAcceptedWithTheSecret() throws Exception { + Authentication auth = authWithSecret(SECRET); + for (String name : new String[]{"hg", "store", "hubble", "vermeer"}) { + Assert.assertTrue(name + " should be accepted with the right secret", + accepts(auth, credential(name, SECRET))); + } + } + + @Test + public void testPasswordIsActuallyChecked() throws Exception { + Authentication auth = authWithSecret(SECRET); + Assert.assertFalse("wrong password must be refused", + accepts(auth, credential("hg", "wrong-password"))); + Assert.assertFalse("empty password must be refused", + accepts(auth, credential("hg", ""))); + Assert.assertFalse("secret as the name must not help", + accepts(auth, credential(SECRET, SECRET))); + } + + @Test + public void testUnknownServiceNameIsRefused() throws Exception { + Authentication auth = authWithSecret(SECRET); + Assert.assertFalse(accepts(auth, credential("nobody", SECRET))); + Assert.assertFalse(accepts(auth, credential("admin", SECRET))); + } + + @Test + public void testMissingOrMalformedCredentialIsRefused() throws Exception { + Authentication auth = authWithSecret(SECRET); + Assert.assertFalse(accepts(auth, null)); + Assert.assertFalse(accepts(auth, "")); + // no colon + Assert.assertFalse(accepts(auth, Base64.getEncoder().encodeToString( + "hg".getBytes(StandardCharsets.UTF_8)))); + } + + @Test + public void testUnconfiguredSecretRefusesEveryone() throws Exception { + for (String secret : new String[]{null, ""}) { + Authentication auth = authWithSecret(secret); + Assert.assertFalse("an unset secret must not fall back to a name check", + accepts(auth, credential("hg", ""))); + Assert.assertFalse(accepts(auth, credential("hg", SECRET))); + } + } + + @Test + public void testNonAsciiSecretDoesNotDependOnTheDefaultCharset() throws Exception { + String secret = "sécrèt-2026"; + Authentication auth = authWithSecret(secret); + Assert.assertTrue(accepts(auth, credential("hg", secret))); + Assert.assertFalse(accepts(auth, credential("hg", "secret-2026"))); + } + + @Test + public void testPublishedSecretRefusesStartup() { + PDConfig config = new PDConfig(); + config.setSecretKey("FXQXbJtbCLxODc6tGci732pkH1cyf8Qg"); + Assert.assertThrows(IllegalStateException.class, config::afterPropertiesSet); + } + + @Test + public void testOwnSecretStarts() { + PDConfig config = new PDConfig(); + config.setSecretKey(SECRET); + config.afterPropertiesSet(); + } + + @Test + public void testRestAuthenticationAcceptsCaseInsensitiveBasicScheme() throws Exception { + RestAuthentication auth = restAuthWithSecret(SECRET); + for (String name : new String[]{"hg", "store", "hubble", "vermeer"}) { + String cred = credential(name, SECRET); + Assert.assertTrue("Basic with uppercase B should be accepted for " + name, + acceptsRest(auth, "Basic " + cred)); + Assert.assertTrue("basic with all lowercase should be accepted for " + name, + acceptsRest(auth, "basic " + cred)); + Assert.assertTrue("BASIC with all uppercase should be accepted for " + name, + acceptsRest(auth, "BASIC " + cred)); + Assert.assertTrue("BaSiC with mixed case should be accepted for " + name, + acceptsRest(auth, "BaSiC " + cred)); + } + } + + @Test + public void testRestAuthenticationRefusesNonBasicOrMalformedSchemes() throws Exception { + RestAuthentication auth = restAuthWithSecret(SECRET); + String cred = credential("hg", SECRET); + Assert.assertFalse("null Authorization header must be refused", + acceptsRest(auth, null)); + Assert.assertFalse("empty Authorization header must be refused", + acceptsRest(auth, "")); + Assert.assertFalse("Basic with empty credentials must be refused", + acceptsRest(auth, "Basic ")); + Assert.assertFalse("basic with empty credentials must be refused", + acceptsRest(auth, "basic ")); + Assert.assertFalse("Bearer scheme must be refused", + acceptsRest(auth, "Bearer " + cred)); + Assert.assertFalse("Digest scheme must be refused", + acceptsRest(auth, "Digest " + cred)); + Assert.assertFalse("Basic without trailing space must be refused", + acceptsRest(auth, "Basic" + cred)); + Assert.assertFalse("invalid credential with basic prefix must be refused", + acceptsRest(auth, "basic " + credential("hg", "wrong-secret"))); + Assert.assertFalse("unknown service name with basic prefix must be refused", + acceptsRest(auth, "basic " + credential("unknown", SECRET))); + Assert.assertFalse("non-base64 token with basic prefix must be refused", + acceptsRest(auth, "basic ???")); + } +} diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java index 0836120c73..d7df26d59e 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/BaseTest.java @@ -17,6 +17,9 @@ package org.apache.hugegraph.pd; +import java.nio.charset.StandardCharsets; +import java.util.Base64; + import org.apache.hugegraph.pd.client.PDConfig; public class BaseTest { @@ -24,9 +27,12 @@ public class BaseTest { protected static String pdGrpcAddr = "127.0.0.1:8686"; protected static String pdRestAddr = "http://127.0.0.1:8620"; protected static String user = "store"; - protected static String pwd = ""; + // Must match the auth.secret-key that travis/start-pd.sh gives the PD + // under test; the shipped config carries no secret by design + protected static String pwd = "pd-ci-test-secret-not-for-production"; protected static String key = "Authorization"; - protected static String value = "Basic c3RvcmU6YWRtaW4="; + protected static String value = "Basic " + Base64.getEncoder().encodeToString( + (user + ":" + pwd).getBytes(StandardCharsets.UTF_8)); protected PDConfig getPdConfig() { return PDConfig.of(pdGrpcAddr).setAuthority(user, pwd); diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java index 4aff85d1e9..5c808b6527 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/BaseServerTest.java @@ -18,15 +18,29 @@ package org.apache.hugegraph.pd.rest; import java.net.http.HttpClient; +import java.nio.charset.StandardCharsets; +import java.util.Base64; import org.junit.After; import org.junit.BeforeClass; public class BaseServerTest { + // Must match the auth.secret-key that travis/start-pd.sh gives the PD + // under test; the shipped config carries no secret by design + protected static final String SECRET = "pd-ci-test-secret-not-for-production"; + protected static final String AUTH_HEADER = "Authorization"; + protected static final String VALID_AUTH = basicAuth("store", SECRET); + protected static HttpClient client; protected static String pdRestAddr; + protected static String basicAuth(String name, String pwd) { + String credential = name + ":" + pwd; + return "Basic " + Base64.getEncoder() + .encodeToString(credential.getBytes(StandardCharsets.UTF_8)); + } + @BeforeClass public static void init() { client = HttpClient.newHttpClient(); diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java index da90f6f0f9..c7c2917464 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java @@ -36,7 +36,7 @@ public void testQueryIndexInfo() throws URISyntaxException, IOException, Interru String url = pdRestAddr + "/"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -55,7 +55,7 @@ public void testQueryClusterInfo() throws URISyntaxException, IOException, Inter String url = pdRestAddr + "/v1/cluster"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -125,7 +125,7 @@ public void testQueryClusterMembers() throws URISyntaxException, IOException, String url = pdRestAddr + "/v1/members"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -139,7 +139,7 @@ public void testQueryStoresInfo() throws URISyntaxException, IOException, Interr String url = pdRestAddr + "/v1/stores"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -153,7 +153,7 @@ public void testQueryGraphsInfo() throws IOException, InterruptedException, JSON String url = pdRestAddr + "/v1/graphs"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -167,7 +167,7 @@ public void testQueryPartitionsInfo() throws IOException, InterruptedException, String url = pdRestAddr + "/v1/highLevelPartitions"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -181,7 +181,7 @@ public void testQueryDebugPartitionsInfo() throws URISyntaxException, IOExceptio String url = pdRestAddr + "/v1/partitions"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); @@ -194,11 +194,95 @@ public void testQueryShards() throws URISyntaxException, IOException, Interrupte String url = pdRestAddr + "/v1/shards"; HttpRequest request = HttpRequest.newBuilder() .uri(new URI(url)) - .header("Authorization", "Basic c3RvcmU6MTIz") + .header(AUTH_HEADER, VALID_AUTH) .GET() .build(); HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); JSONObject obj = new JSONObject(response.body()); assert obj.getInt("status") == 0; } + + @Test + public void testMissingCredentialGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } + + @Test + public void testWrongPasswordGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .header(AUTH_HEADER, basicAuth("store", "wrong-password")) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } + + @Test + public void testEmptyPasswordGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .header(AUTH_HEADER, basicAuth("hg", "")) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } + + private int statusWithoutCredential(String path) throws URISyntaxException, IOException, + InterruptedException { + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(pdRestAddr + path)) + .GET() + .build(); + return client.send(request, HttpResponse.BodyHandlers.ofString()).statusCode(); + } + + @Test + public void testProbePathsNeedNoCredential() throws URISyntaxException, IOException, + InterruptedException { + // != 401, not == 200: /actuator/health answers 503 whenever any health + // indicator is DOWN (low disk space on a CI runner is the usual one) + // and /v1/health reflects cluster state, so == 200 would report a + // transient unhealthy PD as an authentication regression and send + // someone looking in this file. The claim here is only that these + // paths are reachable without a credential. + assert statusWithoutCredential("/v1/health") != 401; + assert statusWithoutCredential("/actuator/health") != 401; + // Nested actuator paths are probe surface too. They stay open because + // actuator has its own handler mapping that the auth interceptor is not + // attached to, not because of the /actuator/** exclusion pattern. + assert statusWithoutCredential("/actuator/metrics/jvm.memory.used") == 200; + } + + @Test + public void testUnexposedActuatorEndpointIsClosed() throws URISyntaxException, IOException, + InterruptedException { + // not in management.endpoints.web.exposure.include, so it must never serve data + assert statusWithoutCredential("/actuator/env") != 200; + } + + @Test + public void testUnknownServiceNameGets401() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/members"; + HttpRequest request = HttpRequest.newBuilder() + .uri(new URI(url)) + .header(AUTH_HEADER, basicAuth("nobody", SECRET)) + .GET() + .build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + assert response.statusCode() == 401; + } } diff --git a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh index 93c3a19dd2..168efe8979 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/static/bin/wait-storage.sh @@ -39,7 +39,30 @@ log() { echo "[wait-storage] $1" } -PD_AUTH_ARGS="-u ${PD_AUTH_USER:-store}:${PD_AUTH_PASSWORD:-admin}" +# PD REST credential. PD checks the password against its auth.secret-key and +# ships no default, so this has to be provided by the deployment. +# The value is deliberately kept out of the inner script's source text and out +# of curl's argv: the inner shell reads it from the environment and hands it to +# curl on stdin as a config file. +PD_AUTH_USER="${PD_AUTH_USER:-store}" +PD_AUTH_PASSWORD="${PD_AUTH_PASSWORD:-}" +# curl -K reads one option per line and takes the value as a quoted string +# whose only escapes are \\ \" \t \n \r \v. Backslash first, then the rest; +# an unescaped line break would end the option early and send a truncated +# credential (curl then warns that the remainder is an unknown option). +escape_curlrc() { + local v=$1 + v=${v//\\/\\\\} + v=${v//\"/\\\"} + v=${v//$'\n'/\\n} + v=${v//$'\r'/\\r} + v=${v//$'\t'/\\t} + v=${v//$'\v'/\\v} + printf '%s' "$v" +} +PD_AUTH_CURL_USER=$(escape_curlrc "${PD_AUTH_USER}") +PD_AUTH_CURL_PASSWORD=$(escape_curlrc "${PD_AUTH_PASSWORD}") +export PD_AUTH_CURL_USER PD_AUTH_CURL_PASSWORD function key_exists { local key=$1 @@ -93,33 +116,77 @@ if env | grep '^hugegraph\.' > /dev/null; then export PD_REST_LIST log "PD REST peers = $PD_REST_LIST" + # Only worth saying where PD is actually polled: topologies without + # pd.peers never send this credential anywhere. + if [ -z "${PD_AUTH_PASSWORD}" ]; then + log "WARN: PD_AUTH_PASSWORD is empty; PD will answer 401 unless it runs without auth" + fi log "Timeout = ${WAIT_STORAGE_TIMEOUT_S}s" timeout "${WAIT_STORAGE_TIMEOUT_S}s" bash -c " log() { echo '[wait-storage] '\"\$1\"; } + # curl stays out of the grep pipeline so its status code is + # readable: a 401 is a wrong secret, not a storage problem, and + # retrying it for 300s only hides that. + # + # 401s are counted, not flagged, so the abort is for the fleet + # and never for one peer. That distinction is real: PD serves + # /v1/stores well before stores finish registering, so the first + # pass of a rolling secret rotation can find one stale peer + # refusing while the healthy peers are merely storeless. A flag + # turned that into a dead Server. Returning 2 only when every + # peer polled refused keeps the fail-fast for a fleet-wide wrong + # secret, which still aborts on pass one instead of retrying for + # the full 300s. check_any_pd_stores() { + refused=0 + peers=0 for peer in \$(echo \"\$PD_REST_LIST\" | tr ',' ' '); do - if curl ${PD_AUTH_ARGS} -f -s \ - --connect-timeout ${WAIT_STORAGE_PD_CONNECT_TIMEOUT_S} \ - --max-time ${WAIT_STORAGE_PD_MAX_TIMEOUT_S} \ - http://\${peer}/v1/stores 2>/dev/null | \ - grep -qi '\"state\"[[:space:]]*:[[:space:]]*\"Up\"'; then + peers=\$((peers + 1)) + body=\$(printf 'user = \"%s:%s\"\n' \ + \"\$PD_AUTH_CURL_USER\" \"\$PD_AUTH_CURL_PASSWORD\" | \ + curl -K - -s -w '\n%{http_code}' \ + --connect-timeout ${WAIT_STORAGE_PD_CONNECT_TIMEOUT_S} \ + --max-time ${WAIT_STORAGE_PD_MAX_TIMEOUT_S} \ + \"http://\${peer}/v1/stores\" 2>/dev/null) + code=\${body##*\$'\n'} + if [ \"\$code\" = 401 ]; then + log \"ERROR: PD at \${peer} refused the credential (401):\" >&2 + log ' PD_AUTH_PASSWORD must match PD auth.secret-key' >&2 + refused=\$((refused + 1)) + continue + fi + if printf '%s' \"\$body\" | grep -qi '\"state\"[[:space:]]*:[[:space:]]*\"Up\"'; then echo \"\$peer\" return 0 fi done + [ \"\$peers\" -gt 0 ] && [ \"\$refused\" -eq \"\$peers\" ] && return 2 return 1 } until PD_REST=\$(check_any_pd_stores); do + # Must stay the first statement in the loop: any command in + # front of it would overwrite \$? and turn the 401 abort back + # into a 300s retry. + rc=\$? + if [ \"\$rc\" -eq 2 ]; then exit 2; fi log 'No Up store yet, retrying in 5s' sleep 5 done log \"Store registration check PASSED via \$PD_REST\" log 'Storage backend is VIABLE' - " || { echo "[wait-storage] ERROR: Timeout waiting for storage backend"; exit 1; } + " || { + rc=$? + if [ "$rc" -eq 124 ]; then + echo "[wait-storage] ERROR: Timeout waiting for storage backend" + else + echo "[wait-storage] ERROR: storage wait aborted, see the message above" + fi + exit 1 + } else log "No pd.peers configured, skipping storage wait" diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh index 0c137489e1..94d4f82fed 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/start-pd.sh @@ -32,6 +32,15 @@ fi PD_DIR=$HOME_DIR/hugegraph-pd/apache-hugegraph-pd-$VersionInBash pushd $PD_DIR -. bin/start-hugegraph-pd.sh +# conf/application.yml ships auth.secret-key empty on purpose, so PD would +# refuse every authenticated REST request. Supply a test-only secret; it must +# match the value the PD test suites send. Keep it inside this subshell: +# install-hstore.sh sources this script and start-store.sh in the same shell, +# and an exported SPRING_APPLICATION_JSON would reach Store's Spring context +# too. +( + export SPRING_APPLICATION_JSON='{"auth":{"secret-key":"pd-ci-test-secret-not-for-production"}}' + . bin/start-hugegraph-pd.sh +) sleep 10 popd diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh new file mode 100755 index 0000000000..ed39b67492 --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-docker-entrypoint.sh @@ -0,0 +1,176 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Checks that the PD Docker entrypoint turns HG_PD_AUTH_SECRET_KEY into valid +# SPRING_APPLICATION_JSON, whatever the secret contains, that the value Spring +# would read back is the secret that went in, and that the same document pins +# the actuator exposure allowlist. The allowlist has to travel with the secret: +# SPRING_APPLICATION_JSON outranks a bind-mounted conf/application.yml, and an +# older one exposing every actuator endpoint would otherwise serve that secret +# back from /actuator/env. + +set -euo pipefail + +ENTRYPOINT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../.." && pwd)/hugegraph-pd/hg-pd-dist/docker/docker-entrypoint.sh" +TMP_DIR=$(mktemp -d "${TMPDIR:-/tmp}/pd-entrypoint-test.XXXXXX") +trap 'rm -rf "${TMP_DIR}"' EXIT + +PASS=0 +FAIL=0 + +[[ -f "${ENTRYPOINT}" ]] || { echo "entrypoint not found at ${ENTRYPOINT}" >&2; exit 1; } +command -v python3 >/dev/null || { echo "python3 required" >&2; exit 1; } + +mkdir -p "${TMP_DIR}/bin" +cp "${ENTRYPOINT}" "${TMP_DIR}/docker-entrypoint.sh" +# Stand in for the launcher: record the generated config instead of starting PD +cat > "${TMP_DIR}/bin/start-hugegraph-pd.sh" <<'STUB' +#!/usr/bin/env bash +printf '%s' "${SPRING_APPLICATION_JSON}" > ./spring.json +STUB +chmod +x "${TMP_DIR}/bin/start-hugegraph-pd.sh" "${TMP_DIR}/docker-entrypoint.sh" + +run_case() { + # run_case [expected exposure] [extra NAME=VALUE ...] + local name="$1" secret="$2" expected_exposure="${3:-health,metrics,prometheus}" + shift 2 + (( $# == 0 )) || shift # drop the exposure argument; the rest is extra env + local out + if ! out=$(cd "${TMP_DIR}" && env \ + "$@" \ + HG_PD_GRPC_HOST=pd0 \ + HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 \ + HG_PD_INITIAL_STORE_LIST=store0:8500 \ + HG_PD_AUTH_SECRET_KEY="${secret}" \ + ./docker-entrypoint.sh 2>&1); then + echo " FAIL ${name}: entrypoint exited non-zero" + printf '%s\n' "${out}" | tail -3 + FAIL=$((FAIL + 1)) + return + fi + + if ! SECRET="${secret}" EXPOSURE="${expected_exposure}" \ + python3 - "${TMP_DIR}/spring.json" <<'PY' +import json, os, sys +with open(sys.argv[1], encoding="utf-8") as fh: + doc = json.load(fh) +got = doc["auth"]["secret-key"] +want = os.environ["SECRET"] +if got != want: + print(" round-trip mismatch: %r != %r" % (got, want)) + sys.exit(1) +exposure = doc.get("management", {}).get("endpoints", {}).get("web", {}) +exposure = exposure.get("exposure", {}).get("include") +if exposure != os.environ["EXPOSURE"]: + print(" actuator exposure is %r, expected %r" + % (exposure, os.environ["EXPOSURE"])) + sys.exit(1) +PY + then + echo " FAIL ${name}: invalid JSON, secret did not round-trip, or exposure is wrong" + FAIL=$((FAIL + 1)) + return + fi + echo " PASS ${name}" + PASS=$((PASS + 1)) +} + +echo "PD docker-entrypoint secret override" +run_case "plain secret" 'aVerySecretValue123' +run_case "carriage return" "$(printf 'a\rb')" +run_case "tab" "$(printf 'a\tb')" +run_case "double quote" 'a"b' +run_case "backslash" 'a\b' +run_case "backslash and quote" 'a\"b' +run_case "non-ascii" 'sécrèt-2026' +run_case "spaces" 'two words' + +# json_escape walks the secret with ${#s} and ${s:i:1} and compares with `<`. +# Both are locale-sensitive, and the base image (eclipse-temurin:11-jre-jammy) +# sets LC_ALL=en_US.UTF-8, so the same secret has to survive a UTF-8 locale +# exactly as it does under C. Pick a UTF-8 locale the host actually has: +# an ungenerated one silently falls back to C and makes the case vacuous. +# Captured rather than piped into grep: `grep -q` exits at the first match, and +# under pipefail the SIGPIPE that gives `locale -a` fails the whole pipeline. +AVAILABLE_LOCALES=$(locale -a 2>/dev/null || true) +UTF8_LOCALE="" +for candidate in C.UTF-8 en_US.UTF-8 en_US.utf8; do + if grep -Fqix -- "${candidate}" <<<"${AVAILABLE_LOCALES}"; then + UTF8_LOCALE="${candidate}" + break + fi +done +if [[ -n "${UTF8_LOCALE}" ]]; then + run_case "non-ascii under ${UTF8_LOCALE}" 'sécrèt-2026' \ + 'health,metrics,prometheus' "LC_ALL=${UTF8_LOCALE}" + run_case "control chars under ${UTF8_LOCALE}" "$(printf 'a\rb\tc')" \ + 'health,metrics,prometheus' "LC_ALL=${UTF8_LOCALE}" +else + echo " SKIP UTF-8 locale cases: no UTF-8 locale on this host" +fi + +# The allowlist is the default, not a pin: an operator who needs another +# endpoint from this image opts in rather than losing the endpoint entirely. +run_case "actuator exposure override" 'aVerySecretValue123' \ + 'health,metrics,prometheus,loggers' \ + 'HG_PD_ACTUATOR_EXPOSURE=health,metrics,prometheus,loggers' + +# ...but not all the way back to the hole this closed: /actuator/env returns +# the SPRING_APPLICATION_JSON entry verbatim, secret included. +for wildcard in '*' 'health,*'; do + if (cd "${TMP_DIR}" && env \ + HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \ + HG_PD_AUTH_SECRET_KEY='aVerySecretValue123' \ + HG_PD_ACTUATOR_EXPOSURE="${wildcard}" \ + ./docker-entrypoint.sh >/dev/null 2>&1); then + echo " FAIL wildcard exposure '${wildcard}' was accepted" + FAIL=$((FAIL + 1)) + else + echo " PASS wildcard exposure '${wildcard}' is refused" + PASS=$((PASS + 1)) + fi +done + +# The secret is required, and must never be echoed to the log +if (cd "${TMP_DIR}" && env \ + HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \ + ./docker-entrypoint.sh >/dev/null 2>&1); then + echo " FAIL missing secret: entrypoint started without HG_PD_AUTH_SECRET_KEY" + FAIL=$((FAIL + 1)) +else + echo " PASS missing secret is refused" + PASS=$((PASS + 1)) +fi + +log_out=$(cd "${TMP_DIR}" && env \ + HG_PD_GRPC_HOST=pd0 HG_PD_RAFT_ADDRESS=pd0:8610 \ + HG_PD_RAFT_PEERS_LIST=pd0:8610 HG_PD_INITIAL_STORE_LIST=store0:8500 \ + HG_PD_AUTH_SECRET_KEY='do-not-log-this-value' \ + ./docker-entrypoint.sh 2>&1) +if printf '%s' "${log_out}" | grep -q 'do-not-log-this-value'; then + echo " FAIL secret was written to the log" + FAIL=$((FAIL + 1)) +else + echo " PASS secret is not logged" + PASS=$((PASS + 1)) +fi + +echo "${PASS} passed, ${FAIL} failed" +[[ "${FAIL}" -eq 0 ]] diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-shipped-config.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-shipped-config.sh new file mode 100755 index 0000000000..46521dd05e --- /dev/null +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-pd-shipped-config.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Every PD configuration that ships in an archive or in the jar must carry the +# same REST hardening: no wildcard actuator exposure (that path is anonymous), +# an auth.secret-key that is present and empty, and no copy of the secret that +# earlier revisions published. A fix applied to one variant and not the others +# is what this catches, so the list below covers the PD distribution, the +# service jar, and the template the cluster test writes onto each PD node. + +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../../../.." && pwd)" +PUBLISHED_SECRET='FXQXbJtbCLxODc6tGci732pkH1cyf8Qg' +# The allowlist these files must carry, spelled out. An exact comparison rather +# than "no wildcard": a missing include:, a reordered or duplicated entry, and +# an extra endpoint are all changes to what this port serves anonymously, and +# each of them used to pass. +EXPECTED_EXPOSURE='health,metrics,prometheus' +FAIL=0 + +check() { + local file="$1" rel="${1#"${ROOT}/"}" bad=0 + [[ -f "$file" ]] || { echo " FAIL ${rel}: missing"; FAIL=1; return; } + + # The actuator exposure specifically: a config that grows an unrelated + # include: above this block must not satisfy the check by accident. + local exposure + exposure=$(awk '/^[[:space:]]*exposure:/ {found = 1; next} + found && /^[[:space:]]*include:/ { + sub(/^[[:space:]]*include:[[:space:]]*/, "") + sub(/[[:space:]]+$/, "") + print; exit + }' "$file") + # YAML quoting is the file's business, not this contract's + exposure=${exposure#\"}; exposure=${exposure%\"} + exposure=${exposure#\'}; exposure=${exposure%\'} + if [[ "$exposure" != "${EXPECTED_EXPOSURE}" ]]; then + echo " FAIL ${rel}: actuator exposure must be exactly" \ + "'${EXPECTED_EXPOSURE}', got '${exposure}'"; FAIL=1; bad=1 + fi + if ! grep -qE '^[[:space:]]*secret-key:[[:space:]]*$' "$file"; then + echo " FAIL ${rel}: auth.secret-key must be present and empty"; FAIL=1; bad=1 + fi + if grep -q "${PUBLISHED_SECRET}" "$file"; then + echo " FAIL ${rel}: contains the published secret"; FAIL=1; bad=1 + fi + # Per file, not the global FAIL: once an earlier file has set that, every + # later file matches it again and a failing file reports itself ok. Kept as + # an if rather than a bare test-and-echo, which would be the function's last + # command and return 1 on a failing file, aborting the loop under set -e. + if [[ "$bad" -eq 0 ]]; then + echo " ok ${rel}" + fi +} + +echo "PD shipped configuration hardening" +for f in "${ROOT}"/hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml* \ + "${ROOT}"/hugegraph-pd/hg-pd-service/src/main/resources/application.yml \ + "${ROOT}"/hugegraph-cluster-test/hugegraph-clustertest-dist/src/assembly/static/conf/pd-application.yml.template; do + check "$f" +done +[[ "$FAIL" -eq 0 ]] && echo "all shipped PD configs pass" || { echo "shipped PD config check failed"; exit 1; } diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh index da6a008f05..6d7c4ce8d0 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-wait-storage.sh @@ -25,6 +25,7 @@ DIST_ROOT="${TMP_DIR}/dist" MOCK_BIN="${TMP_DIR}/mock-bin" CALL_LOG="${TMP_DIR}/curl-calls" ARGS_LOG="${TMP_DIR}/curl-args" +CONFIG_LOG="${TMP_DIR}/curl-config" COUNT_FILE="${TMP_DIR}/store-call-count" TIMEOUT_LOG="${TMP_DIR}/timeout-arg" CASE_OUTPUT="" @@ -54,8 +55,12 @@ assert_contract() { ! grep -q '/v1/health' "${CALL_LOG}" || \ fail "/v1/health must not gate readiness" [[ -s "${ARGS_LOG}" ]] || fail "curl was not called" - if grep -Fv -- '-u test-user:test-password' "${ARGS_LOG}" | grep -q .; then - fail "authentication arguments were not preserved" + if grep -Fq -- 'test-password' "${ARGS_LOG}"; then + fail "credential leaked into curl argv" + fi + [[ -s "${CONFIG_LOG}" ]] || fail "curl was not given a credential config" + if grep -Fv -- 'user = "test-user:test-password"' "${CONFIG_LOG}" | grep -q .; then + fail "authentication credential was not preserved" fi if grep -Fv -- '--connect-timeout 2' "${ARGS_LOG}" | grep -q .; then fail "per-peer connect timeout was not preserved" @@ -67,9 +72,10 @@ assert_contract() { } run_case() { - local scenario="$1" peers="$2" abort_after="$3" + local scenario="$1" peers="$2" abort_after="$3" password="${4:-test-password}" : > "${CALL_LOG}" : > "${ARGS_LOG}" + : > "${CONFIG_LOG}" : > "${COUNT_FILE}" : > "${TIMEOUT_LOG}" : > "${DIST_ROOT}/conf/graphs/hugegraph.properties" @@ -80,11 +86,12 @@ run_case() { MOCK_ABORT_AFTER="${abort_after}" \ MOCK_CALL_LOG="${CALL_LOG}" \ MOCK_ARGS_LOG="${ARGS_LOG}" \ + MOCK_CONFIG_LOG="${CONFIG_LOG}" \ MOCK_COUNT_FILE="${COUNT_FILE}" \ MOCK_TIMEOUT_LOG="${TIMEOUT_LOG}" \ HG_SERVER_PD_REST_ENDPOINT="${peers}" \ PD_AUTH_USER="test-user" \ - PD_AUTH_PASSWORD="test-password" \ + PD_AUTH_PASSWORD="${password}" \ 'hugegraph.backend=hstore' \ 'hugegraph.pd.peers=config-only:8686' \ "${DIST_ROOT}/bin/wait-storage.sh" 2>&1) @@ -132,6 +139,26 @@ url="${!#}" printf '%s\n' "$*" >> "${MOCK_ARGS_LOG}" printf '%s\n' "${url}" >> "${MOCK_CALL_LOG}" +# The credential must arrive as a config file on stdin, never in argv +for arg in "$@"; do + if [[ "${arg}" == "-K" ]]; then + cat >> "${MOCK_CONFIG_LOG}" + break + fi +done + +# Honour -w like curl: append the write-out with %{http_code} substituted +fmt="" +prev="" +for arg in "$@"; do + [[ "${prev}" == "-w" ]] && fmt="${arg}" + prev="${arg}" +done +respond() { + printf '%s\n' "$1" + [[ -z "${fmt}" ]] || printf '%s' "${fmt//\\n/$'\n'}" | sed "s/%{http_code}/$2/" +} + if [[ "${url}" == */v1/health ]]; then printf '{}\n' exit 0 @@ -141,9 +168,25 @@ count=$(cat "${MOCK_COUNT_FILE}" 2>/dev/null || true) count=$((${count:-0} + 1)) printf '%s\n' "${count}" > "${MOCK_COUNT_FILE}" -if [[ "${MOCK_SCENARIO}" == "pd1-up" && \ +if [[ "${MOCK_SCENARIO}" == "auth-401" ]]; then + respond '{"status":-1,"error":"Unauthorized"}' 401 +elif [[ "${MOCK_SCENARIO}" == "one-401" && \ + "${url}" == "http://pd0:8620/v1/stores" ]]; then + # One stale peer mid-rotation, or a pre-1.8 peer that took any password + respond '{"status":-1,"error":"Unauthorized"}' 401 +elif [[ "${MOCK_SCENARIO}" == "one-401" && \ + "${url}" == "http://pd1:8620/v1/stores" ]]; then + respond '{"stores":[{"state":"Up"}]}' 200 +elif [[ "${MOCK_SCENARIO}" == "one-401-pending" && \ + "${url}" == "http://pd0:8620/v1/stores" ]]; then + # Same stale peer, but caught before any store has finished registering + respond '{"status":-1,"error":"Unauthorized"}' 401 +elif [[ "${MOCK_SCENARIO}" == "one-401-pending" && \ + "${url}" == "http://pd1:8620/v1/stores" ]]; then + respond '{"stores":[{"state":"Pending"}]}' 200 +elif [[ "${MOCK_SCENARIO}" == "pd1-up" && \ "${url}" == "http://pd1:8620/v1/stores" ]]; then - printf '{"stores":[{"state":"Up"}]}\n' + respond '{"stores":[{"state":"Up"}]}' 200 elif [[ "${MOCK_SCENARIO}" == "hanging-first" && \ "${url}" == "http://pd0:8620/v1/stores" ]]; then if [[ " $* " == *" --connect-timeout 2 "* && \ @@ -155,15 +198,15 @@ elif [[ "${MOCK_SCENARIO}" == "hanging-first" && \ exit 28 elif [[ "${MOCK_SCENARIO}" == "hanging-first" && \ "${url}" == "http://pd1:8620/v1/stores" ]]; then - printf '{"stores":[{"state":"Up"}]}\n' + respond '{"stores":[{"state":"Up"}]}' 200 elif [[ "${MOCK_SCENARIO}" == "retry" && "${count}" -eq 3 && \ "${url}" == "http://pd0:8620/v1/stores" ]]; then exit 7 elif [[ "${MOCK_SCENARIO}" == "retry" && "${count}" -eq 4 && \ "${url}" == "http://pd1:8620/v1/stores" ]]; then - printf '{"stores":[{"state":"Up"}]}\n' + respond '{"stores":[{"state":"Up"}]}' 200 else - printf '{"stores":[]}\n' + respond '{"stores":[]}' 200 fi EOF @@ -210,4 +253,60 @@ assert_output "ERROR: Timeout waiting for storage backend" assert_contract echo " PASS all-unready timeout" -echo "5 passed, 0 failed" +# A secret with CR/LF must reach curl as one escaped config line, not two. +run_case "pd1-up" "pd0:8620,pd1:8620" 6 "$(printf 'a\r\nb\\c"d')" +assert_equal "line-break secret rc" "0" "${CASE_RC}" +if grep -Fv -- 'user = "test-user:a\r\nb\\c\"d"' "${CONFIG_LOG}" | grep -q .; then + fail "line break or quote in the secret was not escaped for curl -K" +fi +echo " PASS line break in secret" + +# A fleet-wide wrong secret is not a storage problem: finish the pass so every +# refusal is named, then abort rather than retrying it for the full 300s. +run_case "auth-401" "pd0:8620,pd1:8620" 9 +[[ "${CASE_RC}" -ne 0 ]] || fail "a 401 from every peer must abort" +assert_output "refused the credential (401)" +assert_equal "one pass, no retry after 401" "${TWO_CALLS}" "$(cat "${CALL_LOG}")" +[[ "${CASE_OUTPUT}" != *"Timeout waiting"* ]] || fail "401 was reported as a timeout" +echo " PASS 401 from every peer aborts without retry" + +# One refusing peer must not cost the Server: during a rolling secret rotation, +# or against a pre-1.8 peer that answers 200 to any password, the next peer in +# PD_REST_LIST accepts the same credential. Only a refusal from every peer that +# answered is a fleet-wide wrong secret, so a lone 401 stays a retry. +run_case "one-401" "pd0:8620,pd1:8620" 6 +assert_equal "one refusing peer rc" "0" "${CASE_RC}" +assert_equal "kept going past the 401" "${TWO_CALLS}" "$(cat "${CALL_LOG}")" +assert_output "refused the credential (401)" +assert_output "Store registration check PASSED via pd1:8620" +echo " PASS one refusing peer does not end the wait" + +# The same lone 401 caught before any store is Up. PD serves /v1/stores well +# ahead of store registration, so the first pass of a rolling rotation sees the +# stale peer refuse and the healthy peers still storeless. That must retry, not +# abort: the store is on its way, and the accepting peer is right there. +run_case "one-401-pending" "pd0:8620,pd1:8620" 4 +[[ "${CASE_RC}" -ne 0 ]] || fail "a storeless fleet must still fail closed" +assert_equal "retried past a lone 401" "${FOUR_CALLS}" "$(cat "${CALL_LOG}")" +assert_output "refused the credential (401)" +assert_output "No Up store yet, retrying in 5s" +assert_output "ERROR: Timeout waiting for storage backend" +[[ "${CASE_OUTPUT}" != *"storage wait aborted"* ]] || \ + fail "a lone 401 aborted the wait with no Up store anywhere" +echo " PASS one refusing peer with no Up store retries" + +# The standalone RocksDB topology never reaches PD, so it must not warn about +# a PD credential it will not send. +: > "${DIST_ROOT}/conf/graphs/hugegraph.properties" +CASE_OUTPUT=$(env \ + PD_AUTH_PASSWORD="" \ + 'hugegraph.backend=rocksdb' \ + "${DIST_ROOT}/bin/wait-storage.sh" 2>&1) +CASE_RC=$? +assert_equal "no-PD topology rc" "0" "${CASE_RC}" +assert_output "No pd.peers configured, skipping storage wait" +[[ "${CASE_OUTPUT}" != *"PD_AUTH_PASSWORD is empty"* ]] || \ + fail "warned about an unused PD credential with no pd.peers configured" +echo " PASS no credential warning without pd.peers" + +echo "10 passed, 0 failed" diff --git a/hugegraph-store/README.md b/hugegraph-store/README.md index 4e6bc8aca2..f12df4427e 100644 --- a/hugegraph-store/README.md +++ b/hugegraph-store/README.md @@ -259,8 +259,8 @@ curl http://localhost:8520/v1/health # Check logs tail -f logs/hugegraph-store.log -# Verify registration with PD (from PD node) -curl http://localhost:8620/v1/stores +# Verify registration with PD (from PD node). PD REST needs HTTP Basic auth: an internal service name (hg, store, hubble, vermeer) and PD's auth.secret-key value. Without it this returns 401, not the store list. +curl -u hg:"${PD_SECRET}" http://localhost:8620/v1/stores ``` For production deployment, see [Deployment Guide](docs/deployment-guide.md) and [Best Practices](docs/best-practices.md). diff --git a/hugegraph-store/docs/deployment-guide.md b/hugegraph-store/docs/deployment-guide.md index 40c70b0e08..ca39ae509c 100644 --- a/hugegraph-store/docs/deployment-guide.md +++ b/hugegraph-store/docs/deployment-guide.md @@ -2,6 +2,14 @@ This guide provides comprehensive instructions for deploying HugeGraph Store in various environments, from development to production clusters. +> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than `/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, need HTTP Basic auth: one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) and PD's `auth.secret-key` value as the password. A call without it gets HTTP 401 and a `{"status":-1,"error":"Unauthorized"}` body, not the payloads shown below. Export the secret before following a step that uses `${PD_SECRET}`: +> +> ```bash +> read -rs PD_SECRET && export PD_SECRET +> ``` +> +> Store endpoints on port 8520 are unaffected. `-u` puts the secret in curl's process arguments; on a shared host pass it in a `curl -K` file mode 0600 instead, as `hugegraph-pd/docs/configuration.md` shows. + ## Table of Contents - [Deployment Topologies](#deployment-topologies) @@ -472,7 +480,7 @@ curl http://localhost:8620/actuator/health ```bash # Check cluster members -curl http://192.168.1.10:8620/v1/members +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/members # Expected output: { @@ -586,7 +594,7 @@ curl http://localhost:8520/v1/health ```bash # Query PD for registered stores -curl http://192.168.1.10:8620/v1/stores +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores # Expected output: { @@ -678,6 +686,10 @@ For a production-like 3-node distributed deployment, use the compose file at `do ```bash cd docker +# The PD REST secret is required; the Compose file refuses to start without it. Generate it once and keep it, every PD node and PD client needs the same value (docker/README.md has the full .env recipe). +export HG_PD_AUTH_SECRET_KEY="$(openssl rand -hex 24)" +# Hubble reads the secret from a generated, untracked properties file that the Compose file mounts; create it before `up` or Hubble starts unconfigured. +./set-hubble-pd-password.sh hstore-ha HUGEGRAPH_VERSION=1.7.0 docker compose -f docker-compose-3pd-3store-3server.yml up -d ``` @@ -695,8 +707,13 @@ environment: HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 # maps to pd.initial-store-list HG_PD_DATA_PATH: /hugegraph-pd/pd_data # maps to pd.data-path HG_PD_INITIAL_STORE_COUNT: 3 # maps to pd.initial-store-count + HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?} # maps to auth.secret-key; required + # optional; maps to management.endpoints.web.exposure.include + HG_PD_ACTUATOR_EXPOSURE: health,metrics,prometheus ``` +`HG_PD_ACTUATOR_EXPOSURE` is the only way to change the actuator allowlist in this image: the entrypoint emits it in `SPRING_APPLICATION_JSON`, which outranks a mounted `conf/application.yml`. Add an endpoint here to expose it, for example `health,metrics,prometheus,loggers`. A value containing `*` is refused, because every actuator endpoint is anonymous on port 8620 and `/actuator/env` returns the `SPRING_APPLICATION_JSON` entry verbatim, PD's REST secret included. + **Store environment variables** (per node): ```yaml @@ -723,10 +740,7 @@ environment: 2. Store nodes start after all PD nodes are healthy 3. Server nodes start after all Store nodes are healthy -`/v1/health` answers `200` as soon as the PD REST listener is up, so step 1 does -not wait for a raft quorum to form. PD also serves `/v1/ready`, which answers -`200` only while the PD sees a raft leader; `docker/README.md` covers what -pointing the healthchecks at it requires. +`/v1/health` answers `200` as soon as the PD REST listener is up, so step 1 does not wait for a raft quorum to form. PD also serves `/v1/ready`, which answers `200` only while the PD sees a raft leader; `docker/README.md` covers what pointing the healthchecks at it requires. > **Note**: The deprecated env var names (`GRPC_HOST`, `RAFT_ADDRESS`, `RAFT_PEERS`, `PD_ADDRESS`, `BACKEND`, `PD_PEERS`) still work but log a warning. Use the `HG_*` prefixed names for new deployments. @@ -870,28 +884,22 @@ curl -i http://192.168.1.10:8620/v1/ready curl http://192.168.1.20:8520/v1/health ``` -> **Note**: `/v1/ready` ships from the release after `1.7.0`, so the Docker -> examples above, which pin `HUGEGRAPH_VERSION=1.7.0`, need a newer tag or -> images built from source before this check means anything. On `1.7.0` the PD -> answers `200` with `{"status":-1,"error":"Unauthorized!"}` on any path its -> auth interceptor does not exclude, `/v1/ready` included, so match on the body -> rather than the status code. See -> [docker/README.md](../../docker/README.md) for the details. +> **Note**: `/v1/ready` ships from the release after `1.7.0`, so the Docker examples above, which pin `HUGEGRAPH_VERSION=1.7.0`, need a newer tag or images built from source before this check means anything. On `1.7.0` the PD answers `200` with `{"status":-1,"error":"Unauthorized!"}` on any path its auth interceptor does not exclude, `/v1/ready` included, so match on the body rather than the status code. See [docker/README.md](../../docker/README.md) for the details. ### Cluster Status ```bash # PD cluster members -curl http://192.168.1.10:8620/v1/members +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/members # Registered stores -curl http://192.168.1.10:8620/v1/stores +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores # Partitions -curl http://192.168.1.10:8620/v1/partitions +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitions # Graph list -curl http://192.168.1.10:8620/v1/graphs +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/graphs ``` ### Basic Operations Test diff --git a/hugegraph-store/docs/integration-guide.md b/hugegraph-store/docs/integration-guide.md index f35669c698..13164872e0 100644 --- a/hugegraph-store/docs/integration-guide.md +++ b/hugegraph-store/docs/integration-guide.md @@ -2,6 +2,14 @@ This guide explains how to integrate HugeGraph Store with HugeGraph Server, use the client library, and migrate from other storage backends. +> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than `/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, need HTTP Basic auth: one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) and PD's `auth.secret-key` value as the password. A call without it gets HTTP 401. Export the secret before following a procedure that uses `${PD_SECRET}`: +> +> ```bash +> read -rs PD_SECRET && export PD_SECRET +> ``` +> +> Store endpoints on port 8520 are unaffected. + ## Table of Contents - [Backend Configuration](#backend-configuration) @@ -706,7 +714,7 @@ tail -f logs/hugegraph-server.log | grep PD curl http://192.168.1.20:8520/v1/health # Check partition distribution -curl http://192.168.1.10:8620/v1/partitions +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitions # Check if queries are using indexes # (Enable query logging in Server) @@ -733,10 +741,10 @@ ERROR o.a.h.b.s.h.HstoreSession - Write operation failed: Raft leader not found tail -f logs/hugegraph-store.log | grep Raft # Check partition leaders -curl http://192.168.1.10:8620/v1/partitions | grep leader +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitions | grep leader # Check Store node states -curl http://192.168.1.10:8620/v1/stores +curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores ``` **Solutions**: diff --git a/hugegraph-store/docs/operations-guide.md b/hugegraph-store/docs/operations-guide.md index f46b5559d7..e94bbf1555 100644 --- a/hugegraph-store/docs/operations-guide.md +++ b/hugegraph-store/docs/operations-guide.md @@ -2,6 +2,14 @@ This guide covers monitoring, troubleshooting, backup & recovery, and operational procedures for HugeGraph Store in production. +> **PD REST credential.** Calls to a PD REST endpoint on port 8620, other than `/v1/health`, `/v1/ready`, `/actuator/**` and `/v1/prom/targets/*`, need HTTP Basic auth: one of the internal service names (`hg`, `store`, `hubble`, `vermeer`) and PD's `auth.secret-key` value as the password. A call without it gets HTTP 401, which for a mutating step such as `balanceLeaders` means the step did nothing. Export the secret before following a procedure that uses `${PD_SECRET}`: +> +> ```bash +> read -rs PD_SECRET && export PD_SECRET +> ``` +> +> Store endpoints on port 8520 are unaffected. Some PD examples in this guide still omit the credential; add `-u hg:"${PD_SECRET}"` when a call returns 401. + ## Table of Contents - [Monitoring and Metrics](#monitoring-and-metrics) @@ -604,19 +612,19 @@ curl http://192.168.1.10:8620/v1/partitionsAndStatus 2. **Verify Registration**: ```bash - curl http://192.168.1.10:8620/v1/stores + curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/stores # New Store should appear ``` 3. **Trigger Rebalancing** (optional): ```bash - curl -X POST http://192.168.1.10:8620/v1/balanceLeaders + curl -u hg:"${PD_SECRET}" -X POST http://192.168.1.10:8620/v1/balanceLeaders ``` 4. **Monitor Rebalancing**: ```bash # Watch partition distribution - watch -n 10 'curl http://192.168.1.10:8620/v1/partitionsAndStatus' + watch -n 10 'curl -u hg:"${PD_SECRET}" http://192.168.1.10:8620/v1/partitionsAndStatus' ``` 5. **Verify**: Wait for even distribution (may take hours)