diff --git a/docker/README.md b/docker/README.md index 8ed7ea8faf..6fcfdee425 100644 --- a/docker/README.md +++ b/docker/README.md @@ -202,6 +202,32 @@ done curl -fsS http://localhost:8088/about ``` +PD answers two unauthenticated probe endpoints. `/v1/health` is liveness only: +it returns `200` as soon as the REST listener is up, even when the PD has no +raft leader. `/v1/ready` returns `200` only while the PD sees a raft leader, +and `503` otherwise. Each PD answers for itself: a single PD elects itself, and +in a three-PD group the two that can reach each other elect a leader and turn +ready, while a partitioned third keeps answering `503` until it sees that +leader. + +The healthchecks in these files still gate on `/v1/health`, because +`/v1/ready` ships from the next release onwards while the files run published +images. Two things to know before pointing them at readiness: + +- Match on the body, not the status code. As of 1.7.0 PD answers `200` with + `{"status":-1,"error":"Unauthorized!"}` on every path its auth interceptor + does not exclude, a path that does not exist included, so a status-only + probe reads a PD too old to have `/v1/ready` as ready. The body match holds + whichever status a refusal carries. Gate with + `curl -fsS http://localhost:8620/v1/ready | grep -q '"ready":true'` instead. +- Pin `HUGEGRAPH_VERSION` to a release that carries the endpoint, or build the + images from source with `docker-compose.dev.yml`. + +The `HEALTHCHECK` baked into `hugegraph-pd/Dockerfile` is `/v1/health` as well. +Both compose files override it, so it governs `docker run` and anything else +inheriting the image probe, and those keep reading a PD without a quorum as +healthy. + Open `http://localhost:8088` and sign in as `admin` with the password from `.env`. diff --git a/hugegraph-pd/README.md b/hugegraph-pd/README.md index 794dba9b98..c940dae4c9 100644 --- a/hugegraph-pd/README.md +++ b/hugegraph-pd/README.md @@ -284,6 +284,8 @@ docker/docker-compose-3pd-3store-3server.yml PD exposes metrics via REST API at: - Health check: `http://:8620/actuator/health` +- Liveness: `http://:8620/v1/health` (REST listener is up) +- Readiness: `http://:8620/v1/ready` (`200` only while the PD sees a raft leader) - Metrics: `http://:8620/actuator/metrics` ## Community diff --git a/hugegraph-pd/docs/api-reference.md b/hugegraph-pd/docs/api-reference.md index aa8cce8473..58b9698862 100644 --- a/hugegraph-pd/docs/api-reference.md +++ b/hugegraph-pd/docs/api-reference.md @@ -774,6 +774,56 @@ curl http://localhost:8620/actuator/health } ``` +### Liveness and Readiness + +Two unauthenticated endpoints are meant for probes and startup gates: + +| Endpoint | Meaning | Status | +|----------|---------|--------| +| `GET /v1/health` | Liveness: the REST listener is up. Does not consult raft. | always `200` | +| `GET /v1/ready` | Readiness: the raft node is active and sees a leader, so this PD is inside a quorum. | `200` when ready, `503` otherwise | + +```bash +curl -i http://localhost:8620/v1/ready +``` + +**Response** (leader of a healthy cluster): +```json +{ + "ready": true, + "state": "STATE_LEADER", + "isLeader": true +} +``` + +A follower reports `"state": "STATE_FOLLOWER"` with `"isLeader": false`. When +the quorum is lost the PD keeps answering `/v1/health` with `200` but +`/v1/ready` turns into `503` with `"ready": false`. Being unauthenticated, the +body carries no cluster addresses; the leader's address stays on `/v1/members`. + +The answer is served from state the raft callbacks maintain rather than from +the raft node, so it stays prompt while an election is running and never waits +on the node lock. `state` is therefore the last change raft announced. A PD +reports `STATE_UNINITIALIZED` with `"ready": false` from process start until +its first raft callback, which is the ordinary startup window before a quorum +first forms, and jraft emits no callback for candidacy or leadership transfer, +so a candidate reports `STATE_FOLLOWER` with `"ready": false`. + +Point Kubernetes readiness probes, `depends_on` healthchecks and any +"wait for PD" script at `/v1/ready`; keep liveness probes on `/v1/health` +so a PD that merely lost its leader is not restarted. + +Match on the body rather than on the status code alone. A PD that predates this +endpoint does not reliably answer `404` for it: `RestAuthentication` refuses a +request it does not exclude by writing an error envelope, and as of 1.7.0 it +does so without setting a status, so an unknown path answers `200` with +`{"status":-1,"error":"Unauthorized!"}`. A status-only probe therefore reads +such a PD as ready. The body match holds whichever status a refusal carries: a +shell gate should use +`curl -fsS http://:8620/v1/ready | grep -q '"ready":true'`, and a +Kubernetes `httpGet` probe should be paired with a PD image that carries the +endpoint. + ### Metrics ```bash @@ -796,6 +846,30 @@ pd_store_count{state="Offline"} 0.0 pd_partition_count 36.0 ``` +#### Raft membership gauges + +Exported on `/actuator/prometheus` for alerting on quorum loss: + +| Gauge | Value | +|-------|-------| +| `hg_raft_leader` | `1` on the raft leader, `0` elsewhere | +| `hg_raft_has_leader` | `1` while this PD sees a leader (is inside a quorum), `0` otherwise | +| `hg_raft_alive_peers` | Number of alive peers on the leader, itself included; `NaN` elsewhere | + +`hg_raft_alive_peers` counts the peers the leader has heard from within the +leader lease timeout, which jraft derives as 90% of the election timeout by +default. + +A cluster has lost its quorum when `sum(hg_raft_leader) == 0` or when +`hg_raft_has_leader == 0` on every member. Both are briefly true during a +normal election, so alert on them with a `for:` clause longer than the +election timeout rather than on the instantaneous value. + +Do not aggregate `hg_raft_alive_peers` across instances: it is `NaN` on every +node but the leader, and one `NaN` sample turns the result of `sum()` or +`avg()` into `NaN` as well. Select the leader's series instead, for example +`hg_raft_alive_peers and on(instance) (hg_raft_leader == 1)`. + ### Partition API #### List Partitions diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java index 314c9e57ef..d39384909d 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftEngine.java @@ -27,6 +27,8 @@ import java.util.concurrent.CompletableFuture; import java.util.concurrent.CountDownLatch; import java.util.concurrent.ExecutionException; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; import java.util.concurrent.atomic.AtomicReference; @@ -46,6 +48,7 @@ import com.alipay.sofa.jraft.Status; import com.alipay.sofa.jraft.conf.Configuration; import com.alipay.sofa.jraft.core.Replicator; +import com.alipay.sofa.jraft.core.State; import com.alipay.sofa.jraft.entity.PeerId; import com.alipay.sofa.jraft.entity.Task; import com.alipay.sofa.jraft.error.RaftError; @@ -63,14 +66,24 @@ @Slf4j public class RaftEngine { + /** + * Refresh period of the alive peer count behind {@code hg_raft_alive_peers}. jraft's own + * step-down timer walks the same peer set every half election timeout, so a one second + * poll adds nothing next to the work the node already does, and it keeps the gauge fresh + * well inside any scrape interval. + */ + private static final long ALIVE_PEERS_REFRESH_MS = 1000L; + private volatile static RaftEngine instance = new RaftEngine(); private RaftStateMachine stateMachine; private String groupId = "pd_raft"; private PDConfig.Raft config; private RaftGroupService raftGroupService; private RpcServer rpcServer; - private Node raftNode; + private volatile Node raftNode; private RaftRpcClient raftRpcClient; + private volatile int alivePeerCount = -1; + private ScheduledExecutorService alivePeersRefresher; public RaftEngine() { this.stateMachine = new RaftStateMachine(); @@ -133,6 +146,7 @@ public synchronized boolean init(PDConfig.Raft config) { this.raftGroupService = new RaftGroupService(groupId, serverId, nodeOptions, rpcServer, true); this.raftNode = raftGroupService.start(false); + startAlivePeersRefresher(); log.info("RaftEngine start successfully: id = {}, peers list = {}", groupId, nodeOptions.getInitialConf().getPeers()); return this.raftNode != null; @@ -182,6 +196,24 @@ public List backChannelHandlers() { } public void shutDown() { + if (this.alivePeersRefresher != null) { + this.alivePeersRefresher.shutdownNow(); + try { + // Best effort: shutdownNow only interrupts, and a refresh parked in + // listAlivePeers waits on a lock acquire the interrupt does not break, + // for up to the raft rpc connect timeout per unreachable peer. A refresh + // that outlives this wait may publish one stale count over the reset + // below; acceptable while shutDown has no production caller. + if (!this.alivePeersRefresher.awaitTermination(1, TimeUnit.SECONDS)) { + log.warn("Raft alive-peers refresher still running after shutdown; " + + "hg_raft_alive_peers may briefly report a stale value"); + } + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + this.alivePeersRefresher = null; + } + this.alivePeerCount = -1; if (this.raftGroupService != null) { this.raftGroupService.shutdown(); try { @@ -203,7 +235,139 @@ public void shutDown() { } public boolean isLeader() { - return this.raftNode.isLeader(true); + Node node = this.raftNode; + return node != null && node.isLeader(true); + } + + /** + * Whether this PD is ready in the sense {@code GET /v1/ready} answers: the raft node + * is active and sees a leader. + *

+ * A follower only keeps its leader while heartbeats keep arriving inside the election + * timeout, and a leader only keeps its role while it can reach a quorum. Seeing a leader + * therefore means this node is part of a quorum from its own point of view, which is the + * signal a readiness probe needs. Served from the state machine callbacks, not from the + * raft node, so it never waits on the node lock. + *

+ * Derived from {@link #getRaftStatus()} so {@code hg_raft_has_leader} cannot drift from + * the endpoint it is documented to mirror. + */ + public boolean hasLeader() { + return getRaftStatus().isReady(); + } + + /** + * Take a view of the local raft state from the volatile copies the state machine + * callbacks maintain, never from the raft node itself. During an election jraft holds + * the node lock while it reconnects to peers, so a probe that read the node stalled for + * the connect timeout instead of answering its 503 promptly. All fields derive from one + * read of one immutable view, swapped in whole per callback, so they cannot contradict + * each other. + *

+ * The state reported is the last one a callback announced: leader, follower, error or + * shutdown, and {@code STATE_UNINITIALIZED} until the first callback runs. jraft emits + * no callback for candidacy or leadership transfer, so a candidate reads as a follower + * that sees no leader, which yields the same not-ready answer. The view trails the node + * by whatever sits in the FSM queue ahead of the announcement, which is the price of + * never waiting on the node lock. + *

+ * A missing raft node needs no separate branch: before {@link #init} the view still + * holds its initial {@code STATE_UNINITIALIZED}, and {@code shutDown()} drops the node + * only after {@code join()} has let the shutdown callback announce + * {@code STATE_SHUTDOWN}. Neither state is active, so neither reads as ready, and a + * branch on the node would report an uninitialized PD where the callback said error or + * shutdown. + */ + public RaftStatus getRaftStatus() { + RaftStateMachine.ProbeView view = this.stateMachine.getProbeView(); + return new RaftStatus(view.state.isActive() && view.seesLeader, + view.state.name(), State.STATE_LEADER == view.state); + } + + /** + * Immutable view of the raft state behind {@code GET /v1/ready}. It carries no cluster + * addresses: the endpoint is unauthenticated, and the leader's address stays on the + * authenticated {@code /v1/members}. + */ + public static final class RaftStatus { + + private final boolean ready; + private final String state; + private final boolean localLeader; + + RaftStatus(boolean ready, String state, boolean localLeader) { + this.ready = ready; + this.state = state; + this.localLeader = localLeader; + } + + public boolean isReady() { + return this.ready; + } + + /** + * @return the jraft node state name, never null + */ + public String getState() { + return this.state; + } + + public boolean isLocalLeader() { + return this.localLeader; + } + } + + /** + * Number of raft peers, this node included, that the leader has heard from within the + * leader lease timeout, which jraft derives as 90% of the election timeout by default. + * Only the leader tracks replication state, so any other node reports -1. + *

+ * This is the value the refresher last published, at most one refresh period old, and -1 + * until the first refresh runs. The count cannot be read here: {@code listAlivePeers} + * takes the node read lock before it checks for leadership, so a caller that read the + * node would wait out whoever holds the write lock, which during an election is a + * per-peer reconnect bounded only by the rpc connect timeout. + */ + public int getAlivePeerCount() { + return this.alivePeerCount; + } + + private void startAlivePeersRefresher() { + ScheduledExecutorService refresher = + Executors.newSingleThreadScheduledExecutor(runnable -> { + Thread thread = new Thread(runnable, "pd-raft-alive-peers"); + thread.setDaemon(true); + return thread; + }); + refresher.scheduleWithFixedDelay(this::refreshAlivePeerCount, 0, + ALIVE_PEERS_REFRESH_MS, TimeUnit.MILLISECONDS); + this.alivePeersRefresher = refresher; + } + + /** + * Read the alive peer count from the raft node and publish it for the gauge. Runs on the + * refresher thread, never on a request thread, because the read can block for as long as + * an election holds the node write lock. A fixed delay schedule means a blocked refresh + * only delays the next one, and the gauge keeps reporting the last value meanwhile. + */ + void refreshAlivePeerCount() { + Node node = this.raftNode; + if (node == null || !this.stateMachine.isLeader()) { + this.alivePeerCount = -1; + return; + } + try { + this.alivePeerCount = node.listAlivePeers().size(); + } catch (IllegalStateException e) { + // Lost leadership between the check and the call + this.alivePeerCount = -1; + } catch (Throwable e) { + // scheduleWithFixedDelay cancels every later run if the task throws, + // and an Error escaping here would leave the gauge serving its last + // value forever, so this catch has to be wider than Exception. + log.warn("Failed to refresh the raft alive peer count", e); + this.alivePeerCount = -1; + } } /** @@ -232,7 +396,8 @@ public PDConfig.Raft getConfig() { } public PeerId getLeader() { - return raftNode.getLeaderId(); + Node node = this.raftNode; + return node == null ? null : node.getLeaderId(); } /** diff --git a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java index 6fad3347fa..aab90e5331 100644 --- a/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java +++ b/hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/raft/RaftStateMachine.java @@ -36,6 +36,7 @@ import com.alipay.sofa.jraft.Iterator; import com.alipay.sofa.jraft.Status; import com.alipay.sofa.jraft.conf.Configuration; +import com.alipay.sofa.jraft.core.State; import com.alipay.sofa.jraft.core.StateMachineAdapter; import com.alipay.sofa.jraft.entity.LeaderChangeContext; import com.alipay.sofa.jraft.entity.LocalFileMetaOutter; @@ -56,6 +57,27 @@ public class RaftStateMachine extends StateMachineAdapter { private static final String SNAPSHOT_DIR_NAME = "snapshot"; private static final String SNAPSHOT_ARCHIVE_NAME = "snapshot.zip"; private final AtomicLong leaderTerm = new AtomicLong(-1); + // Kept for the readiness probe and the raft gauges, which must answer without + // touching the raft node: during an election jraft holds the node lock while it + // reconnects to peers, and a reader stalls for the connect timeout. One volatile + // immutable pair, swapped in whole per callback, so a reader can never see the + // state of one announcement with the leader visibility of another. + private volatile ProbeView probeView = new ProbeView(State.STATE_UNINITIALIZED, false); + + /** + * Immutable pair of the last announced node state and leader visibility. Written only + * from the FSM callback thread; both fields always come from the same announcement. + */ + static final class ProbeView { + + final State state; + final boolean seesLeader; + + ProbeView(State state, boolean seesLeader) { + this.state = state; + this.seesLeader = seesLeader; + } + } private List taskHandlers; private List stateListeners; @@ -76,6 +98,15 @@ public boolean isLeader() { return this.leaderTerm.get() > 0; } + /** + * The last state and leader visibility a raft callback announced, as one immutable + * view. jraft emits no callback for candidacy or leadership transfer, so a candidate + * reads as a follower that sees no leader. + */ + ProbeView getProbeView() { + return this.probeView; + } + @Override public void onApply(Iterator iter) { while (iter.hasNext()) { @@ -105,17 +136,24 @@ public void onApply(Iterator iter) { @Override public void onError(final RaftException e) { + // A node that errors after it was leader keeps a positive leaderTerm otherwise, and + // the alive-peer refresher would go on reading listAlivePeers off a broken node + this.leaderTerm.set(-1); + this.probeView = new ProbeView(State.STATE_ERROR, false); log.error("Raft StateMachine on error {}", e); } @Override public void onShutdown() { + this.leaderTerm.set(-1); + this.probeView = new ProbeView(State.STATE_SHUTDOWN, false); super.onShutdown(); } @Override public void onLeaderStart(final long term) { this.leaderTerm.set(term); + this.probeView = new ProbeView(State.STATE_LEADER, true); super.onLeaderStart(term); log.info("Raft becomes leader"); @@ -129,12 +167,14 @@ public void onLeaderStart(final long term) { @Override public void onLeaderStop(final Status status) { this.leaderTerm.set(-1); + this.probeView = new ProbeView(State.STATE_FOLLOWER, false); super.onLeaderStop(status); log.info("Raft lost leader "); } @Override public void onStartFollowing(final LeaderChangeContext ctx) { + this.probeView = new ProbeView(State.STATE_FOLLOWER, true); super.onStartFollowing(ctx); Utils.runInThread(() -> { if (!CollectionUtils.isEmpty(stateListeners)) { @@ -145,6 +185,8 @@ public void onStartFollowing(final LeaderChangeContext ctx) { @Override public void onStopFollowing(final LeaderChangeContext ctx) { + // Callbacks run on the single FSM thread, so reading our own field is safe + this.probeView = new ProbeView(this.probeView.state, false); super.onStopFollowing(ctx); } diff --git a/hugegraph-pd/hg-pd-service/pom.xml b/hugegraph-pd/hg-pd-service/pom.xml index ee78863f35..eef38b4073 100644 --- a/hugegraph-pd/hg-pd-service/pom.xml +++ b/hugegraph-pd/hg-pd-service/pom.xml @@ -162,6 +162,12 @@ log4j-jul 2.17.2 + + junit + junit + ${junit.version} + test + diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java index 483974a016..793aa24b05 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java @@ -33,6 +33,7 @@ import org.apache.hugegraph.pd.grpc.Metapb; import org.apache.hugegraph.pd.grpc.Metapb.ShardGroup; import org.apache.hugegraph.pd.model.GraphStatistics; +import org.apache.hugegraph.pd.raft.RaftEngine; import org.apache.hugegraph.pd.service.PDRestService; import org.apache.hugegraph.pd.service.PDService; @@ -76,7 +77,28 @@ private void registerMeters() { Gauge.builder(PREFIX + ".terms", () -> setTerms()) .description("term of partitions in PD") .register(registry); + registerRaftMeters(); + } + /** + * Raft membership gauges so operators can alert on quorum loss. They mirror what + * {@code GET /v1/ready} answers: a PD that sees no leader is outside a quorum. + */ + private void registerRaftMeters() { + RaftEngine raft = RaftEngine.getInstance(); + Gauge.builder(PREFIX + ".raft.leader", () -> raft.getRaftStatus().isLocalLeader() ? 1 : 0) + .description("1 if this PD is the raft leader, 0 otherwise") + .register(registry); + Gauge.builder(PREFIX + ".raft.has.leader", () -> raft.hasLeader() ? 1 : 0) + .description("1 if this PD sees a raft leader, i.e. is part of a quorum, 0 otherwise") + .register(registry); + Gauge.builder(PREFIX + ".raft.alive.peers", () -> { + int alive = raft.getAlivePeerCount(); + return alive < 0 ? Double.NaN : alive; + }) + .description("Number of raft peers, itself included, the leader has heard from " + + "within the leader lease timeout; NaN on non-leader nodes") + .register(registry); } private long updateGraphs() { diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java index 2cddb29feb..b3c30ca9e1 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/StoreAPI.java @@ -22,6 +22,7 @@ import java.util.Date; import java.util.HashMap; import java.util.HashSet; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -31,11 +32,14 @@ import org.apache.hugegraph.pd.model.RestApiResponse; import org.apache.hugegraph.pd.model.StoreRestRequest; import org.apache.hugegraph.pd.model.TimeRangeRequest; +import org.apache.hugegraph.pd.raft.RaftEngine; import org.apache.hugegraph.pd.service.PDRestService; import org.apache.hugegraph.pd.util.DateUtil; import org.apache.hugegraph.pd.util.StoreRestAddressUtil; import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.http.HttpStatus; import org.springframework.http.MediaType; +import org.springframework.http.ResponseEntity; import org.springframework.web.bind.annotation.DeleteMapping; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PathVariable; @@ -378,6 +382,10 @@ class StoreStatistics { * Check Service Health Status * This interface is used to check the health status of the service by accessing the /health * path via a GET request. + *

+ * This is a liveness signal only: it answers 200 as soon as the REST listener is up and + * does not consult the raft state. Use {@link #checkReady()} to find out whether this PD + * can actually serve. * * @return Returns a string indicating the service's health status. Typically, an empty * string indicates the service is healthy. @@ -386,4 +394,27 @@ class StoreStatistics { public Serializable checkHealthy() { return ""; } + + /** + * Check Service Readiness + * Answers 200 only when this PD is part of a raft quorum, that is, the raft node is active + * and knows the current leader. Otherwise answers 503 so that anything gating on PD + * (the compose healthcheck in front of Stores, a Kubernetes readiness probe) + * is held back until the PD can serve. Like /health this endpoint needs no authentication. + * + * @return JSON with the readiness flag, the local raft state and whether this node is the + * leader. It carries no cluster addresses, since the endpoint is unauthenticated. + */ + @GetMapping(value = "/ready", produces = MediaType.APPLICATION_JSON_VALUE) + public ResponseEntity> checkReady() { + RaftEngine.RaftStatus status = RaftEngine.getInstance().getRaftStatus(); + + Map body = new LinkedHashMap<>(); + body.put("ready", status.isReady()); + body.put("state", status.getState()); + body.put("isLeader", status.isLocalLeader()); + HttpStatus httpStatus = status.isReady() ? HttpStatus.OK + : HttpStatus.SERVICE_UNAVAILABLE; + return ResponseEntity.status(httpStatus).body(body); + } } diff --git a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java index 7d10416967..d4b1e026d3 100644 --- a/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java +++ b/hugegraph-pd/hg-pd-service/src/main/java/org/apache/hugegraph/pd/rest/interceptor/AuthenticationConfigurer.java @@ -32,6 +32,7 @@ public class AuthenticationConfigurer implements WebMvcConfigurer { public void addInterceptors(InterceptorRegistry registry) { registry.addInterceptor(restAuthentication) .addPathPatterns("/**") - .excludePathPatterns("/actuator/*", "/v1/health", "/v1/prom/targets/*"); + .excludePathPatterns("/actuator/*", "/v1/health", "/v1/ready", + "/v1/prom/targets/*"); } } diff --git a/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/rest/StoreAPIReadyTest.java b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/rest/StoreAPIReadyTest.java new file mode 100644 index 0000000000..75e9581cb1 --- /dev/null +++ b/hugegraph-pd/hg-pd-service/src/test/java/org/apache/hugegraph/pd/rest/StoreAPIReadyTest.java @@ -0,0 +1,144 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.pd.rest; + +import java.lang.reflect.Proxy; +import java.util.Map; + +import org.apache.hugegraph.pd.raft.RaftEngine; +import org.apache.hugegraph.pd.raft.RaftStateMachine; +import org.apache.hugegraph.testutil.Whitebox; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.springframework.http.HttpStatus; +import org.springframework.http.ResponseEntity; + +import com.alipay.sofa.jraft.Node; +import com.alipay.sofa.jraft.Status; +import com.alipay.sofa.jraft.core.State; +import com.alipay.sofa.jraft.entity.LeaderChangeContext; +import com.alipay.sofa.jraft.entity.PeerId; + +/** + * Pins the status a readiness probe actually receives from {@code GET /v1/ready}: 503 while + * this PD is outside a raft quorum, 200 once it is inside one, with the same body either way. + * The live REST suite can only reach the 200 path, because the PD it talks to is a single node + * group that is always its own leader. + */ +public class StoreAPIReadyTest { + + private static final PeerId LEADER = new PeerId("10.0.0.1", 8610); + + private final StoreAPI api = new StoreAPI(); + + private Node originalRaftNode; + private RaftStateMachine originalStateMachine; + private RaftStateMachine stateMachine; + + @Before + public void setUp() { + RaftEngine engine = RaftEngine.getInstance(); + this.originalRaftNode = engine.getRaftNode(); + this.originalStateMachine = Whitebox.getInternalState(engine, "stateMachine"); + + // A fresh machine so callbacks fired here reach no listeners of the real one + this.stateMachine = new RaftStateMachine(); + Whitebox.setInternalState(engine, "stateMachine", this.stateMachine); + } + + @After + public void tearDown() { + RaftEngine engine = RaftEngine.getInstance(); + Whitebox.setInternalState(engine, "raftNode", this.originalRaftNode); + Whitebox.setInternalState(engine, "stateMachine", this.originalStateMachine); + } + + @Test + public void testNotReadyBeforeRaftStartsAnswers503() { + Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", null); + + ResponseEntity> response = this.api.checkReady(); + + Assert.assertEquals(HttpStatus.SERVICE_UNAVAILABLE, response.getStatusCode()); + Map body = response.getBody(); + Assert.assertNotNull(body); + Assert.assertEquals(Boolean.FALSE, body.get("ready")); + Assert.assertEquals(Boolean.FALSE, body.get("isLeader")); + Assert.assertEquals(State.STATE_UNINITIALIZED.name(), body.get("state")); + } + + @Test + public void testLosingTheLeaderAnswers503() { + Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", untouchableNode()); + // The quorum-loss shape: heartbeats arrive, then they stop + this.stateMachine.onStartFollowing(context()); + this.stateMachine.onStopFollowing(context()); + + ResponseEntity> response = this.api.checkReady(); + + Assert.assertEquals(HttpStatus.SERVICE_UNAVAILABLE, response.getStatusCode()); + Map body = response.getBody(); + Assert.assertNotNull(body); + Assert.assertEquals(Boolean.FALSE, body.get("ready")); + Assert.assertEquals(Boolean.FALSE, body.get("isLeader")); + Assert.assertEquals(State.STATE_FOLLOWER.name(), body.get("state")); + } + + @Test + public void testLeaderAnswers200() { + Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", untouchableNode()); + this.stateMachine.onLeaderStart(5); + + ResponseEntity> response = this.api.checkReady(); + + Assert.assertEquals(HttpStatus.OK, response.getStatusCode()); + Map body = response.getBody(); + Assert.assertNotNull(body); + Assert.assertEquals(Boolean.TRUE, body.get("ready")); + Assert.assertEquals(Boolean.TRUE, body.get("isLeader")); + Assert.assertEquals(State.STATE_LEADER.name(), body.get("state")); + } + + private static LeaderChangeContext context() { + return new LeaderChangeContext(LEADER, 5, Status.OK()); + } + + /** + * A started raft node that fails the test if the endpoint calls it. Readiness is served + * from the state machine callbacks precisely so that it never waits on the node lock. + */ + private static Node untouchableNode() { + return (Node) Proxy.newProxyInstance( + Node.class.getClassLoader(), new Class[]{Node.class}, + (proxy, method, args) -> { + switch (method.getName()) { + case "toString": + return "untouchable raft node"; + case "hashCode": + return System.identityHashCode(proxy); + case "equals": + return proxy == args[0]; + default: + throw new AssertionError("the ready path must not touch the raft " + + "node, it called " + method.getName()); + } + }); + } +} diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java index 95b044c76b..bdacf7d371 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/core/PDCoreSuiteTest.java @@ -22,6 +22,7 @@ import org.apache.hugegraph.pd.raft.IpAuthHandlerTest; import org.apache.hugegraph.pd.raft.RaftEngineIpAuthIntegrationTest; import org.apache.hugegraph.pd.raft.RaftEngineLeaderAddressTest; +import org.apache.hugegraph.pd.raft.RaftEngineReadinessTest; import org.junit.runner.RunWith; import org.junit.runners.Suite; @@ -43,6 +44,7 @@ IpAuthHandlerTest.class, RaftEngineIpAuthIntegrationTest.class, RaftEngineLeaderAddressTest.class, + RaftEngineReadinessTest.class, // StoreNodeServiceTest.class, }) @Slf4j diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/raft/RaftEngineReadinessTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/raft/RaftEngineReadinessTest.java new file mode 100644 index 0000000000..08c35287d8 --- /dev/null +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/raft/RaftEngineReadinessTest.java @@ -0,0 +1,257 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.pd.raft; + +import java.util.Arrays; + +import org.apache.hugegraph.testutil.Whitebox; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import com.alipay.sofa.jraft.Node; +import com.alipay.sofa.jraft.Status; +import com.alipay.sofa.jraft.core.State; +import com.alipay.sofa.jraft.entity.LeaderChangeContext; +import com.alipay.sofa.jraft.entity.PeerId; +import com.alipay.sofa.jraft.error.RaftException; + +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +/** + * Covers the raft-aware readiness signal behind {@code GET /v1/ready} and the + * {@code hg.raft.*} gauges. The signal is served from the volatile copies the state machine + * callbacks maintain, never from the raft node, so the probe stays prompt while an election + * holds the node lock; these tests drive the callbacks the way jraft does. The alive peer + * count is the one value that has to come off the node, so it is refreshed explicitly here, + * the way the refresher thread does it in a running PD. + */ +public class RaftEngineReadinessTest { + + private static final PeerId LEADER = new PeerId("10.0.0.1", 8610); + + private Node originalRaftNode; + private RaftStateMachine originalStateMachine; + + private Node mockNode; + private RaftStateMachine stateMachine; + + @Before + public void setUp() { + RaftEngine engine = RaftEngine.getInstance(); + originalRaftNode = engine.getRaftNode(); + originalStateMachine = Whitebox.getInternalState(engine, "stateMachine"); + + // A fresh machine so callbacks fired here reach no listeners other tests registered + mockNode = mock(Node.class); + stateMachine = new RaftStateMachine(); + Whitebox.setInternalState(engine, "raftNode", mockNode); + Whitebox.setInternalState(engine, "stateMachine", stateMachine); + // The count is cached on the singleton, so clear what an earlier test published + engine.refreshAlivePeerCount(); + } + + @After + public void tearDown() { + RaftEngine engine = RaftEngine.getInstance(); + Whitebox.setInternalState(engine, "raftNode", originalRaftNode); + Whitebox.setInternalState(engine, "stateMachine", originalStateMachine); + } + + private static LeaderChangeContext ctx() { + return new LeaderChangeContext(LEADER, 5, Status.OK()); + } + + @Test + public void testNotReadyBeforeRaftNodeStarts() { + Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", null); + RaftEngine engine = RaftEngine.getInstance(); + engine.refreshAlivePeerCount(); + + RaftEngine.RaftStatus status = engine.getRaftStatus(); + Assert.assertFalse(status.isReady()); + Assert.assertFalse(status.isLocalLeader()); + Assert.assertEquals(State.STATE_UNINITIALIZED.name(), status.getState()); + Assert.assertFalse(engine.hasLeader()); + Assert.assertEquals(-1, engine.getAlivePeerCount()); + } + + @Test + public void testStartedNodeWithoutAnyCallbackIsNotReady() { + RaftEngine.RaftStatus status = RaftEngine.getInstance().getRaftStatus(); + Assert.assertFalse(status.isReady()); + Assert.assertEquals(State.STATE_UNINITIALIZED.name(), status.getState()); + } + + @Test + public void testLeaderIsReady() { + stateMachine.onLeaderStart(5); + when(mockNode.listAlivePeers()).thenReturn(Arrays.asList(LEADER, new PeerId("b", 1), + new PeerId("c", 1))); + RaftEngine engine = RaftEngine.getInstance(); + engine.refreshAlivePeerCount(); + + RaftEngine.RaftStatus status = engine.getRaftStatus(); + Assert.assertTrue(status.isReady()); + Assert.assertTrue(status.isLocalLeader()); + Assert.assertEquals(State.STATE_LEADER.name(), status.getState()); + Assert.assertTrue(engine.hasLeader()); + Assert.assertEquals(3, engine.getAlivePeerCount()); + } + + @Test + public void testFollowerWithLeaderIsReady() { + stateMachine.onStartFollowing(ctx()); + RaftEngine engine = RaftEngine.getInstance(); + engine.refreshAlivePeerCount(); + + RaftEngine.RaftStatus status = engine.getRaftStatus(); + Assert.assertTrue(status.isReady()); + Assert.assertFalse(status.isLocalLeader()); + Assert.assertEquals(State.STATE_FOLLOWER.name(), status.getState()); + Assert.assertTrue(engine.hasLeader()); + // Only the leader tracks replication, followers cannot count alive peers + Assert.assertEquals(-1, engine.getAlivePeerCount()); + } + + @Test + public void testFollowerLosingItsLeaderTurnsNotReady() { + // The quorum-loss shape from the issue: heartbeats stop, jraft announces the loss + stateMachine.onStartFollowing(ctx()); + stateMachine.onStopFollowing(ctx()); + RaftEngine engine = RaftEngine.getInstance(); + + RaftEngine.RaftStatus status = engine.getRaftStatus(); + Assert.assertFalse(status.isReady()); + Assert.assertEquals(State.STATE_FOLLOWER.name(), status.getState()); + Assert.assertFalse(engine.hasLeader()); + } + + @Test + public void testLeaderSteppingDownTurnsNotReady() { + stateMachine.onLeaderStart(5); + stateMachine.onLeaderStop(Status.OK()); + RaftEngine engine = RaftEngine.getInstance(); + engine.refreshAlivePeerCount(); + + RaftEngine.RaftStatus status = engine.getRaftStatus(); + Assert.assertFalse(status.isReady()); + Assert.assertFalse(status.isLocalLeader()); + Assert.assertFalse(engine.hasLeader()); + Assert.assertEquals(-1, engine.getAlivePeerCount()); + } + + @Test + public void testErrorAndShutdownAreNotReadyEvenAfterLeadership() { + stateMachine.onLeaderStart(5); + stateMachine.onError(mock(RaftException.class)); + Assert.assertFalse(RaftEngine.getInstance().getRaftStatus().isReady()); + Assert.assertEquals(State.STATE_ERROR.name(), + RaftEngine.getInstance().getRaftStatus().getState()); + + stateMachine.onShutdown(); + Assert.assertFalse(RaftEngine.getInstance().getRaftStatus().isReady()); + Assert.assertEquals(State.STATE_SHUTDOWN.name(), + RaftEngine.getInstance().getRaftStatus().getState()); + } + + @Test + public void testProbeNeverTouchesTheRaftNode() { + // The point of serving from callbacks: an election holds the node lock while jraft + // reconnects to peers, so the probe and the gauges must answer without the node. + // getAlivePeerCount() included: it reads what the refresher thread published + stateMachine.onStartFollowing(ctx()); + stateMachine.onStopFollowing(ctx()); + RaftEngine engine = RaftEngine.getInstance(); + + engine.getRaftStatus(); + engine.hasLeader(); + engine.getAlivePeerCount(); + + verifyNoInteractions(mockNode); + } + + @Test + public void testAlivePeerCountSurvivesLeadershipLossRace() { + stateMachine.onLeaderStart(5); + when(mockNode.listAlivePeers()).thenThrow(new IllegalStateException("Not leader")); + RaftEngine.getInstance().refreshAlivePeerCount(); + + Assert.assertEquals(-1, RaftEngine.getInstance().getAlivePeerCount()); + } + + @Test + public void testShutdownStateSurvivesTheNodeBeingDropped() { + // shutDown() drops the raft node after the shutdown callback ran, so the probe has + // to keep reporting what the callback announced rather than falling back to + // uninitialized, which would read as a PD that has not started yet + stateMachine.onLeaderStart(5); + stateMachine.onShutdown(); + Whitebox.setInternalState(RaftEngine.getInstance(), "raftNode", null); + RaftEngine engine = RaftEngine.getInstance(); + + RaftEngine.RaftStatus status = engine.getRaftStatus(); + Assert.assertFalse(status.isReady()); + Assert.assertFalse(status.isLocalLeader()); + Assert.assertEquals(State.STATE_SHUTDOWN.name(), status.getState()); + Assert.assertFalse(engine.hasLeader()); + } + + @Test + public void testAlivePeerCountStopsAfterALeaderErrorsOrShutsDown() { + // The terminal callbacks clear the leader term, so the refresher stops reading the + // count off a node that errored or shut down and the gauge falls back to NaN + stateMachine.onLeaderStart(5); + when(mockNode.listAlivePeers()).thenReturn(Arrays.asList(LEADER, new PeerId("b", 1))); + RaftEngine engine = RaftEngine.getInstance(); + engine.refreshAlivePeerCount(); + Assert.assertEquals(2, engine.getAlivePeerCount()); + + stateMachine.onError(mock(RaftException.class)); + engine.refreshAlivePeerCount(); + Assert.assertEquals(-1, engine.getAlivePeerCount()); + + stateMachine.onLeaderStart(6); + engine.refreshAlivePeerCount(); + Assert.assertEquals(2, engine.getAlivePeerCount()); + + stateMachine.onShutdown(); + engine.refreshAlivePeerCount(); + Assert.assertEquals(-1, engine.getAlivePeerCount()); + // Twice, once per leader phase: the terminal states never reached the node + verify(mockNode, times(2)).listAlivePeers(); + } + + @Test + public void testAlivePeerCountOnALeaderAlsoSkipsTheNode() { + // Even on a leader, where the count means something, the gauge reads the published + // value: listAlivePeers takes the node read lock before it checks for leadership, so + // a scrape that called it would wait out an election that holds the write lock + stateMachine.onLeaderStart(5); + RaftEngine engine = RaftEngine.getInstance(); + + engine.getAlivePeerCount(); + + verifyNoInteractions(mockNode); + } +} diff --git a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java index fb2b71d480..da90f6f0f9 100644 --- a/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java +++ b/hugegraph-pd/hg-pd-test/src/main/java/org/apache/hugegraph/pd/rest/RestApiTest.java @@ -25,6 +25,7 @@ import org.json.JSONException; import org.json.JSONObject; +import org.junit.Assert; import org.junit.Test; public class RestApiTest extends BaseServerTest { @@ -62,6 +63,62 @@ public void testQueryClusterInfo() throws URISyntaxException, IOException, Inter assert obj.getInt("status") == 0; } + @Test + public void testHealthNeedsNoAuth() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/v1/health"; + HttpRequest request = HttpRequest.newBuilder().uri(new URI(url)).GET().build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + Assert.assertEquals(200, response.statusCode()); + // A 200 alone does not prove the path is anonymous: as of 1.7.0 the auth interceptor + // refuses with 200 and an error envelope. checkHealthy() returns an empty body, which + // separates the two whichever status a refusal carries. + Assert.assertTrue("expected an empty body, got " + response.body(), + response.body().isEmpty()); + } + + @Test + public void testReadyNeedsNoAuthAndReflectsRaft() throws URISyntaxException, IOException, + InterruptedException, JSONException { + // The CI PD is a single-node raft group, so it is its own leader and must be ready + String url = pdRestAddr + "/v1/ready"; + HttpRequest request = HttpRequest.newBuilder().uri(new URI(url)).GET().build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + Assert.assertEquals("expected 200, body=" + response.body(), 200, response.statusCode()); + JSONObject obj = new JSONObject(response.body()); + Assert.assertTrue(obj.getBoolean("ready")); + Assert.assertTrue(obj.getBoolean("isLeader")); + Assert.assertEquals("STATE_LEADER", obj.getString("state")); + // Unauthenticated, so it must not disclose cluster addresses + Assert.assertFalse("the anonymous body must not carry the leader address", + obj.has("leader")); + } + + @Test + public void testRaftGaugesExported() throws URISyntaxException, IOException, + InterruptedException { + String url = pdRestAddr + "/actuator/prometheus"; + HttpRequest request = HttpRequest.newBuilder().uri(new URI(url)).GET().build(); + HttpResponse response = client.send(request, HttpResponse.BodyHandlers.ofString()); + Assert.assertEquals(200, response.statusCode()); + String body = response.body(); + // Micrometer only writes a {...} block when the meter carries tags, and the sample + // line is the one that starts with the metric name, unlike its HELP and TYPE lines + Assert.assertTrue("missing hg_raft_leader gauge", + body.matches("(?sm).*^hg_raft_leader(\\{[^}]*\\})? .*")); + Assert.assertTrue("missing hg_raft_has_leader gauge", + body.matches("(?sm).*^hg_raft_has_leader(\\{[^}]*\\})? .*")); + Assert.assertTrue("missing hg_raft_alive_peers gauge", + body.matches("(?sm).*^hg_raft_alive_peers(\\{[^}]*\\})? .*")); + // Single-node CI cluster: this PD is the leader and hears from itself + Assert.assertTrue("hg_raft_leader should be 1 on a single-node leader", + body.matches("(?sm).*^hg_raft_leader(\\{[^}]*\\})? 1\\.0.*")); + Assert.assertTrue("hg_raft_has_leader should be 1 on a single-node leader", + body.matches("(?sm).*^hg_raft_has_leader(\\{[^}]*\\})? 1\\.0.*")); + Assert.assertTrue("hg_raft_alive_peers should be 1 on a single-node leader", + body.matches("(?sm).*^hg_raft_alive_peers(\\{[^}]*\\})? 1\\.0.*")); + } + @Test public void testQueryClusterMembers() throws URISyntaxException, IOException, InterruptedException, JSONException { diff --git a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh index ab73255b8c..754e2baae1 100755 --- a/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh +++ b/hugegraph-server/hugegraph-dist/src/assembly/travis/test-start-hugegraph-pd.sh @@ -111,6 +111,27 @@ wait_for_pd() { return 1 } +# Wait until PD readiness answers, or timeout. Same gate the docs recommend: +# -f rejects the 503, the body match rejects a 200 that is an auth envelope rather +# than a readiness answer. No credentials, and a single-node group elects itself. +# Captured rather than piped: under pipefail a grep -q SIGPIPE could misread a +# ready PD, and wait_for_pd() above uses the same shape. The curl timeouts bound +# each attempt and the loop bounds itself on the wall clock, so a PD that accepts +# the connection and then stops answering cannot park this loop past STARTUP_WAIT. +# A per-round counter could not: an attempt costs up to --max-time plus the sleep, +# which is more than the counter would add. +wait_for_pd_ready() { + local deadline=$((SECONDS + STARTUP_WAIT)) + while (( SECONDS < deadline )); do + local body + body=$(curl --connect-timeout 2 --max-time 5 -fsS \ + "$PD_URL/v1/ready" 2>/dev/null || true) + grep -Eq '"ready"[[:space:]]*:[[:space:]]*true' <<<"$body" && return 0 + sleep 2 + done + return 1 +} + # Wait until bin/pid is non-empty or timeout wait_for_pid_file() { local elapsed=0 @@ -199,6 +220,13 @@ else fail "PD health endpoint not responding after ${STARTUP_WAIT}s" fi +info "Waiting up to ${STARTUP_WAIT}s for PD readiness endpoint..." +if wait_for_pd_ready; then + pass "PD readiness endpoint reports a raft leader at $PD_URL/v1/ready" +else + fail "PD readiness endpoint did not report ready=true after ${STARTUP_WAIT}s" +fi + cleanup # ── test 2: foreground mode blocks ─────────────────────────────────────────── diff --git a/hugegraph-store/docs/deployment-guide.md b/hugegraph-store/docs/deployment-guide.md index de07904d64..40c70b0e08 100644 --- a/hugegraph-store/docs/deployment-guide.md +++ b/hugegraph-store/docs/deployment-guide.md @@ -719,10 +719,15 @@ environment: ``` **Startup ordering** is enforced via `depends_on` with `condition: service_healthy`: -1. PD nodes start first and must pass healthchecks (`/v1/health`) +1. PD nodes start first and must pass healthchecks (`/v1/health`, liveness only) 2. Store nodes start after all PD nodes are healthy 3. Server nodes start after all Store nodes are healthy +`/v1/health` answers `200` as soon as the PD REST listener is up, so step 1 does +not wait for a raft quorum to form. PD also serves `/v1/ready`, which answers +`200` only while the PD sees a raft leader; `docker/README.md` covers what +pointing the healthchecks at it requires. + > **Note**: The deprecated env var names (`GRPC_HOST`, `RAFT_ADDRESS`, `RAFT_PEERS`, `PD_ADDRESS`, `BACKEND`, `PD_PEERS`) still work but log a warning. Use the `HG_*` prefixed names for new deployments. **Deploy**: @@ -855,13 +860,24 @@ kubectl port-forward svc/hugegraph-store 8500:8500 -n hugegraph ### Health Check ```bash -# PD health +# PD liveness (REST listener up) curl http://192.168.1.10:8620/v1/health +# PD readiness (200 only while the PD sees a raft leader, 503 otherwise) +curl -i http://192.168.1.10:8620/v1/ready + # Store health curl http://192.168.1.20:8520/v1/health ``` +> **Note**: `/v1/ready` ships from the release after `1.7.0`, so the Docker +> examples above, which pin `HUGEGRAPH_VERSION=1.7.0`, need a newer tag or +> images built from source before this check means anything. On `1.7.0` the PD +> answers `200` with `{"status":-1,"error":"Unauthorized!"}` on any path its +> auth interceptor does not exclude, `/v1/ready` included, so match on the body +> rather than the status code. See +> [docker/README.md](../../docker/README.md) for the details. + ### Cluster Status ```bash