Feature: add pg_query_state to gp_stats_collector - #1934
Conversation
There was a problem hiding this comment.
Any possibility that we could integrate this into the existing CI workflow?
There was a problem hiding this comment.
Once the YAGPCC is donated to the ASF, the new project will be called cloudberry-command-center. So, is it better to align this new project name?
There was a problem hiding this comment.
We can also add the standard ASF license header for .md file.
| ================================================================================ | ||
| This product includes software derived from pg_query_state | ||
| (https://github.com/postgrespro/pg_query_state), under the PostgreSQL License: | ||
|
|
||
| Copyright (c) 2016-2025, Postgres Professional | ||
|
|
||
| Permission to use, copy, modify, and distribute this software and its | ||
| documentation for any purpose, without fee, and without a written agreement | ||
| is hereby granted, provided that the above copyright notice and this | ||
| paragraph and the following two paragraphs appear in all copies. | ||
|
|
||
| IN NO EVENT SHALL POSTGRES PROFESSIONAL BE LIABLE TO ANY PARTY FOR DIRECT, | ||
| INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST | ||
| PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN | ||
| IF POSTGRES PROFESSIONAL HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
|
||
| POSTGRES PROFESSIONAL SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT | ||
| NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A | ||
| PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, | ||
| AND POSTGRES PROFESSIONAL HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, | ||
| UPDATES, ENHANCEMENTS, OR MODIFICATIONS. | ||
|
|
There was a problem hiding this comment.
You can append it to the Apache Cloudberry includes codes from this file part.
|
It's the last part of functionality from #1085 - sent data about query execution progress to the outer agent. As in #1085 we sent signal to the all backend processes, they report progress, and external agent could aggregate it and show to the user. The whole idea is the same but implementation is quite differ since neither existing postgres extensions nor #1085 proposal is ready for production use in Apache Cloudberry. |
3 participants
What does this PR do?
Adds a signal-based runtime query-state facility to the
gp_stats_collectorextension. It lets a session inspect the live execution state of another running backend on demand - walking its active plan tree across the QD and all QEs - without waiting for the query to finish, pushing batches to the UDS(unix domain socket).New SQL API (extension v1.2, schema
gpsc):gpsc.pg_query_state(pid, trace_id)- fan out a poll to the query running onpid; each participating backend walks its plan tree and logs a per-node snapshot.gpsc.pg_query_state_backends(pid)- list the(segid, pid)QE backends taking part in that query.cbdb_mpp_query_state(gpsc.gp_segment_pid[], trace_id)- QE-side dispatch target.The extension embeds the
pg_query_statesignal layer, which depends on three PostgreSQL core changes folded directly into the tree (configure enables the extension by default, so the tree must build without a manual patch step):ProcSignalhandlers (procsignal.c/.h,postgres.c);instrument.c/.h);EXPLAINentry points (explain.c/.h).Type of Change
Breaking Changes
None. Core changes are additive (new signal reasons, a new Instrumentation field, new runtime-EXPLAIN paths); existing behavior is unchanged.
Test Plan
Unit tests added/updated
Integration tests added/updated
Passed
make installcheckpg_regress (
gpcontrib/gp_stats_collector/test): catalog contract (function/type registration, exec location) and input-validation errors.isolation2 (
gpcontrib/gp_stats_collector/test/isolation2), multi-session:gpsc_pqs_backends- idle backend yields an empty list;gpsc_pqs_running- happy path: poll a query suspended on the QEs via a fault injector;gpsc_pqs_perms- permission gate (non-owner non-superuser denied);gpsc_pqs_disabled-STAT_DISABLEDreports no backends;gpsc_pqs_seg_count- one backend per primary segment.gpsc-crash-test- parallel tracing onmake installcheck-parallel, core dumps checkingA dedicated CI workflow builds Cloudberry with the extension across ubuntu22.04/rocky8/rocky9 and runs the pg_regress, isolation2, and core regression suites.
Impact
Performance:
No steady-state cost. Work happens only when a poll is issued: one signal fan-out plus a bounded shared-memory round-trip; the shmem lock is held only around the request, not during dispatch.
User-facing changes:
New
gpsc.pg_query_state*functions and thepg_query_state.enableGUC. Functions are granted to PUBLIC; access is gated in C so a caller may poll a backend only if it is a superuser or owns the target query.Dependencies:
None.
Checklist
Additional Context
The signal layer under
src/pg_query_state/is derived from pg_query_state (PostgreSQL License). Derived files carry dual license headers (ASF +Portions Copyright Postgres Professional) and the rootLICENSErecords a PostgreSQL-License notice for it. Happy to provide provenance details for IP clearance.