clusterer_controller: zero-config HA clusterer control module with automatic sharing tag management via encrypted UDP multicast#4074
Conversation
|
Thank you very much for the contribution, I really like the idea behind it. I do see though that it is marked as a Draft - is it still work in progress, or has it reached to its final state? Let us know when it is ready to review. |
|
Hi, |
9034536 to
2cf5b36
Compare
Port of the clusterer_controller module and its clusterer-module integration to the devel branch: zero-config HA clusterer control over encrypted UDP multicast, per-cluster node identity, hybrid native+controller topologies, cl_ctr_* MI/variables/functions, join flood-DoS hardening, and input validation. Devel adaptations: - cc_discover_bin_sockets() iterates protos[PROTO_BIN].listeners as struct socket_info_full (next/prev), reading the embedded socket_info. - The clusterer core edits are re-seated onto devel's clusterer.c / clusterer_mod.c / node_info.h, merging cleanly with the new bridges feature and the clusterer_enable_rerouting guard. Robustness / log hygiene: - cc_decrypt_pkt distinguishes bootstrap-key failures (real wrong-password / foreign-cluster / tampering -> WARN) from transient session-key mismatches during a (re)key or split-brain heal (-> DBG). - the split-brain defer budget resets on a fresh higher-IP JOIN_REQ so a lower-IP node waits for a live higher-IP peer instead of self-promoting (bounded by CC_JOIN_DEFER_HARDMAX); JOIN_REQ sends are rate-limited. - clusterer: the seed sync-fallback on a fresh start is downgraded from ERROR to DBG (the SYNC_IN_PROGRESS partial-sync failure path is unchanged). - cc_elect_master enforces "MASTER_ALIVE keepalive armed <=> I am the elected master": a node demoted purely by election now stops broadcasting MASTER_ALIVE, fixing an oscillation where a lower-IP node flapped between two masters. Validated end-to-end on a 3-node cluster: controller / native / hybrid modes, master failover and election stability (staggered + simultaneous starts converge to a single stable master), multiple controller clusters sharing one BIN socket (distinct multicast per cluster), MI outputs and error paths, buffer overrun/underrun fuzzing, wrong-password rejection, and config-mismatch. Docs and generated README included (admin guide + HA test appendix).
2cf5b36 to
34f1042
Compare
|
Now it's ready for review, thanks! |
992d376 to
f93b82d
Compare
… consistency Two startup sanity checks for a previously silent misconfiguration: - clusterer_controller is only meaningful when the clusterer module has use_controller=1 (the global switch that pre-creates the controller-managed cluster stubs, sets each one's controller_managed flag so they never touch the DB, and arms the guard that stops the controller from hijacking a native cluster of the same id). If use_controller=0 there is no controller-managed cluster and those safety mechanisms are off, so mod_init now FAILS (refuse to start) rather than driving clusters clusterer never authorised. Hybrid setups keep use_controller=1 - only the per-cluster kind differs - so this never trips them. - The mirror case, use_controller=1 but no clusterer_controller module bound the controller API, logs an ERROR at clusterer child_init (the pre-created controller-managed stubs would never obtain an identity). clusterer does NOT abort here: its native/hybrid clusters still work; only the controller-stub clusters are dead. Plumbing: the clusterer_ctrl binds struct now carries clusterer's use_controller value, and load_clusterer_ctrl_binds() sets clusterer_ctrl_bound so clusterer can tell whether a controller registered. Verified on all permutations: pure controller, hybrid, native-only, and both mismatches.
f93b82d to
4348580
Compare
|
Follow-up commit Since this PR is already open for review, I added this as a distinct follow-up commit rather than squashing it into the main one, so the incremental change is easy to review and the existing review isn't disrupted by a force-push of the main commit. It enforces consistency between
Hybrid environments are unaffected. |
…ples
Reviewed all config examples so every cluster is clearly and self-containedly
defined, per reviewer feedback:
- Show modparam("clusterer","cluster_id",N) for each controller-managed cluster
in the full-config and multi-cluster examples (previously several relied on
the implicit auto-create-on-capability-registration path, so the cluster
never appeared "defined" in the snippet).
- Hybrid DB example: note that native cluster 10 is defined by rows in the
clusterer DB table (not a modparam), and fix the misleading "this node id in
cluster 10" comment (my_node_id is a single global id across all DB clusters).
- Add the missing "Hybrid, no DB" (static) example to the admin guide, with the
required db_mode=0 (without it my_node_info/neighbor_node_info are ignored).
- Comment use_controller as the global switch throughout.
No behavior change - documentation only. README regenerated.
Add a per-cluster 'cluster_options' modparam - the same "key=value, key=value"
idiom as my_node_info - so a cluster is marked controller-managed with:
modparam("clusterer", "cluster_options", "cluster_id=1, use_controller=1")
cluster_id is required; use_controller is a 0/1 flag defaulting to 0 (native),
and only use_controller=1 pre-creates the controller-managed stub (which never
touches the DB and is guarded against hijacking a native cluster of the same
id). Native clusters need no cluster_options line at all. Every other clusterer
setting (db_mode, ping_*, my_node_id, sharing_tag, ...) stays a global modparam.
The controller-managed ids and the clusterer_controller 'cluster' entries must
match EXACTLY. clusterer_controller aborts at startup, naming the offending id,
if either side references a cluster the other does not:
- a managed id with no 'cluster' config has no BIN socket or crypto params;
- a 'cluster' config for an unmanaged id has nothing legitimate to drive.
The controller loads the clusterer's controller-managed id set through the ctrl
binds (managed_count/managed_ids) to run this check pre-fork.
The interim 'use_controller'/'cluster_id' int modparams (never released) are
kept registered only to fail with a migration hint.
Docs (admin guide, tests appendix, README) rewritten to the cluster_options
form; every example gives each cluster an explicit definition.
82e123e to
b7a173e
Compare
|
Config API for controller-managed clusters is now the per-cluster Same The controller-managed ids and the Verified locally (the build links wolfSSL, so the controller runs without a node): the new syntax loads, a managed id with no controller config aborts, a controller config for an unmanaged id aborts, and matching config starts. Docs (admin guide, tests appendix, README) and the PR description are updated to this form. |
…ude clusterer_controller by default
Make clusterer_controller an opt-in module and ensure the bundled clusterer
module is completely unchanged when it is not built.
Build wiring:
- Add clusterer_controller to exclude_modules in Makefile.conf.template, so a
stock build skips it (like the other modules with external-lib deps). Enable
it via include_modules.
- The top-level Makefile exports CLUSTERER_CTRL_SUPPORT=1 iff clusterer_controller
is in the configured build (mirrors the module-selection rule, so it is stable
across 'make all' and single-module rebuilds); clusterer/Makefile turns that
into -DCLUSTERER_CTRL_SUPPORT.
clusterer side, all under #ifdef CLUSTERER_CTRL_SUPPORT:
- the clusterer_ctrl API (clusterer_ctrl.c) and the cluster_options /
use_controller / cluster_id modparams + load_clusterer_ctrl_binds export;
- the controller stub pre-create loop, the child_init guard, the shm current_id
mirror, the on-demand stub, and the shtag_managed / controller_managed logic.
- The per-cluster identity and hybrid-db_mode accessors (cluster_self_id,
cl_db_mode, GET_CURRENT_ID, use_controller) get #else fallbacks to the stock
globals (current_id / db_mode / 0), so their call sites compile to the exact
upstream object code with no per-site #ifdef. add_node_info's internal self_id
parameter is likewise gated (falls back to current_id).
Result: a build without clusterer_controller produces the stock clusterer module -
no cluster_options parameter (rejected as unknown), no behavioural change, no added
exports. Verified with unifdef -UCLUSTERER_CTRL_SUPPORT against the base: no semantic
difference from upstream. Enabling the controller rebuilds clusterer with the hooks;
the two are a matched pair.
Docs: new "Building the Module" section (admin guide + README).
|
Follow-up Previously the clusterer-side integration (the
So clusterer_controller can be completely omitted — a build without it produces the stock upstream clusterer module: no Verified with |
Crypto is libsodium-only (see ce8e805)Worth stressing for reviewers: as of ce8e805 the module's cryptography settled on libsodium, and the earlier wolfSSL / OpenSSL-based paths were dropped entirely — there is no TLS-library fallback anymore. clusterer_controller now uses XChaCha20-Poly1305 + Argon2id for the shared-secret / at-rest key material and a Noise_NNpsk0 (Curve25519 / ChaCha20-Poly1305 / SHA-256) handshake for the join, all on libsodium primitives. Why the switch:
libsodium is therefore a hard build dependency of the module now; there is no |
Why this PR also touches
|
libsodium is a hard build requirement: the module is built sodium-only (XChaCha20-Poly1305 + Argon2id, plus X25519 / HKDF-SHA256 / RNG from libsodium), and the wolfSSL AES-256-GCM/scrypt fallback is not built. This keeps the module binary small and gives one consistent crypto suite across every build.
No behaviour change; a code-quality pass on the single-file module. - cc_seal_and_send(): every one of the ten packet senders repeated the same ~13-line tail (encrypt in place, build the multicast sockaddr, sendto, error log). Factor it into one helper. The multicast destination is now resolved once per cluster in mod_init (main process) into cl->mcast_dest and inherited by the forked workers - so it no longer rebuilds inet_addr()/htons() on every send, and mod_destroy's GOODBYE (which runs in the main process) uses the same path. Senders keep only their own success log via the helper's return value. - cc_peer_by_ip_locked(): the "find a peer entry by IP" scan was open-coded in many places; add a helper and use it where the lookup is standalone (cc_upsert_peer_locked, cc_update_peer_bin_locked). The fused election / member-list / prune loops are left as-is. - cc_recv_one(): reuse the is_bootstrap flag instead of re-memcmp'ing the packet magic three more times; drop the unreachable payload_len < 0 check (the minimum-length gate at the top already guarantees it is non-negative). Verified on a local two-node sodium cluster: join via KEY_GRANT + NODE_ASSIGN, sticky backup designation, master-death failover promotion, and graceful GOODBYE on shutdown - zero controller errors. Both build flavors (sodium-only and wolfSSL fallback) compile clean.
…se_NNpsk0
The join handshake used a bespoke construction: an ephemeral X25519 ECDH whose
shared secret was fed, with the password and a per-exchange nonce, into HKDF to
derive a key that XOR-wrapped the master_salt (cc_wrap_salt), plus a manual
nonce echo to bind the exchange. Replace it with a standard, analysable
handshake:
Noise_NNpsk0_25519_ChaChaPoly_SHA256, PSK = the Argon2id bootstrap key.
-> psk, e JOIN_REQ carries Noise message 1 (fresh ephemeral)
<- e, ee KEY_GRANT carries Noise message 2; its AEAD payload = master_salt
A compact Noise core (SymmetricState + CipherState + the NNpsk0 read/write) is
added on libsodium primitives (X25519, ChaCha20-Poly1305, SHA-256, HMAC-SHA256);
it was validated against the RFC 5869 HKDF vector and for self-consistency,
tamper-detection and wrong-PSK rejection in a standalone harness before wiring
in. The handshake hash binds the whole transcript, so the manual join_nonce
echo, the cc_peer_t.join_nonce field, and CC_JOIN_NONCE_SZ are removed - a stale
KEY_GRANT for a superseded JOIN_REQ now simply fails Noise msg-2 decryption.
KEY_HANDOFF (a one-shot, which NNpsk0's 2-message shape does not fit) switches
to an anonymous crypto_box_seal of the master_salt to the next master's
long-lived X25519 key (learned via ALIVE); sender authenticity still comes from
the session-key envelope. cc_wrap_salt and cc_ecdh_shared are deleted.
The multicast transport, outer AEAD framing (bootstrap/session magic), sequence
replay check and rate limiter are unchanged; wrong-password nodes are still
rejected at the bootstrap envelope before the handshake runs. Verified on a
local two-node cluster: join, sticky backup, master-death failover, GOODBYE, and
wrong-password shutdown - all clean.
Update the admin guide (and regenerated README) for the crypto changes:
- crypto is now libsodium-only (XChaCha20-Poly1305 + Argon2id); the wolfSSL
fallback and its AES-256-GCM/scrypt description are removed, and the build
now requires libsodium;
- the Phase 1 join handshake is described as Noise_NNpsk0_25519_ChaChaPoly_
SHA256 (PSK = bootstrap key) instead of the hand-rolled ECDH-and-XOR wrap;
- JOIN_REQ / KEY_GRANT carry Noise messages 1 / 2; KEY_HANDOFF uses a
crypto_box sealed to the next master's key;
- password / bootstrap-key wording updated (Argon2id, Noise PSK).
The election result was only visible as a cluster-wide roles list; a node could not see its OWN transition on its own line - e.g. a rejoining higher-IP peer would silently demote this backup to a plain member with nothing marking the change. Track the last-known self role (enum cc_role) per cluster and, at the end of cc_elect_master(), emit a single "my role changed: X -> Y" line whenever it differs. This covers every election trigger (goodbye, join, timer) in one place. Zero-initialised to member, matching a not-yet-joined node. The post-goodbye "re-election complete" line drops its now-redundant role clause and just names the elected master.
tm_init_cluster() read this node cluster id once, at mod_init, and baked it into the ";cid=<id>" Via parameter (and into a cached tm_node_id used to decide whether an anycast reply is ours). That assumes the id is known and stable by mod_init, which holds for a statically configured clusterer but not for a controller-managed one: there the id is assigned at runtime, after the node joins, and can change on re-election. So mod_init froze the still-unassigned id (-1, printed as its unsigned form 18446744073709551615) into every outgoing Via, and every node compared incoming cids against -1 - t_anycast_replicate() could never route a reply to the owning node. Read the id live instead: pre-build only the fixed ";<param>=" prefix at init, and have tm_via_cid() append the current get_my_id() per request (returning no parameter while the node still has no id); compare incoming cids against the live get_my_id() too. cl_get_my_id() already returns the runtime id, so this needs nothing from the caller and self-corrects across re-elections.
The block flagged the sharing-tag override, its clearing, and the per-node shtag status as future work, but all three shipped: cl_ctr_shtag_force, cl_ctr_shtag_auto, and the shtag_mode field of cl_ctr_list_config. Reduce it to the one item still outstanding - node maintenance mode - and note that it would build on the shtag commands already in place, so the source no longer reads as unfinished where it is not.
aad3a24 to
5dc976d
Compare
2 participants
This PR adds
clusterer_controller, a module that drives the existingclusterermodule's topology automatically at runtime — a zero-config HA control plane. No database, static node lists, or per-node config: every node runs an identical config, the only difference being the BIN socket IP.It is strictly opt-in, per cluster. A
clusterercluster is controller-managed only when it has acluster_options "cluster_id=N, use_controller=1"line; every other cluster (DB-backed or static) behaves exactly as before. The module is excluded from the default build (enable viainclude_modules), and when it is not built theclusterermodule is byte-for-byte the stock upstream one (verified withunifdefagainst the base).The full reference — every parameter, MI command, script variable/function, the node state machine, and the hybrid topologies — lives in the module docs (
clusterer_controller_admin.xml/ the regenerated README). This description is the summary.How it works
Nodes discover each other over authenticated, encrypted UDP multicast and elect a master deterministically (highest IP; a
master_stickinessdefault keeps a live master in place so joins cause no handover). Roles are master / backup / member, the backup always being the highest-IP non-master. The master allocates anode_idto every member and feeds identity + peer list intoclustererat runtime through a newclusterer_ctrlAPI — so every existing clusterer consumer (dialog, usrloc, dispatcher, …) keeps working with no changes.The cluster session key is generated once at bootstrap and preserved across every master change, so failover (backup promoted in ~3 s on a
GOODBYEor a missed 1 sMASTER_ALIVE) needs no re-keying and no re-join storm.Split-brain is prevented at join (simultaneous starters defer to the highest-IP one, so independent-key lone masters never form) and healed at runtime (same-key masters yield by IP; divergent-key masters converge via a bootstrap-key
MASTER_BEACON).Minimal config (identical on every node, only the BIN socket IP differs)
With
manage_shtags=1(default) the master is the sole authority for sharing-tag failover — no event routes or MI needed;cl_ctr_shtag_force/cl_ctr_shtag_autogive a manual override.Hybrid & multiple clusters
One instance can run native (DB/static) clusters and controller-managed clusters side by side, and several controller clusters at once (isolated by multicast group and/or port, plus a cleartext
cluster_idin every packet, matched before decryption). Acluster_idis exclusively one kind — declaring the same id both ways is rejected at startup. Node identity is per-cluster (a node may benode_id=1in one cluster and2in another), so hybrids and multi-controller-cluster instances replicate correctly — validated live.Security
cluster_idare bound as AAD, so a captured packet can't be re-stamped onto another cluster on a shared group.manage_shtags/master_stickiness/query_time) is advertised in every heartbeat and enforced byon_config_mismatch(reject/warn/adopt).libsodium is a hard build requirement (sodium-only; no wolfSSL linked).
Devel / 4.1 notes
clusterer_controllerand theclusterermodule are touched — no core files.clusterer(coexists with the new inter-clusterbridgesfeature andclusterer_enable_rerouting);cc_discover_bin_sockets()uses devel'ssocket_info_fulllistener list.JOIN_REQrate-limiting, and the seed sync-fallback downgraded from ERROR to DBG.cidis now rendered from the live node id (see the pinned comment), sot_anycast_replicate()works under controller-managed clusters where the id is assigned at runtime.Testing
Built against OpenSIPS 4.1.0-dev and validated end-to-end on a live 3-node cluster (
10.22.23.191–193) with dialog replication and sharing-tag failover:make all, no version-control mismatch, crypto init, no leaks/crasheskill -9) — backup promoted ~3 s on the preserved key, no election loopmaster_stickiness0 and 1MASTER_BEACONmergeJOIN_REJECTsent; fake-MASTER_ALIVEflood ignoredFull walk-throughs in
modules/clusterer_controller/doc/clusterer_controller_tests.xml.MI under 4.1:
opensips-cli0.3.3 misparses the 4.1 MI listing (reports "no command" even foruptime); usemi_httpuntil the CLI is updated — a tooling-version issue, not a module/core defect.Roadmap (not in this PR)
Node maintenance mode; statistics (
get_statistics); events (E_CL_CTR_*on state transitions); per-node identity with enrolment/revocation (plus a PAKE to resist offline password guessing on captured bootstrap packets); IPv6 multicast. Detailed under Planned Features in the admin guide.