Skip to content

Check host availability in order to maintain TCP connectors - #2579

Open
nluaces wants to merge 3 commits into
skupperproject:mainfrom
nluaces:tcp-connectors-host-availability
Open

nluaces wants to merge 3 commits into
skupperproject:mainfrom
nluaces:tcp-connectors-host-availability

Conversation

@nluaces

@nluaces nluaces commented Sep 18, 2026

Copy link
Copy Markdown
Member

Resolves #2078

Summary by CodeRabbit

  • New Features

    • Added automatic health monitoring for host-based connectors.
    • Unreachable connectors are excluded from bridge configuration and reported with an error status.
    • Connector health changes now refresh routing configuration automatically.
    • Site connector, listener, and multi-key listener bindings are reconciled as site state changes.
  • Bug Fixes

    • Connector additions and removals after startup are now reflected without restarting the service.
    • Bindings are properly released when the system adaptor stops.

@nluaces nluaces self-assigned this Sep 18, 2026
@coderabbitai

coderabbitai Bot commented Sep 18, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

📝 Walkthrough

Walkthrough

The change adds host connector health checks to Bindings, updates Kubernetes router configuration and connector status when health changes, and adds non-Kubernetes binding lifecycle management and reconciliation for site resources.

Changes

Connector health monitoring

Layer / File(s) Summary
Binding health monitoring
internal/site/bindings.go
Bindings tracks host connectors, performs concurrent TCP health checks, reports health transitions, and includes only healthy host connectors in bridge configuration.
Kubernetes health integration
internal/kube/site/extended_bindings.go
Extended bindings start and stop health checks, skip unhealthy host connectors, refresh router configuration, and update connector status errors.
Non-Kubernetes binding lifecycle and reconciliation
internal/nonkube/controller/system_adaptor_handler.go, internal/nonkube/controller/system_adaptor_handler_test.go
SystemAdaptorHandler builds, starts, reconciles, and stops bindings. Tests cover binding cleanup and connector additions and removals.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Controller
  participant Bindings
  participant ConnectorTarget
  participant Router
  Controller->>Bindings: Start health-check loop
  Bindings->>ConnectorTarget: Dial host and port
  ConnectorTarget-->>Bindings: Return reachability result
  Bindings->>Router: Refresh bridge configuration
  Bindings->>Controller: Report connector health change
  Controller->>Router: Apply updated router configuration
Loading

Merge Risk: 🟠 High · up to 68734

Concurrent health and connector updates can misroute traffic or crash Kubernetes controller processing. These issues should be fixed before merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding host availability checks to maintain TCP connectors in Kubernetes and non-Kubernetes environments.
Linked Issues check ✅ Passed The changes satisfy #2078. Bindings tracks host-and-port connectors and performs TCP health checks. Healthy connectors enter the bridge configuration, and unhealthy connectors do not. Kubernetes hea…
Out of Scope Changes check ✅ Passed The changes stay within #2078. The new binding lifecycle, reconciliation helpers, router refresh logic, and tests support host connector health and configuration failover in Kubernetes and non-Kuberne…
  • Fix all pre-merge checks with AI

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: e99a1ca1-5720-45a4-aba6-b54ded2b32bd

📥 Commits

Reviewing files that changed from the base of the PR and between d562c4c and 6873410.

📒 Files selected for processing (4)
  • internal/kube/site/extended_bindings.go
  • internal/nonkube/controller/system_adaptor_handler.go
  • internal/nonkube/controller/system_adaptor_handler_test.go
  • internal/site/bindings.go

Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Comment on lines +626 to +633
func (a *ExtendedBindings) onConnectorHealthChanged(changedConnectors []site.ConnectorHealthChange) {
if a.site != nil {
a.logger.Info("Connector health status changed, updating router config", slog.Any("connectors", changedConnectors))
if err := a.site.updateRouterConfig(a); err != nil {
a.logger.Error("Failed to update router config on health check change", slog.Any("error", err))
}
for _, res := range changedConnectors {
connector := a.bindings.GetConnector(res.Name)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🔎 Supported by static analysis

🏁 Script executed:

sed -n '1,220p' internal/kube/site/extended_bindings.go
sed -n '600,670p' internal/kube/site/extended_bindings.go
rg -n 'updateRouterConfig|GetConnector|updateConnector|deleteConnector|connectors|onConnectorHealthChanged' internal/kube/site internal/site/bindings.go

Repository: skupperproject/skupper

Length of output: 18741


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- internal/site/bindings.go:1-220 ---'
sed -n '1,220p' internal/site/bindings.go
printf '%s\n' '--- internal/site/bindings.go:300-550 ---'
sed -n '300,550p' internal/site/bindings.go
printf '%s\n' '--- extended_bindings.go:360-610 ---'
sed -n '360,610p' internal/kube/site/extended_bindings.go
printf '%s\n' '--- event processor files ---'
fd -t f . internal | rg 'watchers|event|processor'
printf '%s\n' '--- EventProcessor references ---'
rg -n -C 4 'type EventProcessor|NewEventProcessor|EventProcessor|controller\.|Process.*Event|Handle.*Event' internal/kube internal | head -n 500
printf '%s\n' '--- router update and lifecycle references ---'
sed -n '240,300p' internal/kube/site/site.go
sed -n '1015,1075p' internal/kube/site/site.go
sed -n '1,130p' internal/kube/site/bindings.go

Repository: skupperproject/skupper

Length of output: 50378


🏁 Script executed:

#!/bin/bash
set -e
sed -n '1,220p' internal/site/bindings.go
sed -n '300,550p' internal/site/bindings.go
sed -n '360,610p' internal/kube/site/extended_bindings.go
fd -t f . internal | rg 'watchers|event|processor'
rg -n -C 4 'type EventProcessor|NewEventProcessor|EventProcessor|Process.*Event|Handle.*Event' internal/kube internal
sed -n '240,300p' internal/kube/site/site.go
sed -n '1015,1075p' internal/kube/site/site.go
sed -n '1,130p' internal/kube/site/bindings.go

Repository: skupperproject/skupper

Length of output: 50379


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- event processor core ---'
sed -n '170,285p' internal/kube/watchers/watchers.go
printf '%s\n' '--- site binding watcher/handlers ---'
rg -n -C 5 'Watch.*Connector|checkConnector|ConnectorUpdated|ConnectorDeleted|UpdateConnector|SetBindingEventHandler|bindings\.' internal/kube/site --glob '*.go' | head -n 260
printf '%s\n' '--- site router update ---'
sed -n '1018,1068p' internal/kube/site/site.go
printf '%s\n' '--- site initialization around binding setup ---'
rg -n -C 12 'NewSite|SetSite|init\(|Watch.*Connector|Watch.*Listener|WatchAttached' internal/kube/site/site.go internal/kube/site/*.go | head -n 320
printf '%s\n' '--- exact relevant mutation definitions ---'
sed -n '180,275p' internal/site/bindings.go
sed -n '480,610p' internal/kube/site/extended_bindings.go

Repository: skupperproject/skupper

Length of output: 49095


🏁 Script executed:

#!/bin/bash
set -e
sed -n '170,285p' internal/kube/watchers/watchers.go
rg -n -C 5 'Watch.*Connector|checkConnector|ConnectorUpdated|ConnectorDeleted|UpdateConnector|SetBindingEventHandler|bindings\.' internal/kube/site --glob '*.go' | head -n 260
sed -n '1018,1068p' internal/kube/site/site.go
rg -n -C 12 'NewSite|SetSite|init\(|Watch.*Connector|Watch.*Listener|WatchAttached' internal/kube/site/site.go internal/kube/site/*.go | head -n 320
sed -n '180,275p' internal/site/bindings.go
sed -n '480,610p' internal/kube/site/extended_bindings.go

Repository: skupperproject/skupper

Length of output: 48904


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- router update consumer ---'
rg -n -C 8 'func UpdateRouterConfig|type ConfigUpdate|Apply\(config' internal/kube internal/qdr --glob '*.go' | head -n 220
printf '%s\n' '--- controller paths into Site binding methods ---'
rg -n -C 5 'CheckConnector|CheckListener|CheckAttachedConnector|AttachedConnectorUpdated|UpdateConnector\(|UpdateListener\(|UpdateMultiKeyListener\(' internal --glob '*.go' | head -n 300

Repository: skupperproject/skupper

Length of output: 30539


🏁 Script executed:

#!/bin/bash
set -e
rg -n -C 8 'func UpdateRouterConfig|type ConfigUpdate|Apply\(config' internal/kube internal/qdr --glob '*.go' | head -n 220
rg -n -C 5 'CheckConnector|CheckListener|CheckAttachedConnector|AttachedConnectorUpdated|UpdateConnector\(|UpdateListener\(|UpdateMultiKeyListener\(' internal --glob '*.go' | head -n 300

Repository: skupperproject/skupper

Length of output: 29543


Serialize health-change processing with controller updates.

Bindings.PerformHealthChecks invokes this callback from the health-check goroutine after releasing healthMu. The controller event loop can concurrently call Bindings.UpdateConnector, which mutates Bindings.connectors without a shared lock. The callback also passes ExtendedBindings to updateRouterConfig; ExtendedBindings.Apply iterates its maps while controller handlers may mutate them. GetConnector reads the same unprotected binding map.

This can cause a concurrent-map panic or build router configuration from mixed state. Dispatch the callback through the controller event loop, or protect all shared binding state with one synchronization mechanism.

Comment on lines +79 to +80
if os.Getuid() == 0 {
api.DefaultRootDataHome = tempDir

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n '1,240p' internal/nonkube/controller/system_adaptor_handler_test.go
rg -n 'DefaultRootDataHome|t\.Parallel\(' internal/nonkube pkg/nonkube

Repository: skupperproject/skupper

Length of output: 8094


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- environment.go ---'
cat -n pkg/nonkube/api/environment.go
printf '%s\n' '--- controller assignments and relevant test setup ---'
rg -n -C 8 'DefaultRootDataHome|GetInternalOutputPath|GetDataHome' internal/nonkube/controller
printf '%s\n' '--- all non-kube test references ---'
rg -n -C 5 'DefaultRootDataHome|GetInternalOutputPath|GetDataHome' internal/nonkube pkg/nonkube --glob '*_test.go' --glob '*.go'

Repository: skupperproject/skupper

Length of output: 50378


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- controller test functions and packages ---'
rg -n '^(func Test|package )' internal/nonkube/controller --glob '*_test.go'
printf '%s\n' '--- controller test path consumers ---'
rg -l 'GetInternalOutputPath|GetDefaultOutput|GetDataHome|GetHostDataHome|DefaultRootDataHome' internal/nonkube/controller --glob '*_test.go' |
while read -r f; do
  printf '%s\n' "--- $f"
  rg -n 'DefaultRootDataHome|GetInternalOutputPath|GetDefaultOutput|GetDataHome|GetHostDataHome' "$f"
done
printf '%s\n' '--- direct controller test ordering/concurrency controls ---'
rg -n 't\.Parallel|t\.Run|shuffle|TestMain' internal/nonkube/controller --glob '*_test.go' || true

Repository: skupperproject/skupper

Length of output: 8363


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- system adaptor production path ---'
cat -n internal/nonkube/controller/system_adaptor_handler.go | sed -n '1,125p'
printf '%s\n' '--- runtime address lookup ---'
rg -n -C 12 'func GetLocalRouterAddress|GetInternalOutputPath|RuntimeSiteStatePath' internal/nonkube/client/runtime
printf '%s\n' '--- no-setup controller tests and constructors ---'
cat -n internal/nonkube/controller/input_resource_handler_test.go | sed -n '1,140p'
rg -n -C 8 'func NewInputResourceHandler|GetInternalOutputPath|PathProvider' internal/nonkube/controller/input_resource_handler.go internal/nonkube/controller/*.go

Repository: skupperproject/skupper

Length of output: 37744


Restore api.DefaultRootDataHome in both reconciliation tests.

When tests run as root, GetDataHome uses api.DefaultRootDataHome. Each test sets it to t.TempDir(), but cleanup deletes that directory while the global retains its path. A subsequent test that uses GetInternalOutputPath can then access a deleted path. Save the original value and restore it with t.Cleanup at both lines 80 and 156.

Comment thread internal/site/bindings.go
Comment on lines +518 to +523
if _, exists := b.hostConnectors[res.Name]; !exists {
continue
}
prevHealthy, ok := b.connectorHealth[res.Name]
if !ok || prevHealthy != res.Healthy {
b.connectorHealth[res.Name] = res.Healthy

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject health results for superseded endpoints.

A health check can start for one endpoint while reconciliation changes the same connector name to another endpoint. The current existence check accepts the old result and applies it to the replacement endpoint. This can remove a healthy replacement connector or restore an unavailable connector.

Compare the current Host and Port with the result before updating connectorHealth.

Proposed fix
-		if _, exists := b.hostConnectors[res.Name]; !exists {
+		current, exists := b.hostConnectors[res.Name]
+		if !exists || current.Host != res.Host || current.Port != res.Port {
 			continue
 		}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if _, exists := b.hostConnectors[res.Name]; !exists {
continue
}
prevHealthy, ok := b.connectorHealth[res.Name]
if !ok || prevHealthy != res.Healthy {
b.connectorHealth[res.Name] = res.Healthy
current, exists := b.hostConnectors[res.Name]
if !exists || current.Host != res.Host || current.Port != res.Port {
continue
}
prevHealthy, ok := b.connectorHealth[res.Name]
if !ok || prevHealthy != res.Healthy {
b.connectorHealth[res.Name] = res.Healthy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TcpConnectors not removed or added on the Router Config when target host availability changes

1 participant