Skip to content

Never emit an empty synchronous_standby_names list - #73

Open
souravbiswassanto wants to merge 1 commit into
masterfrom
fix-sync-standby-names-empty
Open

souravbiswassanto wants to merge 1 commit into
masterfrom
fix-sync-standby-names-empty

Conversation

@souravbiswassanto

@souravbiswassanto souravbiswassanto commented Sep 19, 2026 •

Copy link
Copy Markdown
Member

Problem

The role scripts build synchronous_standby_names from $REPLICAS, skipping the pod's own index:

for ((i = 0; i < $REPLICAS; i++)); do
    if [[ $self_idx == $i ]]; then echo "skip $i"; else names+="\"$sts_prefix$i\"",; fi
done
names=${names%,}
echo "synchronous_standby_names = '${SYNC_REPLICATION_MODE:-ANY} ${NUM_SYNC_REPLICAS:-1} ("$names")'"

At one replica that loop skips its only iteration and $names is empty, while
SYNC_REPLICATION_MODE / NUM_SYNC_REPLICAS still come from the CR and describe the eventual
topology. The emitted value is 'FIRST 2 ()'.

Postgres does not read an empty list as "no synchronous standbys" — it is a parse error, and the
server refuses to start:

LOG:     invalid value for parameter "synchronous_standby_names": "FIRST 2 ()"
DETAIL:  syntax error at or near ")"
FATAL:   configuration file "/var/pv/data/postgresql.conf" contains errors

scripts/run.sh then re-runs the role script on its one-second loop, and start.sh prepends a
fresh block each pass, so the config also accumulates duplicate entries while the pod sits
not-ready:

$ grep standby /var/pv/data/postgresql.conf
hot_standby = on
synchronous_standby_names = 'FIRST 2 ()'
hot_standby = on
synchronous_standby_names = 'FIRST 2 ()'

How it is reached today

A PITR restore. postgres/pkg/controller/restore.go collapses the CR to a single replica for the
sync and fscopy strategies, while leaving streamingMode: Synchronous in place:

if *db.Spec.Replicas > 1 && (rs == ReplicationStrategyFSCopy || rs == ReplicationStrategySync) {
    ...
    db.Spec.Replicas = ptr.To(int32(1))
}

sync is the default (postgres_helpers.go SetDefaults), so this affects any synchronous
cluster with replicas > 1 being restored — the user need not have chosen a strategy at all. The
restore's WAL replay completes and then the primary never starts; the CR sits in Provisioning.

Reported against KubeDB v2026.1.19 with PG 16.9, streamingMode: Synchronous,
synchronousReplicationConfig: {mode: First, numSyncReplicas: 2}, replicas: 3.

Fix

Guard the empty list. Postgres accepts synchronous_standby_names = '' and treats every
synchronous_commit level as local, so commits wait for local flush only — no error, and no
hang waiting for standbys that do not exist. The next reconcile rewrites the file with a real list
once the peers are up.

Applied to all 40 emitters (majors 9–18 × primary/start.sh, standby/run.sh,
standby/warm_stanby.sh, standby/ha_backup_job.sh). The majors are independent copies, so each
one needs the guard. No behaviour change when the list is non-empty.

Verification

Against a real PostgreSQL 16.15:

value result
'FIRST 2 ()' invalid value for parameter ... syntax error at or near ")" — reproduces the report exactly
'' starts
'FIRST 2 ("a","b")' starts

Replaying the block from role_scripts/16/primary/start.sh:

BEFORE:  REPLICAS=1 -> synchronous_standby_names = 'FIRST 2 ()'
         REPLICAS=3 -> synchronous_standby_names = 'FIRST 2 ("test-pg-1","test-pg-2")'
AFTER:   REPLICAS=1 -> synchronous_standby_names = ''
         REPLICAS=3 -> synchronous_standby_names = 'FIRST 2 ("test-pg-1","test-pg-2")'
  • bash -n passes on all 66 scripts in role_scripts/.
  • shfmt -l -ci -i 4 reports the same 56 files before and after, so this adds no formatting drift.

Related

https://github.com/kubedb/postgres/pull/938 removes the mismatch at the source by restoring asynchronously while the CR is
collapsed to one pod. Either change alone fixes the reported case; this one is the defensive half
and covers every other path that can produce an empty list.

Shipping this to existing clusters needs a postgres-init release plus a PostgresVersion
catalog bump in kubedb/installer.

🤖 Generated with Claude Code

The role scripts build synchronous_standby_names from $REPLICAS, skipping
the pod's own index. At one replica that list is empty, and the emitted
value becomes 'FIRST 2 ()' -- with the mode and quorum still coming from
SYNC_REPLICATION_MODE/NUM_SYNC_REPLICAS, which describe the eventual
topology rather than the current one.

Postgres does not read an empty list as "no synchronous standbys". It is a
parse error:

  LOG:   invalid value for parameter "synchronous_standby_names": "FIRST 2 ()"
  DETAIL:  syntax error at or near ")"
  FATAL:  configuration file "/var/pv/data/postgresql.conf" contains errors

so the server never starts. scripts/run.sh then re-runs the role script on
its one-second loop and start.sh prepends another block each pass, so the
file also accumulates duplicate entries while the pod sits not-ready.

This is reachable today: a PITR restore with replicationStrategy sync or
fscopy -- the default for postgres -- collapses the CR to a single replica
while leaving streamingMode Synchronous, so any synchronous cluster with
replicas > 1 fails to come back from a restore.

Fall back to an empty string when the list is empty. Postgres accepts
synchronous_standby_names = '' and treats every synchronous_commit level as
local, so commits wait for local flush only: no error and no hang. The next
reconcile rewrites the file with a real list once the peers exist.

Verified against postgres 16.15: 'FIRST 2 ()' reproduces the error above,
'' and 'FIRST 2 ("a","b")' both start.

Applied to all 40 emitters (majors 9-18 x primary/start, standby/run,
standby/warm_stanby, standby/ha_backup_job); the majors are independent
copies, so each needs the guard. No behaviour change when the list is
non-empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: souravbiswassanto <saurov@appscode.com>
@coderabbitai

coderabbitai Bot commented Sep 19, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 69c85ba8-671c-48fd-aaf5-73ef418a9656


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant