Failover Is a Data-Integrity Decision
A PostgreSQL high-availability design is incomplete if it can promote a standby but cannot guarantee that the former primary has stopped accepting writes. Promotion is mechanically simple. Preventing two writable database nodes is the difficult part.
This distinction matters during ambiguous failures. If a database node loses access to the cluster control plane but remains reachable by some application servers, an HA manager may promote a standby while the isolated primary continues processing transactions. PostgreSQL does not provide distributed write coordination between independent timelines. Once both nodes accept writes, the result is not merely replication lag; it is divergent database history.
Rebuilding one node from the chosen survivor does not merge committed transactions from both timelines. Recovery becomes a business-level reconciliation exercise involving missing orders, duplicate identifiers, inconsistent external side effects, and audit questions. A sound architecture therefore treats promotion as the final step in a fencing workflow, not the first response to a failed health check.
Why Health Checks Are Not Proof of Failure
A failed TCP connection proves only that the observer cannot establish that connection. It does not prove that the database host is powered off, that PostgreSQL has stopped, or that every application path to the primary is unavailable.
Consider a hypothetical three-node topology: a primary and synchronous standby in one data center, plus a quorum or control-plane member in another fault domain. A network partition separates the primary from the HA manager and standby, but an application subnet can still reach it. From the majority partition, the primary appears dead. From the application perspective, it remains writable.
Automatic failover based only on PostgreSQL reachability can now create two primaries. The same risk exists when the operating system freezes temporarily, storage stalls, DNS fails selectively, firewall rules differ between subnets, or the HA agent crashes while PostgreSQL remains healthy.
Failure detection should combine multiple signals, but additional probes do not eliminate uncertainty. Distributed systems cannot always distinguish a failed node from an isolated one. The architecture must make uncertainty safe by preventing the minority or disconnected node from serving writes.
Fencing Must Remove the Old Primary’s Ability to Write
Fencing is an externally enforced action that makes the former primary incapable of continuing as a writer. The strongest method is power fencing through a server management controller, hypervisor, or cloud control plane. Terminating a virtual machine or removing power does not depend on the health of PostgreSQL, the guest operating system, or the local HA agent.
Storage fencing can also be effective when implemented by infrastructure that guarantees exclusive access. Examples include revoking a host’s SAN access or transferring a storage reservation. This requires careful validation: revoking one path is insufficient if multipathing leaves another path active, and shared storage may introduce failure coupling beyond the database layer.
Process-level actions such as pg_ctl stop, systemd stop requests, or remote SSH commands are weaker. They depend on a responsive operating system and working management network. They are useful during planned switchovers but should not be treated as definitive fencing during an ambiguous outage.
# Appropriate for a controlled switchover, not proof of fencing
sudo -u postgres pg_ctl -D /var/lib/postgresql/16/main stop -m fast
# Verify that the local server is no longer accepting connections
pg_isready -h old-primary.example.net -p 5432Even a successful stop command requires care. An automation system should verify the target identity before acting. Hostname reuse, stale inventory, copied machine identifiers, or an incorrect cloud instance ID can cause an operator to fence the wrong server. Fencing credentials should be narrowly scoped, audited, tested, and stored outside the database nodes they control.
Quorum Controls Authority, Not Physical Safety
Consensus-backed HA managers use quorum to decide which partition is authorized to coordinate the cluster. A three- or five-member distributed configuration store prevents two partitions from both obtaining a majority, provided membership and failure semantics are correct.
Quorum is necessary, but it is not equivalent to fencing. Losing leadership may stop a cooperative HA agent, yet PostgreSQL can remain running if the agent is paused, blocked, misconfigured, or killed. A primary-side watchdog that shuts down PostgreSQL after losing quorum improves safety, but it is still an in-guest mechanism. External fencing provides a stronger boundary.
Quorum placement must reflect independent failure domains. Three voting processes on three virtual machines attached to the same switch, storage array, power domain, or hypervisor cluster may look redundant while sharing the failure that matters. Conversely, placing a deciding voter across a high-latency or unstable WAN can make local HA depend on WAN health.
Engineers should model which failures are expected to trigger local failover and which should require operator intervention. There is no universally correct quorum layout. The right design follows the desired behavior during loss of a host, rack, availability zone, management network, database network, or entire site.
Replication Mode Does Not Solve Split Brain
Synchronous replication constrains transaction acknowledgement; it does not elect a writer or fence an old primary. A primary can continue if its configuration permits progress after losing synchronous standbys, while an HA manager independently promotes another node.
Settings such as synchronous_standby_names and synchronous_commit determine commit behavior, but the effective durability guarantee also depends on operational changes. Temporarily relaxing synchronous requirements during maintenance may be reasonable, yet automation must not silently convert that temporary state into the permanent failure policy.
SHOW synchronous_commit;
SHOW synchronous_standby_names;
SELECT application_name, state, sync_state,
write_lsn, flush_lsn, replay_lsn
FROM pg_stat_replication;Strict synchronous commit can reduce acknowledged-transaction loss during failover, but it may sacrifice write availability when the required standby disappears. Quorum synchronous replication can tolerate selected standby failures at additional infrastructure and latency cost. None of these choices eliminates the need to establish a single authorized primary.
Promotion Requires a Precondition Chain
A robust failover workflow should encode explicit preconditions. First, the control plane must have authority under its quorum rules. Second, it must determine the best promotion candidate using timeline, write-ahead log position, and recovery state rather than a static priority alone. Third, it must fence the old primary or obtain equally strong evidence that it cannot write. Only then should it promote the selected standby and redirect clients.
SELECT pg_is_in_recovery();
SELECT pg_last_wal_receive_lsn(), pg_last_wal_replay_lsn();
SELECT pg_last_xact_replay_timestamp();These queries help characterize a standby, but no single metric proves that it contains every acknowledged transaction. The answer depends on replication mode and the last successful acknowledgement path. Byte lag also does not describe replay complexity: a small amount of WAL can represent a transaction that takes significant time to apply.
After promotion, routing changes must not assume that every connection immediately follows the new topology. Connection pools can retain sessions to the old address. DNS caches can outlive expected time-to-live values. Layer 4 load balancers may pass TCP checks before PostgreSQL is ready for application traffic. Readiness checks should verify role and transaction capability, for example by confirming that pg_is_in_recovery() is false on the intended writer.
Protecting Against Stale Primaries Rejoining
The old primary must not simply restart and reconnect to the service endpoint. After promotion, the cluster has a new timeline. The returning node should remain quarantined until automation establishes whether it can be rewound or must be rebuilt.
pg_rewind can resynchronize a former primary when its prerequisites are satisfied and the required WAL remains available. It is not a conflict-resolution utility. It discards divergence on the old timeline while aligning the data directory with the selected source.
pg_rewind \
--target-pgdata=/var/lib/postgresql/16/main \
--source-server='host=new-primary dbname=postgres user=rewind_user'The procedure requires PostgreSQL to be stopped on the target, suitable authentication, and either data checksums or wal_log_hints=on. Required WAL must be reachable from the source or archive. If those conditions are not met, take a new base backup instead of improvising an unsupported repair.
Service discovery should register a node only after it has joined the correct timeline, entered the intended role, and passed replication checks. Starting PostgreSQL should not automatically make it eligible for production traffic.
What Production Monitoring Must Prove
Monitoring should validate safety properties, not merely process availability. Alert when more than one reachable node reports pg_is_in_recovery() = false. Track timeline changes, replication-slot retention, WAL archive health, replay position, synchronous-standby state, control-plane quorum, and fencing execution results. Monitor the management interfaces used for fencing independently from the database network.
Audit every promotion with the identity of the decision maker, quorum state, candidate position, fencing target, fencing result, new timeline, and routing change. Without this evidence, post-failure analysis becomes guesswork.
Regular exercises should include asymmetric partitions rather than only clean process failures. Test loss of database traffic while management traffic remains, the reverse condition, a hung operating system, unavailable fencing infrastructure, exhausted WAL storage, and an old primary returning after promotion. Confirm that failed fencing blocks automatic promotion unless the documented policy explicitly accepts possible split brain.
High Availability Is Not Disaster Recovery
A well-fenced local cluster provides high availability for failures within its design envelope. It does not provide disaster recovery from site loss, corruption replicated to every standby, destructive credentials, or operator error. DR requires independently protected backups, recoverable WAL archives, appropriate geographic separation, and rehearsed restoration procedures.
Cross-site automatic failover deserves particular caution. A WAN partition can resemble site failure, and fencing equipment at the remote site may be unreachable precisely when needed. If the surviving site cannot prove that the original writer is disabled, manual promotion may be the correct engineering decision. A longer recovery time is often preferable to two sites committing incompatible business transactions.
The central design rule is simple: do not promote because the primary is invisible. Promote only when the cluster has authority, the recovery candidate is understood, and the previous writer has been made incapable of writing. That rule turns PostgreSQL failover from optimistic automation into an enforceable data-integrity mechanism.
