Engineering Oracle RAC Services for Application Continuity and Predictable Failover

Failover Is an Application Property

Oracle RAC can restart an instance or relocate a service, but that does not automatically make an application highly available. The application experiences a failed socket, an uncertain transaction outcome, invalid session state, and possibly a reconnection storm. RAC protects database availability; services, FAN-aware clients, Transaction Guard, and Application Continuity determine whether that availability reaches the user.

The engineering unit is therefore not the instance. It is the complete request path: database service, SCAN listeners, Oracle Notification Service, client driver, connection pool, transaction boundaries, and application retry behavior. A weakness in any one layer can turn a routine instance restart into minutes of errors.

Use Services as Workload Contracts

Applications should connect to user-defined services, never instance names or the default database service. A service defines placement, role, load-balancing behavior, failover semantics, and maintenance policy. Different workloads normally need different contracts. A latency-sensitive OLTP service should not inherit the same placement or draining policy as batch processing.

For an administrator-managed two-instance cluster, an OLTP service using Transparent Application Continuity might be created as follows:

srvctl add service -db PAYDB -service pay_oltp -pdb PAYPDB \
  -preferred PAYDB1,PAYDB2 \
  -role PRIMARY -policy AUTOMATIC \
  -clbgoal SHORT -rlbgoal SERVICE_TIME \
  -failovertype AUTO -failover_restore AUTO \
  -commit_outcome TRUE -retention 86400 \
  -replay_init_time 300 -notification TRUE \
  -drain_timeout 300 -stopoption IMMEDIATE

srvctl start service -db PAYDB -service pay_oltp
srvctl config service -db PAYDB -service pay_oltp

SHORT favors connection distribution suitable for short-lived work, while SERVICE_TIME enables runtime load-balancing advice based on observed service quality. These settings only influence capable clients; they cannot repair a pool that ignores FAN and fills every connection using a static host list.

Use TRANSACTION with failover_restore LEVEL1 when explicitly configuring Application Continuity. Use AUTO and failover_restore AUTO for Transparent Application Continuity where supported by the database release and client. Verify the exact SRVCTL syntax against the installed Grid Infrastructure release because service attributes evolve across releases.

Transaction Guard Solves the Commit Ambiguity

The most dangerous database failure is not a visible rollback. It is a lost response after the database has committed. If the application blindly retries, it may create a second payment, order, or message. If it refuses to retry, a successfully committed operation may be reported as failed.

Transaction Guard assigns a logical transaction identifier and preserves the outcome of the last transaction. The service attribute commit_outcome TRUE enables that capability for supported clients and Application Continuity. Retention must exceed the realistic interval in which reconnection and outcome determination can occur. It is not a substitute for business idempotency, especially where a request invokes external systems outside the Oracle transaction.

Replay is safe only while the database and driver can establish that the original work did not produce an outcome that would be duplicated. This is why generic application retry loops around COMMIT are weaker than database-aware replay. Keep idempotency keys for cross-system workflows, but do not use them as an excuse to ignore commit-outcome handling.

Request Boundaries Define the Replay Unit

Application Continuity records database calls during a request and can replay them on a surviving instance. The request boundary must match a logical unit of application work. Oracle pools such as Universal Connection Pool and OCI Session Pool can establish boundaries when connections are borrowed and returned. A connection retained indefinitely by application code creates an oversized replay unit, higher memory consumption, and more opportunities for replay to become disabled.

Session state deserves particular scrutiny. NLS settings, package globals, temporary tables, application contexts, sequence usage, and calls with external side effects can affect replay. Mutable session state should be initialized consistently when a connection is acquired. For conventional Application Continuity, session_state DYNAMIC is the conservative choice when the application changes state during requests.

Drivers and pools must be certified for the selected mode. A correct database service cannot make an old JDBC driver FAN-aware. Inventory the actual driver JAR, ODP.NET package, OCI library, and pool version deployed in every application image rather than accepting a framework-level version declaration.

FAN Must Replace TCP Timeout Detection

Without Fast Application Notification, a pool may discover failure only after TCP retransmission, an operating-system timeout, or a validation query. Those paths are slow and produce synchronized application errors. FAN publishes service and instance state changes through Oracle Notification Service so that capable pools can remove dead connections immediately and rebalance when capacity returns.

The connect descriptor should use SCAN and the service name:

PAYDB =
  (DESCRIPTION=
    (CONNECT_TIMEOUT=5)
    (TRANSPORT_CONNECT_TIMEOUT=3)
    (RETRY_COUNT=3)
    (RETRY_DELAY=1)
    (ADDRESS=(PROTOCOL=TCP)(HOST=pay-scan.example.com)(PORT=1521))
    (CONNECT_DATA=(SERVICE_NAME=pay_oltp.example.com)))

Connection timeouts bound establishment attempts; they do not replace FAN. Also validate DNS resolution, SCAN VIP reachability, listener registration, ONS routing, firewalls, and load balancers. Middleboxes that silently discard idle sessions or delay resets can defeat otherwise sound failover behavior.

Draining Is the Planned-Maintenance Control Plane

For planned work, the objective is to stop new requests reaching an instance while allowing active requests to complete. FAN-aware pools receive the service event and stop borrowing affected connections. Connections are closed at request boundaries, gradually evacuating the instance.

srvctl relocate service -db PAYDB -service pay_oltp \
  -oldinst PAYDB1 -newinst PAYDB2 \
  -drain_timeout 300 -stopoption IMMEDIATE -force

The drain timeout is an engineering decision, not a decorative default. It should exceed normal high-percentile request duration but remain shorter than the maintenance window and operational recovery objective. If requests routinely exceed the timeout, investigate long transactions, abandoned connections, blocked calls, and pool misuse rather than continually increasing it.

After the timeout, IMMEDIATE terminates remaining sessions and allows replay where possible. A purely transactional stop can wait behind badly behaved sessions. Conversely, an aggressively short timeout converts manageable maintenance into forced recovery. Establish separate policies for OLTP, batch, and administrative services.

Failure Modes That Survive a Correct RAC Configuration

Connection storms: Hundreds of application processes reconnect simultaneously to the surviving instance. Pools need bounded creation rates, sensible minimum sizes, and retry jitter. The remaining instance must have CPU, processes, sessions, and listener capacity for the transferred load.
Replay disabled: Unsupported calls, incomplete request boundaries, mutable state, or explicit application behavior can make a request non-replayable. The application still needs a controlled error path.
External side effects: Sending email, publishing to a nontransactional API, or modifying a remote system from within a request cannot necessarily be undone or replayed safely. Use an outbox pattern or another transactional handoff.
Service placement collapse: A service may technically fail over but place all critical workloads on one node. Model simultaneous service demand and protect capacity with Resource Manager where appropriate.
Data Guard role transition mismatch: A service intended for the primary must be defined consistently at both sites with the correct role. Client connect data must expose both sites through an appropriate naming or global service design.

Measure the Client Experience

Database instance status is insufficient. Monitor service response time, connection-pool borrow latency, active and pending connections, FAN event receipt, reconnect rate, replay attempts, replay success, request errors, and transaction latency during topology changes. Correlate these with listener logs, Clusterware events, database alert logs, CPU saturation, and session limits.

Confirm runtime service attributes from the database:

SELECT name, network_name, failover_type, failover_method,
       commit_outcome, retention_timeout, replay_initiation_timeout
FROM   dba_services
WHERE  name = 'pay_oltp';

SELECT inst_id, service_name, COUNT(*) sessions
FROM   gv$session
WHERE  service_name = 'pay_oltp.example.com'
GROUP  BY inst_id, service_name
ORDER  BY inst_id;

Run controlled tests for instance abort, service relocation, listener loss, node isolation, and planned patching. Capture user-visible error rate and recovery time, not merely the time Clusterware takes to restart a resource. Include in-flight reads, writes, commits, long transactions, and requests with session state.

Operational Acceptance Criteria

A production service should not be approved because srvctl config service looks correct. Approval should require a supported client, verified FAN delivery, explicit request boundaries, tested transaction-outcome behavior, bounded reconnection, adequate surviving-node capacity, and repeatable maintenance procedures.

Oracle RAC availability becomes useful only when the application can consume it. Treat service configuration and client behavior as version-controlled infrastructure, validate them after every driver or Grid Infrastructure upgrade, and make failover testing part of normal release engineering rather than an annual disaster-recovery ceremony.

Oracle documents the current service attributes in the Oracle RAC Administration and Deployment Guide and provides planned-maintenance command patterns in the Oracle Database High Availability Overview.