Cache Fusion Is a Workload Property, Not Just a Network Feature
Oracle RAC Cache Fusion allows an instance to obtain a current or consistent-read block directly from another instance’s buffer cache instead of rereading it from shared storage. This memory-to-memory transfer is fundamental to RAC scalability, but it does not make every workload scale linearly. When multiple instances repeatedly modify the same blocks, block ownership moves between nodes and foreground sessions inherit the cost of Global Cache Service coordination. A fast interconnect reduces the transfer component, but it cannot eliminate serialization caused by hot blocks, delayed redo flushes, CPU starvation, or inefficient application placement.
The first architectural distinction is between useful remote access and destructive block pinging. A reporting service reading blocks modified by an OLTP service may generate many gc cr block 2-way transfers without causing a serious problem. Conversely, two application instances concurrently updating rows stored in the same small set of table or index blocks can generate fewer transfers while producing severe gc current block busy waits. Counts describe traffic; latency distributions, wait classes, SQL attribution, and affected business operations describe impact. Capacity reviews that look only at aggregate interconnect bandwidth frequently miss this distinction.
Oracle AI Database 26ai provides an additional diagnostic view, V$CACHE_FUSION_HISTOGRAM, available from Release Update 23.8, that breaks LMS processing latency into buckets. This is especially useful because an average can hide a small but operationally important tail above 10 or 100 milliseconds. The view complements, rather than replaces, AWR, ASH, operating-system counters, and application latency telemetry. Oracle documents the view and its latency buckets in the 26ai Database Reference.
Separate Transport Latency from Block Contention
Global cache wait names describe the outcome of a request. A gc current block 2-way wait means a current block arrived after a two-hop exchange, while a three-way event involves an additional forwarding step through the resource master. A busy event is materially different: the serving instance could not ship the block immediately. The block may be pinned, another conversion may be in progress, or Oracle may need redo associated with the block change to reach stable storage before transferring it. Consequently, an apparent Cache Fusion problem can originate in LGWR latency, overloaded LMS processes, CPU scheduling, or application concurrency rather than in the network.
Collect deltas over the exact performance interval instead of interpreting cumulative statistics since instance startup. The following query provides an initial comparison between instances, but its values must be sampled twice and subtracted. A reset caused by instance restart must also be recognized by the monitoring system.
SELECT inst_id,
event,
total_waits,
time_waited_micro,
ROUND(time_waited_micro / NULLIF(total_waits, 0), 1) AS avg_us
FROM gv$system_event
WHERE event IN ('gc current block 2-way',
'gc current block 3-way',
'gc current block busy',
'gc cr block 2-way',
'gc cr block 3-way',
'gc cr block busy',
'gc buffer busy acquire',
'gc buffer busy release')
ORDER BY inst_id, event;Do not collapse all gc waits into one metric. Two-way and three-way transfer latency is sensitive to interconnect and LMS processing, whereas busy waits indicate delayed service or contention. gc buffer busy acquire and release show that a local session cannot access a buffer while a global operation is outstanding; they often amplify an underlying hot-block problem. Alert separately on request rate, total database time, and tail latency. A rise in wait count caused by higher throughput is not automatically degradation, while a stable count with a sharply growing tail can be an early indication of CPU or fabric congestion.
Locate the Objects That Move Between Instances
Once database time establishes that Cache Fusion is significant, attribute it to SQL, services, objects, and blocks. ASH is appropriate when the Diagnostic Pack is licensed. Querying only CURRENT_OBJ# is useful but insufficient because some waits may not expose an object, index branch blocks can dominate even when table rows are well distributed, and an object-level aggregate can conceal one repeatedly transferred block.
SELECT ash.inst_id,
ash.event,
ash.current_obj#,
o.owner,
o.object_name,
o.subobject_name,
ash.sql_id,
COUNT(*) AS ash_samples
FROM gv$active_session_history ash
LEFT JOIN dba_objects o
ON o.object_id = ash.current_obj#
WHERE ash.sample_time > SYSTIMESTAMP - INTERVAL '15' MINUTE
AND ash.event LIKE 'gc%'
GROUP BY ash.inst_id, ash.event, ash.current_obj#,
o.owner, o.object_name, o.subobject_name, ash.sql_id
ORDER BY ash_samples DESC
FETCH FIRST 30 ROWS ONLY;For a focused reproduction, retain CURRENT_FILE# and CURRENT_BLOCK# from ASH and map them with DBMS_UTILITY.DATA_BLOCK_ADDRESS_FILE, segment metadata, or appropriate extent queries. Typical findings include right-growing index leaf blocks, monotonically populated table blocks, freelist or segment-header contention in legacy objects, and application rows that act as global counters. Sequence caching can reduce contention around sequence state, but it does not fix an index whose leading column concentrates inserts at the right edge. Reverse-key indexes, hash partitioning, redesigned keys, or partition-aware routing can distribute activity, although each changes range-scan behavior, partition maintenance, or application semantics.
A controlled demonstration is simple: connect equal update loops through services running on two instances and repeatedly modify the same small group of rows. Record application latency and RAC statistics, then rerun after routing the writers to one preferred instance. The comparison demonstrates whether block ownership movement is material without pretending to be a universal benchmark. Run such a test only in an isolated environment and preserve the same commit rate, connection count, and SQL plan between executions.
Use Services to Create Affinity Without Sacrificing Availability
The most reliable mitigation for write-intensive hot data is usually workload affinity. Define services around business or data ownership boundaries and give each service a preferred instance with one or more available failover instances. This keeps normal-path writers close to the blocks they modify while preserving the ability to relocate after failure. Affinity does not require permanent instance pinning: the application should connect through service names, honor planned draining, and retry according to transaction semantics rather than caching physical host addresses.
Good service boundaries might correspond to tenant groups, geographic partitions, order-processing shards, or read-versus-write roles. The boundary must match actual block access, not merely organizational ownership. If every request updates a shared control row or common index root, separating sessions by tenant may leave the hottest structure global. Validate service placement by correlating connection-pool identity, GV$SESSION.SERVICE_NAME, ASH samples, and application transaction latency. After a failover, temporarily increased transfers are expected as the surviving instance acquires block ownership; an indefinite increase indicates that clients remain split or the data model defeats the intended affinity.
Engineer the Interconnect as a Latency-Critical Dependency
The private interconnect requires redundancy, consistent MTU, sufficient headroom, and freedom from loss and microbursts. Oracle Clusterware Redundant Interconnect Usage can create up to four HAIP addresses and distribute Oracle RAC, ASM, and ACFS traffic across interfaces classified as private. If an interface becomes unusable, Clusterware moves its HAIP to a surviving interface. This protects availability, but the remaining path still needs capacity for the failure state. Design bandwidth and switch buffers for a link or switch failure during peak database and ASM activity, not merely for steady-state averages.
$ oifcfg getif
$ cluvfy comp nodecon -n all -verbose
$ ip -s link show dev bond1
$ ethtool -S eno2These checks should confirm the configured role, node-to-node connectivity, drops, errors, carrier transitions, and queue-level faults. Validate MTU end to end with nonfragmenting probes appropriate to the platform, and verify both redundant paths independently. Never assume that a healthy public network proves the private fabric is healthy. Also avoid setting CLUSTER_INTERCONNECTS as a routine tuning measure; Oracle recommends allowing Clusterware to select HAIP-enabled interconnects, and explicitly configured values can bypass the intended topology. OIFCFG changes affect database interconnect selection at instance startup, so a configuration edit is not proof that running instances adopted it.
On Exadata, the RDMA Network Fabric and network resource management reduce CPU overhead and prioritize latency-sensitive traffic such as Cache Fusion. They do not make workload contention disappear. A block pinned by a remote foreground or delayed by redo persistence remains unavailable regardless of fabric speed. Monitor database-server CPU run queues, LMS utilization, LGWR latency, fabric errors, and global-cache tails together; otherwise an engineer may replace network components while the real bottleneck is scheduler delay or hot data.
Validate Failure Behavior, Not Only Peak Throughput
A production design is incomplete until it has been tested during interconnect path loss, instance eviction, service relocation, and rolling maintenance. During a planned test, capture application error rates and latency alongside service placement, HAIP state, interface counters, cluster logs, and Cache Fusion histograms. Confirm that losing one path does not create packet loss, MTU inconsistency, or saturation on the survivor. Also verify that connection pools stop using the failed or drained instance and that transaction retries do not duplicate non-idempotent work.
The operational objective is not zero global-cache traffic. RAC is designed to coordinate shared access, and remote reads can be entirely reasonable. The objective is bounded transfer latency, controlled write ownership, redundant fabric capacity, and application behavior that remains correct when placement changes. Engineers achieve that by treating Cache Fusion as an end-to-end system involving SQL access patterns, block layout, redo persistence, CPU scheduling, Clusterware networking, services, and retry semantics. When those layers are measured together, RAC performance becomes explainable and failure behavior becomes testable instead of surprising.
