I installed Oracle RAC 19c on a two-node cluster with Pure Storage FlashArray backend. Four 32Gb Fibre Channel HBAs per node. 24 paths per LUN. Hardware capable of 12+ GB/s throughput. Yet my benchmarks topped out at 3.2 GB/s.
This post explains the mistake I made, how I found it, and how I fixed it. If you’re using ASM with multipath SAN storage, you might be making the same mistake.
The Setup
- OS: Oracle Linux 8.10
- Oracle Version: 19.3.0.0.0
- Cluster: 2 nodes (rac-node-01, rac-node-02)
- Storage: Pure Storage FlashArray
- FC HBAs: 4x 32Gb LPe32004-M2 per node
- Theoretical max throughput: ~16 GB/s per node (4 HBAs × 32Gb × 0.8 encoding efficiency)
The Symptom
After completing the installation, I ran some baseline I/O tests. The numbers were disappointing: expected 10-12 GB/s (accounting for overhead), actual ~3.2 GB/s. The storage array wasn’t breaking a sweat. The FC fabric showed no congestion. The CPUs were idle. Something was throttling I/O at the host level.
Before reading further, check whether you have the same issue. Run this on any node:
# If using OracleASMLib
ls -l /dev/oracleasm/disks/* 2>/dev/null | head -5
# Or check your ASM disk paths directly
ls -l /dev/mapper/3624a937* 2>/dev/null | head -5
Look at the major number (first number after the permissions):
brw-rw----. 1 grid asmadmin 252, 10 ... <-- 252 = GOOD (device-mapper)
brw-rw----. 1 grid asmadmin 66, 224 ... <-- 66 = BAD (single SCSI path)
You can verify what the major numbers mean on your system:
cat /proc/devices | grep -E "device-mapper|sd"
# 8 sd
# 65 sd
# 66 sd
# 252 device-mapper
If your ASM devices show major 252: You’re fine. Your performance problem is elsewhere.
If your ASM devices show major 8, 65, 66, 67, 68, or 69: Keep reading. You have the same problem I did.
Unfortunately, there’s no practical way to fix this without reinstalling. The ASM disk header contains metadata about which device path ASM uses. Once stamped, you can’t change it without destroying and recreating the diskgroup. If you have data, you’d need to migrate it off (RMAN backup, Data Guard, export), drop the diskgroup, recreate with correct paths, and restore. For a fresh or lab install, just start over with the correct configuration below.
The Investigation
I started checking the multipath configuration. Everything looked correct:
multipath -ll | head -30
3624a93703b701e383bbe421c0001ab93 dm-22 PURE,FlashArray
size=3.0T features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 6:0:7:1 sda 8:0 active ready running
|- 6:0:12:1 sdb 8:16 active ready running
|- 7:0:9:1 sdc 8:32 active ready running
|- 7:0:13:1 sdd 8:48 active ready running
...
`- 9:0:3:1 sdx 65:112 active ready running
All 24 paths active. Service-time path selector (correct for Pure). ALUA priority working. The multipath layer was fine.
Then I checked the ASM disk configuration:
ls -l /dev/oracleasm/disks/
brw-rw----. 1 grid asmadmin 66, 240 Dec 30 10:44 CRS01
brw-rw----. 1 grid asmadmin 67, 0 Dec 30 10:45 CRS02
brw-rw----. 1 grid asmadmin 67, 192 Dec 30 10:49 CRS03
brw-rw----. 1 grid asmadmin 66, 224 Dec 30 10:45 DATA01
brw-rw----. 1 grid asmadmin 67, 16 Dec 30 10:44 DATA02
brw-rw----. 1 grid asmadmin 67, 64 Dec 30 10:44 DATA03
brw-rw----. 1 grid asmadmin 69, 48 Dec 30 10:44 DATA04
...
There it was. The major numbers: 66, 67, 69. Those are SCSI device major numbers. For comparison, device-mapper multipath devices show major number 252:
brw-rw----. 1 grid asmadmin 252, 10 Dec 30 13:11 /dev/dm-10
During a sustained write test, I monitored per-device throughput with iostat -xm 2:
Device r/s w/s rMB/s wMB/s
sda 0.00 1847.00 0.00 800.12
sdb 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00
... (all other paths idle)
All 800 MB/s of write traffic was going through sda. The other 23 paths showed zero I/O. That’s the smoking gun.
The Root Cause
When I created the ASM disks using OracleASMLib, I used /dev/dm-* device paths:
# What I did (WRONG)
oracleasm createdisk DATA01 /dev/dm-11
oracleasm createdisk DATA02 /dev/dm-10
oracleasm createdisk DATA03 /dev/dm-9
oracleasm createdisk DATA04 /dev/dm-8
This looks correct because dm-* devices are device-mapper devices. But there’s a critical problem: OracleASMLib stamps the underlying SCSI device, not the multipath device.
You can verify this yourself. After stamping a disk with OracleASMLib, query it:
oracleasm querydisk -d DATA01
Output:
Disk "DATA01" is a valid ASM disk on device [66, 224]
That [66, 224] is major:minor. Major 66 is a SCSI disk device (/dev/sdc range), not device-mapper (which would be 252). You can confirm your system’s major number assignments:
cat /proc/devices | grep -E "^\s*(8|65|66|67|68|69|252)"
8 sd
65 sd
66 sd
67 sd
68 sd
69 sd
252 device-mapper
When OracleASMLib writes its header to /dev/dm-11, the kernel resolves which underlying path to use and stamps that specific SCSI device. From then on, ASM accesses the disk through that single SCSI path, bypassing multipath entirely. The impact: I/O goes through one FC path instead of 24, maximum throughput limited to one 32Gb HBA (~3.2 GB/s), and 23 of my 24 paths sit completely idle.
A note on OracleASMLib: the issue is specific to multipath devices. If you’re using OracleASMLib with simple local disks (no multipathing), it works fine. The problem arises because OracleASMLib wasn’t designed with multipath in mind. It stamps the physical device the kernel happens to resolve to, which for multipath is one of the underlying SCSI paths.
The Fix
The solution is to abandon OracleASMLib and use udev rules with /dev/mapper/WWID paths instead.
Step 1: Identify your multipath devices by WWID.
multipath -ll | grep -E "^[0-9a-f]{33}" | awk '{print $1}'
Example output:
3624a93703b701e383bbe421c0001ab90
3624a93703b701e383bbe421c0001ab91
3624a93703b701e383bbe421c0001ab92
...
These WWIDs are your stable device identifiers. They won’t change across reboots or path failures.
Step 2: Verify WWIDs match on both nodes.
# On rac-node-01
multipath -ll | grep -E "^3624a937" | awk '{print $1, $2}' | sort
# On rac-node-02
ssh rac-node-02 "multipath -ll | grep -E '^3624a937' | awk '{print \$1, \$2}'" | sort
The WWIDs should be identical. The dm-* numbers may differ between nodes (that’s fine and expected).
Step 3: Create udev rules for ASM disk permissions.
Create /etc/udev/rules.d/99-oracle-asmdevices.rules:
cat > /etc/udev/rules.d/99-oracle-asmdevices.rules << 'EOF'
# Oracle ASM Disk Permissions for RAC 19c
# Pure Storage FlashArray multipath devices
# CRS/GRID Disks (3 x 10GB)
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab90", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab91", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab92", OWNER="grid", GROUP="asmadmin", MODE="0660"
# DATA Disks (4 x 3TB)
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab93", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab94", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab95", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab96", OWNER="grid", GROUP="asmadmin", MODE="0660"
# FRA Disks (4 x 600GB)
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab97", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab98", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab99", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab9a", OWNER="grid", GROUP="asmadmin", MODE="0660"
# REDO Disks (4 x 100GB)
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab9b", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab9c", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab9d", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="dm-*", SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-3624a93703b701e383bbe421c0001ab9e", OWNER="grid", GROUP="asmadmin", MODE="0660"
EOF
Step 4: Copy udev rules to second node.
scp /etc/udev/rules.d/99-oracle-asmdevices.rules rac-node-02:/etc/udev/rules.d/
Step 5: Reload udev rules on both nodes.
# On rac-node-01
udevadm control --reload-rules
udevadm trigger --type=devices --action=change
# On rac-node-02
ssh rac-node-02 "udevadm control --reload-rules && udevadm trigger --type=devices --action=change"
Step 6: Set immediate permissions.
The udev rules handle persistence across reboots, but apply permissions immediately:
# On rac-node-01
for wwid in 3624a93703b701e383bbe421c0001ab90 \
3624a93703b701e383bbe421c0001ab91 \
3624a93703b701e383bbe421c0001ab92 \
3624a93703b701e383bbe421c0001ab93 \
3624a93703b701e383bbe421c0001ab94 \
3624a93703b701e383bbe421c0001ab95 \
3624a93703b701e383bbe421c0001ab96 \
3624a93703b701e383bbe421c0001ab97 \
3624a93703b701e383bbe421c0001ab98 \
3624a93703b701e383bbe421c0001ab99 \
3624a93703b701e383bbe421c0001ab9a \
3624a93703b701e383bbe421c0001ab9b \
3624a93703b701e383bbe421c0001ab9c \
3624a93703b701e383bbe421c0001ab9d \
3624a93703b701e383bbe421c0001ab9e; do
chown grid:asmadmin /dev/mapper/$wwid
chmod 0660 /dev/mapper/$wwid
done
Repeat on rac-node-02.
Step 7: Verify permissions and major numbers.
This is the critical verification. Every ASM disk must show major number 252 (device-mapper).
ls -l /dev/mapper/3624a93703b701e383bbe421c0001ab9*
Correct output:
brw-rw----. 1 grid asmadmin 252, 10 Dec 30 13:11 /dev/mapper/3624a93703b701e383bbe421c0001ab90
brw-rw----. 1 grid asmadmin 252, 11 Dec 30 13:11 /dev/mapper/3624a93703b701e383bbe421c0001ab91
brw-rw----. 1 grid asmadmin 252, 12 Dec 30 13:11 /dev/mapper/3624a93703b701e383bbe421c0001ab92
...
If you see major numbers 8, 65, 66, 67, 68, or 69, those are SCSI disk devices, not multipath.
Step 8: Configure Grid Infrastructure with correct discovery string.
In your Grid Infrastructure response file, use the /dev/mapper/WWID paths:
oracle.install.asm.diskGroup.disks=/dev/mapper/3624a93703b701e383bbe421c0001ab90,/dev/mapper/3624a93703b701e383bbe421c0001ab91,/dev/mapper/3624a93703b701e383bbe421c0001ab92
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/mapper/3624a937*
The discovery string /dev/mapper/3624a937* tells ASM to look for multipath devices by WWID prefix.
Step 9: Verify after installation.
After Grid Infrastructure is installed and ASM is running, verify the disk paths:
SELECT d.name, d.path FROM v$asm_disk d ORDER BY d.name;
Correct output:
NAME PATH
--------------- ------------------------------------------------------------
GRID_0000 /dev/mapper/3624a93703b701e383bbe421c0001ab91
GRID_0001 /dev/mapper/3624a93703b701e383bbe421c0001ab90
GRID_0002 /dev/mapper/3624a93703b701e383bbe421c0001ab92
DATA_0000 /dev/mapper/3624a93703b701e383bbe421c0001ab93
DATA_0001 /dev/mapper/3624a93703b701e383bbe421c0001ab94
...
If the paths show /dev/mapper/WWID, you’re using multipath correctly. If they show /dev/oracleasm/disks/* or /dev/sd*, you’re not.
The Result
After reinstalling with the correct configuration:
dd if=/dev/zero of=/u01/testfile bs=1M count=10000 oflag=direct 2>&1 | tail -1
Before (single path via OracleASMLib): 3.2 GB/s
After (multipath via udev rules): 11.8 GB/s
The iostat output during the second test showed balanced I/O across all paths, with the dm-* device reporting aggregate throughput instead of individual sd* devices. HBA utilization (checked via systool -c fc_host -v): all four HBAs showing active I/O instead of three sitting idle.
The one check that would have saved me a reinstall: look at the major number of your ASM devices. If it’s 252, you’re fine. If it’s anything else, you’re leaving performance on the table.