'Multipathd: Asm!.Asm_ctl_spec: Failed To Store Path Info' - error in /var/log/mesage

The Problem

The following message will be seen in /var/log/messages when CRS stack is stopped/started.

When stopping the clusterware :

Nov 15 15:35:25 geeklab multipathd: ofsctl: remove path (uevent)
Nov 15 15:35:25 geeklab multipathd: ofsctl: spurious uevent, path not in pathvec
Nov 15 15:35:25 geeklab multipathd: uevent trigger error
Nov 15 15:35:26 geeklab kernel: ACFSK-0039: Module unloaded.
Nov 15 15:35:26 geeklab multipathd: asm!.asm_ctl_vbg8: remove path (uevent)
Nov 15 15:35:26 geeklab multipathd: asm!.asm_ctl_vbg8: spurious uevent, path not in pathvec
Nov 15 15:35:26 geeklab multipathd: uevent trigger error

When starting the clusterware:

Nov 15 15:39:01 geeklab kernel: [Oracle OKS] Module load succeeded. Build information: (LOW DEBUG)- (USM_11.2.0.2.0_LINUX.X64_100804.1) , Built on 2010/08/17 00:52:36
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_spec: add path (uevent)
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_spec: failed to store path info
Nov 15 15:39:01 geeklab multipathd: uevent trigger error
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vmb: add path (uevent)
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vmb: failed to store path info
Nov 15 15:39:01 geeklab multipathd: uevent trigger error
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vdbg: add path (uevent)
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vdbg: failed to store path info
Nov 15 15:39:01 geeklab multipathd: uevent trigger error
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vbg0: add path (uevent)
Nov 15 15:39:01 geeklab kernel: [Oracle ADVM] Module load succeeded. Build information: (LOW DEBUG) - USM_11.2.0.2.0_LINUX.X64_100804.1 Built on 2010/08/17 00:55:46
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vbg0: failed to store path info
Nov 15 15:39:01 geeklab multipathd: uevent trigger error
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vbg1: add path (uevent)
Nov 15 15:39:01 geeklab multipathd: asm!.asm_ctl_vbg1: failed to store path info
Nov 15 15:39:01 geeklab multipathd: uevent trigger error

Recently 11.2.0.2 has been installed or upgraded from previous version.

The Solution

The following devices have been created under /dev/asm as part of configuring ADVM/ACFS.

# ls -la /dev/asm
total 0
drwxrwx--- 2 root asmadmin 280 Nov 4 10:49 .
drwxr-xr-x 17 root root 4960 Nov 4 07:59 ..
brwxrwx--- 1 root asmadmin 252, 0 Nov 4 07:59 .asm_ctl_spec
brwxrwx--- 1 root asmadmin 252, 10 Nov 4 07:59 .asm_ctl_vbg0
brwxrwx--- 1 root asmadmin 252, 11 Nov 4 07:59 .asm_ctl_vbg1
brwxrwx--- 1 root asmadmin 252, 12 Nov 4 07:59 .asm_ctl_vbg2
brwxrwx--- 1 root asmadmin 252, 13 Nov 4 07:59 .asm_ctl_vbg3
brwxrwx--- 1 root asmadmin 252, 14 Nov 4 07:59 .asm_ctl_vbg4
brwxrwx--- 1 root asmadmin 252, 15 Nov 4 07:59 .asm_ctl_vbg5
brwxrwx--- 1 root asmadmin 252, 16 Nov 4 07:59 .asm_ctl_vbg6
brwxrwx--- 1 root asmadmin 252, 17 Nov 4 07:59 .asm_ctl_vbg7
brwxrwx--- 1 root asmadmin 252, 18 Nov 4 07:59 .asm_ctl_vbg8
brwxrwx--- 1 root asmadmin 252, 1 Nov 4 07:59 .asm_ctl_vdbg
brwxrwx--- 1 root asmadmin 252, 2 Nov 4 07:59 .asm_ctl_vmb

1. In some platforms, opening /dev/ofsctl by a 3rd party application could prevent CRS stack from being stopped properly.

2. This message can be suppressed by adding the following 2 lines in blacklist element in /etc/multipath.conf file.

# vi /etc/multipath.conf
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]*"
wwid "xxxxxxxxxxxxxxxxxxxxxxxxxx"
# wwid "*"
devnode "^asm/*"
devnode "ofsctl"
}