With a 12c Standard ASM (or a pre 12c Grid Infrastructure) all ASM clients connect locally using Oracle Bequeath NT Protocol (bypassing the listener process). Example from a 2-node 12c Standard Cluster with Standard ASM (OCR client):


SELECT s.inst_id, s.osuser, s.username, s.program, i.network_service_banner
FROM gv$session s, gv$session_connect_info i
WHERE s.sid = i.sid
AND   s.inst_id = i.inst_id
AND   s.program like 'crsd.bin%'
AND   i.network_service_banner like '%Protocol%'
ORDER BY 1;

   INST_ID OSUSER USERNAME PROGRAM                                           NETWORK_SERVICE_BANNER
---------- ------ -------- ------------------------------------------------ --------------------------------------------------------------------------------
     1     root   SYS      crsd.bin@clstdb01.trivadis.com (TNS V1-V3)        Oracle Bequeath NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
     2     root   SYS      crsd.bin@clstdb02.trivadis.com (TNS V1-V3)        Oracle Bequeath NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production

As you see in the example above, in a non-Flex ASM environment the ASM password file generally does not play an important role for the cluster and can be easily re-created.

But things change with Flex ASM... Because the ASM Flex instance is not always available on all cluster nodes, there is a change in the way the clients connect to ASM instance.

With 12c Flex ASM all local clients still connect to ASM using the Oracle Bequeath NT Protocol (bypassing the listener process). However, all remote clients connect using the TCP/IP NT Protocol via a new ASM Listener process (a new resource started on every HUB node in a cluster). For this architecture to work, the ASM password file needs to be placed in a disk group (shared storage) and the disk group ASM compatibility (compatible.asm) need to be set to at least the version 12.1.0.0.0.

srvctl config asm
ASM home: <CRS home>
Password file: +GRID/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

With the Flex ASM configuration the ASM password file plays a substantial role for the client connectivity and is a critical cluster component - which is not the case with older releases (or 12c Standard ASM). During the cluster start, the crsd.bin process on all nodes (but the OCR Writer), will connect remotely to the ASM Flex instance. Example from a 4-node Standard Cluster with Flex ASM (the OCR Writer is active on the cldb04.trivadis.com cluster node):

SELECT s.inst_id, s.osuser, s.username, s.program, i.network_service_banner
FROM gv$session s, gv$session_connect_info i
WHERE s.sid = i.sid
AND   s.inst_id = i.inst_id
AND   s.program like 'crsd.bin%'
AND   i.network_service_banner like '%Protocol%'
ORDER BY 1;

   INST_ID OSUSER USERNAME PROGRAM                                          NETWORK_SERVICE_BANNER
---------- ------ -------- ------------------------------------------------ --------------------------------------------------------------------------------
     4     root   SYS      crsd.bin@cldb04.trivadis.com (TNS V1-V3)         Oracle Bequeath NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
     4     root   SYS      crsd.bin@cldb02.trivadis.com (TNS V1-V3)         TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
     4     root   SYS      crsd.bin@cldb01.trivadis.com (TNS V1-V3)         TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
     4     root   SYS      crsd.bin@cldb03.trivadis.com (TNS V1-V3)         TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production

What happens internally during the cluster start ?

During the start the crsd.bin process on all nodes (but the OCR Writer), will connect remotely to the ASM Flex instance using a new ASM user crsuser__asm_001:

SQL> SELECT * FROM v$pwfile_users;

USERNAME         SYSDB SYSOP SYSAS SYSBA SYSDG SYSKM     CON_ID
---------------- ----- ----- ----- ----- ----- ----- ----------
SYS              TRUE  TRUE  TRUE  FALSE FALSE FALSE          0
CRSUSER__ASM_001 TRUE  FALSE TRUE  FALSE FALSE FALSE          0
ASMSNMP          TRUE  FALSE FALSE FALSE FALSE FALSE          0
ORACLE_001       TRUE  FALSE FALSE FALSE FALSE FALSE          0

The credentials are stored not only in the ASM password file, but also in the OLR (Oracle Local Registry). With the current implementation (version 12.1.0.2 plus PSU 12.1.0.2.1) re-creating the ASM password file after a failure (e.g. re-creating the ASM disk group hosting the password file) will not suffice to recover the system. There is currently also no procedure to re-create the credentials for the crsuser__asm_001 user (ASM password file as well as OLR). In such a case, you will be able to start the cluster only on the OCR writer node, on other cluster nodes during the unsuccessful tries to start the ora.storage resource, you will see the following errors reported in the ohasd_orarootagent_root.trc trace file:

2014-10-26 21:31:55.380587*:kgfn.c@1684: kgfnRecordErrPriv: rec=1
2014-10-26 21:31:57.010607 : default:2260707072: clsCredDomClose: Credctx deleted 0x7fd1603ed920
2014-10-26 21:31:57.010607*:kgfn.c@4868: kgfnConnect2: failed to connect
2014-10-26 21:31:57.010607*:kgfn.c@4887: kgfnConnect: conn=(nil)
2014-10-26 21:31:57.010607*:kgfp.c@669: kgfpInitComplete2 hdl=0x7fd1601f4ff8 conn=0x7fd1601f5010 ok=0
2014-10-26 21:31:57.010607*:kgfo.c@947: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid username/password; logon denied

2014-10-26 21:31:57.163377*:kgfn.c@6356: kgfnGetNodeType: flags=0x10
2014-10-26 21:31:57.163377*:kgfn.c@6369: kgfnGetNodeType: ntyp=1
2014-10-26 21:31:57.163377*:kgfn.c@4644: kgfnConnect2: kgfnGetBeqData failed
2014-10-26 21:31:57.163377*:kgfn.c@4680: kgfnConnect: srvr valid
2014-10-26 21:31:57.163377*:kgfn.c@5972: kgfnConnect2Int: sysasm=0 envflags=0x10 srvrflags=0x1 unam=crsuser__asm_001 password is NOT NULL pstr=_ocr
2014-10-26 21:31:57.163377*:kgfn.c@6121: kgfnConnect2Int: hosts=1
2014-10-26 21:31:57.163377*:kgfn.c@6134: kgfnConnect2Int: cstr=(DESCRIPTION=(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(LOAD_BALANCE=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.0.17)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))

Conclusion

In case you use a Flex ASM configuration, make sure that you incorporate the ASM password file into your Grid Infrastructure backup/recovery strategy, as it cannot be simply re-created with the necessary credentials. The password file can be backed up with the following command:

ASMCMD> pwcopy +GRID/orapwASM <Backup_Location>/orapwASM_<timestamp>
copying +GRID/orapwASM -> <Backup_Location>/orapwASM_<timestamp>

For the restore use the pwcopy ASMCMD command (use the --asm parameter, especially if you change the ASM password file location - the corresponding Clusterware resources will be updated automatically):

ASMCMD> pwcopy --asm <Backup_Location>/orapwASM_<timestamp> +GRID/orapwASM
copying <Backup_Location>/orapwASM_<timestamp> -> +GRID/orapwASM

An enhancement request (Bug 19585275 - 12C_ASM_X64: NEED A WAY TO RE-SETUP THE CREDENTIAL FOR CRSUSER__ASM_001 USER) has been already filed. With the future implementation the ASM password file should be backed up automatically, just like OCR and voting disks.

Update on 2014.12.09

If you were forced to re-create the ASM password file and do not have a valid backup of it, there is still a way to recover. You need to find out from the cluster configuration the automatically generated password for the user crsuser__asm_001.

First of all, you need to find out the correct path to the credential:

grid@cldb01:~/ [+ASM1] crsctl query credmaint -path ASM/Self
Path                                           Credtype   ID   Attrs            
                                                                                
/ASM/Self/159b7866d7ca5f63bf3e6099f0abf432     userpass   0    create_time=2014
                                                               -10-10 18:42:58,
                                                               modify_time=2014
                                                               -10-10 18:42:58,
                                                               expiration_time=
                                                               NEVER,bootstrap=
                                                               FALSE            

With the correct path you can query the user password:

grid@cldb01:~/ [+ASM1] crsctl get credmaint -path \
> /ASM/Self/159b7866d7ca5f63bf3e6099f0abf432 \
> -credtype userpass -id 0 -attr passwd -local

W0mqtCgA8Zj33h6GDC6gFvHlo4OhA


"W0mqtCgA8Zj33h6GDC6gFvHlo4OhA" string is in my test environment the automatically generated password for the user crsuser__asm_001. Knowing the correct password allows you to re-create correctly the user crsuser__asm_001 (after re-creating the ASM password). At the end don't forget to grant  the roles sysasm and sysdba to the user crsuser__asm_001.