oracle RAC 集群无法启动

RDBMS 11.2.0.4

节点1无法启动，节点2 启动正常。

节点1的集群状态一直这样

[[email protected] admin]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

但是节点2的集群状态是正常的。重启节点1和节点2 ，问题依旧。

在关闭节点1的集群的时候，提示没有集群，加参数 -f后强制可以关闭。

查看节点1的相关log ：

2022-06-24 00:05:30.584:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:05:35.813:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:05:41.031:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:05:41.244:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:05:46.470:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:05:51.645:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:05:56.871:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:06:02.084:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:06:02.294:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"
2022-06-24 00:06:07.516:
[/u01/app/grid/bin/oraagent.bin(23516)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/grid/log/ractest1/agent/ohasd/oraagent_grid/oraagent_grid.log"

继续查看相关log 

-- asm log 

Fatal NI connect error 12649, connecting to:
 (LOCAL=NO)

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.4.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  Time: 01-JUN-2022 14:38:25
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12649

TNS-12649: Unknown encryption or data integrity algorithm
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
opiodr aborting process unknown ospid (47070) as a result of ORA-609
Wed Jun 01 14:43:25 2022


***********************************************************************

2022-06-24 00:06:23.291: [ora.asm][532666112]{0:0:2} [start] InstAgent::start call sysresv
2022-06-24 00:06:23.291: [ora.asm][532666112]{0:0:2} [start] Container:start scls_clean_oracle_ipc Container +ASM1 dbHome /u01/app/grid
2022-06-24 00:06:23.301: [ora.asm][532666112]{0:0:2} [start] clsnInstConnection::makeConnectStr UsrOraEnv  m_oracleHome /u01/app/grid Crshome /u01/app/grid
2022-06-24 00:06:23.301: [ora.asm][532666112]{0:0:2} [start] makeConnectStr = (DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/u01/app/grid/bin/oracle)(ARGV0=oracle+ASM1)(ENVS='ORACLE_HOME=/u01/app/grid,ORACLE_SID=+ASM1')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(SID=+ASM1)))
2022-06-24 00:06:23.302: [ora.asm][532666112]{0:0:2} [start] InstConnection::connectInt: server not attached
2022-06-24 00:06:23.314: [ora.asm][532666112]{0:0:2} [start] ORA-12547: TNS:lost contact

2022-06-24 00:06:23.314: [ora.asm][532666112]{0:0:2} [start] InstConnection::connectInt (1) Exception OCIException
2022-06-24 00:06:23.314: [ora.asm][532666112]{0:0:2} [start] InstConnection:connect:excp OCIException OCI error 12547
2022-06-24 00:06:23.314: [ora.asm][532666112]{0:0:2} [start] InstConnection::connectInt: server not attached
2022-06-24 00:06:23.323: [ora.asm][532666112]{0:0:2} [start] ORA-12547: TNS:lost contact

尝试启动asm实例，报错

[[email protected] ~]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jun 24 00:30:09 2022

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

SQL> conn /  as sysasm
ERROR:
ORA-12547: TNS:lost contact


SQL> exit

 

MOS上搜文档，查到相关文档：11gR2 ASM Fails to Start With "ORA-12547: TNS:lost contact" and ORA-12649 due to Incorrect Network Security Setting (Doc ID 1505025.1)

与该文档中的相关日志进行对比，一抹一样。

该文档定位为grid用户下的sqlnet.ora中设置了加密。查看相关内容，果然加密

[[email protected] admin]$ more sqlnet.ora
# sqlnet.ora.ractest1 Network Configuration File: /u01/app/grid/network/admin/sqlnet.ora.ractest1
# Generated by Oracle configuration tools.

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

ADR_BASE = /u01/app/gridbase

#SQLNET.ENCRYPTION_SERVER = REQUIRED
#SQLNET.ENCRYPTION_TYPES_SERVER = RC4_256
#SQLNET.CRYPTO_CHECKSUM_SERVER = REQUIRED
#SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER = RC4_256
[[email protected] admin]$ 

另一个节点就没有加密：

[[email protected] admin]$ more sqlnet.ora 
# sqlnet.ora.ractest2 Network Configuration File: /u01/app/grid/network/admin/sqlnet.ora.ractest2
# Generated by Oracle configuration tools.

NAMES.DIRECTORY_PATH= (TNSNAMES, EZCONNECT)

ADR_BASE = /u01/app/gridbase

[[email protected] admin]$ 

根据mos上的说明，屏蔽掉加密，再次启动，正常启动。问题解决。

参考文档：11gR2 ASM Fails to Start With "ORA-12547: TNS:lost contact" and ORA-12649 due to Incorrect Network Security Setting (Doc ID 1505025.1)

文档内容：

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 and later
Information in this document applies to any platform.

SYMPTOMS

11gR2 ASM fails to start while GI is restarting:

• Output from "crsctl start res ora.asm -init" command 

CRS-2672: Attempting to start 'ora.asm' on 'racnode1'
ORA-12547: TNS:lost contact
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-12547: TNS:lost contactCRS-2674: Start of 'ora.asm' on 'racnode1' failed
CRS-2679: Attempting to clean 'ora.asm' on 'racnode1'
ORA-12547: TNS:lost contact
CRS-2681: Clean of 'ora.asm' on 'racnode1' succeeded
CRS-4000: Command Start failed, or completed with errors.

  

• Output from sqlplus

$ export ORACLE_SID=<+ASMn>
$ export ORACLE_HOME=<GRID_HOME>
$ $ORACLE_HOME/bin/sqlplus / as sysasm

SQL*Plus: Release 11.2.0.2.0 Production on Fri Nov 9 16:38:52 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-12547: TNS:lost contact

  

• <GRID_USER_HOME_PATH>/oradiag_<GRID_USER>/diag/clients/user_<GRID_USER>/host_<HOST_ID>/trace/sqlnet.log 

Fatal NI connect error 12547, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=/orcl/grid/oracle/product/11.2.0.2/grid/bin/oracle)(ARGV0=oracle+ASM1)(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))')(DETACH=NO))(CONNECT_DATA=(CID=(PROGRAM=sqlplus)(HOST=racnode1)(USER=oracle))))

 VERSION INFORMATION:
       TNS for Linux: Version 11.2.0.2.0 - Production
       Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.2.0 - Production
 Time: 09-NOV-2012 17:36:36
 Tracing not turned on.
 Tns error struct:
   ns main err code: 12547

TNS-12547: TNS:lost contact
   ns secondary err code: 12560
   nt main err code: 517

TNS-00517: Lost contact
   nt secondary err code: 32
   nt OS err code: 0

• <GRID_HOME>/rdbms/log/+asm1_ora_<PID>.trc

*** 2012-11-09 18:12:51.661
opiino: Attach failed due to ORA-12649

 

CAUSE

sqlnet.ora changed in $TNS_ADMIN or $GRID_HOME/network/admin directory to add encryption, i.e. 

SQLNET.ENCRYPTION_TYPES_SERVER={3DES168}
SQLNET.ENCRYPTION_SERVER=REQUIRED

SOLUTION

Restore sqlnet.ora from backup and start ASM from sqlplus.