Ora-27300, ora-27301, ora-27302, ora-27303, tns-2518, tns-12549, tns-12560, tns-00519 and other alarm processing

RDBMS 10.2.0.4

AIX 6.1 

Reference documents ：

ORA-27300 ORA-27301 ORA-27302 ORA-27303 skgpspawn5 (Doc ID 1125188.1)

Repeating TNS-12518, TNS-12549, ... Error: 11: 'Resource Temporarily Unavailable' Errors in listener log (Doc ID 430686.1)

https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwaix/oracle-grid-infrastructure-installation-checklist.html#GUID-71A93E07-7E50-449C-B425-02F04A2EA8E6   -- 12c Of aix Installation document
 

stay alert log The following alarms appear in: skgpspawn3,skgpspawn5

ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
Wed Jul 20 10:29:07 2022
Process J000 died, see its trace file
Wed Jul 20 10:29:07 2022
kkjcre1p: unable to spawn jobq slave process
Wed Jul 20 10:29:07 2022

ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn5
ORA-27303: additional information: skgpspawn5

On the monitor's log in , The following content keeps appearing ：

TNS-12518: TNS:listener could not hand off client connection
 TNS-12549: TNS:operating system resource quota exceeded
  TNS-12560: TNS:protocol adapter error
   TNS-00519: Operating system resource quota exceeded
    IBM/AIX RISC System/6000 Error: 12: Not enough space

At the same time, the application cannot access the database .

By the way, there were some small problems a few days ago , I reported this occasionally before , Midnight newspaper , But later, you can use the database . So I didn't care too much .

After a few days, I found ,lsnrctl stauts Find out hung Live in the . Check and find that the listener log has 4.2G 了 . Clean up the listener log , still lsnrctl status Meeting hung live , Check the listener log , There is no listener log generated . But the business did not report a failure . Then normal reload The monitor was .

A day later , The business informed that the database could not be connected , Take a look , There are a lot of TNS Report errors ,alert log Also began to report errors .

Then check AIX Parameters of , as follows , Find out maxuproc yes 8192,ncargs yes 256.

# lsattr -El sys0 -a ncargs -a maxuproc -a maxpout -a minpout -a fullcore
ncargs   256   ARG/ENV list size in 4K byte blocks             True
maxuproc 8192 Maximum number of PROCESSES allowed per user    True
maxpout  8193  HIGH water mark for pending write I/Os per file True
minpout  4096  LOW water mark for pending write I/Os per file  True
fullcore false Enable full CORE dump                           True

meanwhile , see 10g Official documents of the database , It is found that the parameter setting is not optimal （ The official recommended setting is 16384）, Then set the parameters as follows ：

chdev -l sys0 -a 'minpout=4096 maxpout=8193' -a ncargs=1024 -a maxuproc=16384 -a fullcore=true	

The parameters after setting are as follows ：

# lsattr -El sys0 -a ncargs -a maxuproc -a maxpout -a minpout -a fullcore
ncargs   1024  ARG/ENV list size in 4K byte blocks             True
maxuproc 16384 Maximum number of PROCESSES allowed per user    True
maxpout  8193  HIGH water mark for pending write I/Os per file True
minpout  4096  LOW water mark for pending write I/Os per file  True
fullcore true  Enable full CORE dump                           True

And then , restart AIX, After starting the listener and database . Problem solving .（ In fact, you can not restart AIX, Just restart the listener and database ）

And then , After more than a week of observation , There has never been any warning of the above problems .

END.