ORACLE OMS啟動失敗之BEA-300040 I/O error while writing node manager status

        今天處理一個oracle優(yōu)化管理工具oms啟動異常的問題。oms問題處理之前，oms所在服務(wù)器上的一套asm和db所用

的審計目錄文件系統(tǒng)inode耗盡導(dǎo)致asm和db異常宕機(jī)，經(jīng)清理審計目錄的日志后asm和db正常啟動，但是oms啟動異常。

一、OMS啟動異常分析

1、OMS啟動異常信息

[oracle@orcl bin]$ ./emctl start oms

Oracle Enterprise Manager Cloud Control 12c Release 4  

Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.

Starting Oracle Management Server...

Starting WebTier...

WebTier Successfully Started

Oracle Management Server Could Not Be Started

Check EM Server log file for details: /u01/app/oracle/oms/domains/GCDomain/servers/ORCL_OMS1/logs/ORCL_OMS1.out

Oracle Management Server is Down

[oracle@ orcl bin]$ 

2、查看 ORCL_OMS1.out日志

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.audit.AuditServiceImpl registerInternal

WARNING: Cannot register to audit service for component "JPS".

Mar 19, 2020 5:55:44 PM oracle.security.jps.internal.policystore.ldap.LdapPolicyStore initial

INFO: Container is not ready at the point null

<Mar 19, 2020 5:55:53 PM CST> <Error> <NodeManager> <BEA-300040> <I/O error while writing node manager status change message.

java.io.IOException: No space left on device

        at java.io.UnixFileSystem.createFileExclusively(Native Method)

        at java.io.File.checkAndCreate(File.java:1705)

        at java.io.File.createTempFile0(File.java:1726)

        at java.io.File.createTempFile(File.java:1803)

        at weblogic.nodemanager.util.ConcurrentUnixFile.write(ConcurrentUnixFile.java:38)

        Truncated. see log file for complete stacktrace

> 

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <Waiting for the process to die: 6906>

<Mar 19, 2020 5:55:54 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>

<Mar 19, 2020 5:55:54 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

3、根據(jù).out日志提示，跟之前asm和db啟動異常報錯一致，也是之前oracle數(shù)據(jù)庫審計日志過多導(dǎo)致文件系統(tǒng)inode耗盡

其他進(jìn)程無法寫入導(dǎo)致。

二、問題處理

1、對于oms來說，單純的重啟oms無法解決問題，因?yàn)閚odemanager進(jìn)程未正常停止

2、使用ps -ef|grep java，把oms相關(guān)的java進(jìn)程殺掉kill -9 java_process_number

3、再次嘗試啟動oms進(jìn)程，成功啟動