NFS未解之謎之IO持續(xù)占用問(wèn)題

某核心交易庫(kù)，報(bào)警IOwait超過(guò)30%，看似很普通的一條告警，實(shí)則暗藏玄機(jī)；登陸主機(jī)查看到有很多RMAN備份腳本在跑；

平時(shí)不到一小時(shí)的任務(wù)，跑了6個(gè)多小時(shí)了。
該備份任務(wù)通過(guò)nfs掛載的方式，懷疑是nfs有問(wèn)題，

果然，進(jìn)入到mount目錄，ll都無(wú)法正常顯示結(jié)果，卡住不動(dòng)，隨即到備份服務(wù)器上面，也就是nfs server查看有沒(méi)有異常，發(fā)現(xiàn)之前部署的一個(gè)監(jiān)控腳本，在平時(shí)load和io都很低。與今天故障時(shí)間段不太一樣。

返回到備份客戶(hù)端服務(wù)器，ps -ef |grep nfs發(fā)現(xiàn)有很多cat進(jìn)程
[root@trandb1 log]# ps -ef |grep nfs
root 9700 2 0 2017 ? 00:00:00 [nfsv4.0-svc]
oracle 88889 88888 0 10:05 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190619_9868_1
oracle 90224 90223 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9888_1
oracle 90566 90565 0 10:06 ? 00:00:00 cat ./nfs/full_data_TRANDB_20190619_9872_1
oracle 90571 90570 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190619_9869_1
oracle 90576 90575 0 10:06 ? 00:00:00 cat ./nfs/full_data_TRANDB_20190619_9872_1
oracle 90584 90583 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190619_9868_1
oracle 90588 90587 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9884_1
oracle 90593 90592 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9885_1
oracle 90597 90596 0 10:06 ? 00:00:00 cat ./nfs/full_data_TRANDB_20190619_9865_1
oracle 90606 90605 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9881_1
oracle 90616 90615 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190619_9871_1
oracle 90626 90625 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9887_1
oracle 90631 90630 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9888_1
oracle 90641 90640 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190619_9871_1
oracle 90645 90644 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9880_1
oracle 91999 91998 0 10:06 ? 00:00:00 cat ./nfs/full_data_TRANDB_20190620_9883_1
oracle 92488 92487 0 10:06 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190620_9880_1
oracle 93837 93836 0 10:07 ? 00:00:00 cat ./nfs/arch_TRANDB_20190620_9890_1
oracle 94011 94010 0 10:07 ? 00:00:00 cat ./nfs/full_data_TRANDB_20190620_9886_1
oracle 94238 94237 0 10:07 ? 00:00:01 cat ./nfs/full_data_TRANDB_20190619_9865_1
root 98024 17863 0 10:09 pts/7 00:00:00 grep nfs
root 130976 2 0 2017 ? 00:00:00 [nfsiod]

通過(guò)操作系統(tǒng)kill掉這些pid，但是立馬就會(huì)派生出來(lái)。后umount掉目錄后，沒(méi)有了。暫時(shí)沒(méi)有找出原因，記錄一下。