ORA-19909(一次DataGuard Failover導(dǎo)致的故障

      某醫(yī)院客戶，核心數(shù)據(jù)庫(kù)有兩個(gè)備庫(kù)。因想將其中一個(gè)備庫(kù)主機(jī)挪作測(cè)試使用，所以，想把數(shù)據(jù)庫(kù)切換成讀寫模式。這樣，就涉及到了做備庫(kù)的failover動(dòng)作。這個(gè)failover是在另外一個(gè)備庫(kù)，主庫(kù)都存活的情況下進(jìn)行的。

      Faiover進(jìn)行得很順利。Failover之后，修改了此數(shù)據(jù)庫(kù)的DG的幾個(gè)參數(shù)，當(dāng)天以為事情就這么完了。

      第二日，客戶告知另一個(gè)備庫(kù)的數(shù)據(jù)沒(méi)有同步。于是登錄過(guò)去，看到告警日志中有如下錯(cuò)誤

MRP0: Incarnation has changed! Retry recovery...

Errors in file /oradata/u01/app/diag/rdbms/orcldg/orcl/trace/orcl_pr00_3272.trc:

ORA-19906: recovery target incarnation changed during recovery

Managed Standby Recovery not using Real Time Apply

Recovery interrupted!

Recovered data files to a consistent state at change 19274002051

Tue Jun 09 16:37:39 2020

 started logmerger process

Tue Jun 09 16:37:39 2020

Managed Standby Recovery starting Real Time Apply

Warning: Recovery target destination is in a sibling branch

of the controlfile checkpoint. Recovery will only recover

changes to datafiles.

Datafile 1 (ckpscn 19274002051) is orphaned on incarnation#=3

MRP0: Detected orphaned datafiles! 

Recovery will possibly be retried after flashback...

Errors in file /oradata/u01/app/diag/rdbms/orcldg/orcl/trace/orcl_pr00_27714.trc:

ORA-19909: datafile 1 belongs to an orphan incarnation

ORA-01110: data file 1: '/oradata/datafile/system01.dbf'

通過(guò)RMAN查看incarnation

connected to target database: ORCL (DBID=1385973181)

RMAN> list incarnation;

using target database control file instead of recovery catalog

List of Database Incarnations

DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time

------- ------- -------- ---------------- --- ---------- ----------

1       1       ORCL     1385973181       PARENT  1          13-SEP-14

2       2       ORCL     1385973181       PARENT  4733207525 12-FEB-15

3       3       ORCL     1385973181       CURRENT 8619762801 08-JUL-17

4       4       ORCL     1385973181       ORPHAN  19273953428 09-JUN-20

其中incarnation 4正是昨天做failover產(chǎn)生的。

解決起來(lái)其實(shí)也相對(duì)較為簡(jiǎn)單：

reset database incarnation to 3;

然后在開(kāi)啟日志應(yīng)用即可

Wed Jun 10 11:10:26 2020

RFS[5]: Assigned to RFS process 19310

RFS[5]: Allowing overwrite of partial archivelog for thread 2 sequence 40151

RFS[5]: Opened log for thread 2 sequence 40151 dbid 1385973181 branch 948759254

Completed: alter database recover managed standby database using current logfile disconnect

Archived Log entry 72765 added for thread 2 sequence 40151 rlc 948759254 ID 0x5d2e975e dest 2:

Media Recovery Log /oradata/arch/2_40151_948759254.dbf

Media Recovery Waiting for thread 1 sequence 1748

Archivelog record exists, but no file is found

RFS[5]: Opened log for thread 1 sequence 36160 dbid 1385973181 branch 948759254

Archived Log entry 72766 added for thread 1 sequence 36160 rlc 948759254 ID 0x5d2e975e dest 2:

MRP0: Log being waited on was inappropriate for recovery. Retry recovery...

Errors in file /oradata/u01/app/diag/rdbms/orcldg/orcl/trace/orcl_pr00_19224.trc:

ORA-16426: Recovery requested an incorrect log from which to apply redo data.

Managed Standby Recovery not using Real Time Apply

Wed Jun 10 11:10:34 2020

Recovery interrupted!

Wed Jun 10 11:10:54 2020

 started logmerger process

Wed Jun 10 11:10:55 2020

Managed Standby Recovery starting Real Time Apply

Parallel Media Recovery started with 16 slaves

Waiting for all non-current ORLs to be archived...

All non-current ORLs have been archived.

Media Recovery Log /oradata/arch/2_40151_948759254.dbf

Media Recovery Log /oradata/arch/1_36160_948759254.dbf

Media Recovery Log /oradata/arch/2_40152_948759254.dbf

Media Recovery Log /oradata/arch/1_36161_948759254.dbf

此次故障，應(yīng)當(dāng)在做failover之前修改主備庫(kù)的DG相關(guān)參數(shù)。作為數(shù)據(jù)庫(kù)工程師，應(yīng)該考慮更為周全。

幸而影響范圍較小。