溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
環(huán)境：RHEL5.8 RAC 11.2.0.3.0
1：查看ORC和voting disk信息：
In 11g Release 2 your voting disk data is automatically backed up in the OCR whenever there is a configuration change.
所以恢復時恢復備份OCR即可，這里和10g是不同的，不需要備份voting disk，備份OCR即可
2：使用cluvfy 工具檢查OCR完整性
[grid@rac1 ~]$ cluvfy comp ocr -n all
Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ASM Running check passed. ASM is running on all specified nodes
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
Disk group for ocr location "+CRSDATA" available on all the nodes
NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.
OCR integrity check passed
Verification of OCR integrity was successful.
3：使用ocrcheck檢測OCR內容的完整性
[grid@rac1 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3016
Available space (kbytes) : 259104
ID : 1236405787
Device/File Name : +CRSDATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user --如果使用root用戶執(zhí)行ocrcheck時，會顯示Logical corruption check succeeded
4：檢測voting disk的信息
[grid@rac1 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 2b1bd0c122584f5abf72033b2b2d26bd (/dev/asm-b_crs) [CRSDATA]
2. ONLINE 2bc03776cdd94f5cbfb9165c473fdb0e (/dev/asm-c_crs) [CRSDATA]
3. ONLINE 3b43c39513a64f2dbf7083a9510ada89 (/dev/asm-d_crs) [CRSDATA]
Located 3 voting disk(s).
從上面看出，OCR和voting disk都位于+CRSDATA磁盤組 ，注意+CRSDATA磁盤組還有ASM的啟動參數(shù)文件，ASM啟動是根據(jù)磁盤頭的kfdhdb.spfile指向ASM上的此磁盤的UA NUMBER從而讀取spfile文件
5：手動備份一份OCR信息：
[root@rac1 grid]# ocrconfig -export /tmp/ocr_20130717.dmp
[root@rac1 grid]# ll /tmp/ocr_20130717.dmp -h
-rw------- 1 root root 102K Jul 17 14:45 /tmp/ocr_20130717.dmp
6：查看OCR自動備份信息
[grid@rac1 ~]$ ocrconfig -showbackup
rac1 2013/07/16 15:45:24 /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr
rac2 2013/07/16 08:13:38 /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup01.ocr
rac2 2013/07/16 04:14:09 /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup02.ocr
rac2 2013/07/16 00:14:38 /u01/app/11.2.0.3/grid/cdata/ad-cluster/day.ocr
rac2 2013/07/07 04:40:11 /u01/app/11.2.0.3/grid/cdata/ad-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
7：保存一份ASM參數(shù)文件，如果提前沒保存，可以到$CRS_HOME/dbs/init.ora獲取一份，后面此啟動參數(shù)的詳細內容
[grid@rac1 dbs]$ sqlplus / as sysasm
SQL> create pfile='/tmp/asm_pfile_130717.txt' from spfile;
File created.
8：破壞保存OCR信息的磁盤組+CRSDATA
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-b_crs bs=1024 count=1000
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-c_crs bs=1024 count=1000
9：破壞了磁盤b和c后，都檢測通過，沒報錯，在rac1和rac2停止crs
[root@rac1 dev]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
.....................
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac2 dev]# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1'
.....................
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 dev]# ps -ef |grep ora_
root 16189 32265 0 16:26 pts/0 00:00:00 grep ora_
[root@rac1 dev]# ps -ef |grep asm_
root 16195 32265 0 16:26 pts/0 00:00:00 grep asm_
10:再啟動CRS,報錯
[root@rac1 dev]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 ~]# tail -50f /u01/app/11.2.0.3/grid/log/rac1/alertrac1.log
[cssd(16559)]CRS-1637:Unable to locate configured voting file with ID 2b1bd0c1-22584f5a-bf72033b-2b2d26bd; details at (:CSSNM00020:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:15.947
[cssd(16559)]CRS-1637:Unable to locate configured voting file with ID 2bc03776-cdd94f5c-bfb9165c-473fdb0e; details at (:CSSNM00020:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:15.947
[cssd(16559)]CRS-1705:Found 1 configured voting files but 2 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:15.948
[cssd(16559)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/11.2.0.3/grid/log/rac1/cssd/ocssd.log
2013-07-17 16:28:16.073
[cssd(16559)]CRS-1603:CSSD on node rac1 shutdown by user.
ocrcheck檢測報錯：
[root@rac1 dev]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage
11：強制關閉CRS:
[root@rac1 dev]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
[root@rac1 dev]# crsctl stop crs -f
CRS-2797: Shutdown is already in progress for 'rac1', waiting for it to complete
CRS-2797: Shutdown is already in progress for 'rac1', waiting for it to complete
CRS-4133: Oracle High Availability Services has been stopped.
12：以獨占模式啟動rac1
[root@rac1 dev]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac1'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2676: Start of 'ora.drivers.acfs' on 'rac1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
12：創(chuàng)建CRSVOTEDISK磁盤組以及spfile
[grid@rac1 ~]$ asmcmd
ASMCMD> ls
空的
[grid@rac1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Wed Jul 17 16:58:18 2013
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile
NAME TYPEVALUE
------------------------------------ ----------- ------------------------------
spfile string
SQL> create diskgroup CRSVOTEDISK normal redundancy disk '/dev/asm-b_crs','/dev/asm-c_crs', '/dev/asm-d_crs'
2 attribute 'compatible.asm'='11.2.0.0.0', 'compatible.rdbms'='11.2.0.0.0';
create diskgroup CRSVOTEDISK normal redundancy disk '/dev/asm-b_crs','/dev/asm-c_crs', '/dev/asm-d_crs'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15033: disk '/dev/asm-d_crs' belongs to diskgroup "CRSDATA" --這里報錯是因為asm-d_crs沒清除磁盤頭信息
清除asm-d_crs磁盤頭信息
[root@rac1 dev]# dd if=/dev/zero of=/dev/asm-d_crs bs=1024 count=1000
SQL> create diskgroup CRSVOTEDISK normal redundancy disk '/dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'
2 attribute 'compatible.asm'='11.2.0.0.0', 'compatible.rdbms'='11.2.0.0.0';
Diskgroup created.
SQL> create spfile='+CRSVOTEDISK' from pfile='/tmp/asm_pfile_130717.txt';
File created.
SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@rac1 ~]$ asmcmd
ASMCMD> ls
CRSVOTEDISK/
ASMCMD> ls CRSVOTEDISK
ad-cluster/
ASMCMD> ls CRSVOTEDISK/ad-cluster/
ASMPARAMETERFILE/
ASMCMD> ls CRSVOTEDISK/ad-cluster/ASMPARAMETERFILE
REGISTRY.253.821034567
13：Restore OCR from backup:
將原磁盤組+CRSDATA改為新建立的磁盤組 +CRSVOTEDISK
[root@rac1 dev]# vim /etc/oracle/ocr.loc
ocrconfig_loc=+CRSVOTEDISK
local_only=FALSE
[root@rac1 dev]# ocrconfig -restore /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr
可以看到增加了一個OCRFILE文件夾
ASMCMD> ls CRSVOTEDISK/ad-cluster
ASMPARAMETERFILE/
OCRFILE/
ASMCMD> ls CRSVOTEDISK/ad-cluster/OCRFILE -l
Type Redund Striped Time Sys Name
OCRFILE MIRROR COARSE JUL 17 17:00:00 Y REGISTRY.255.821036449
ASMCMD> ls CRSVOTEDISK/ad-cluster/ASMPARAMETERFILE -l
Type Redund Striped Time Sys Name
ASMPARAMETERFILE MIRROR COARSE JUL 17 17:00:00 Y REGISTRY.253.821034567
檢測成功
[root@rac1 dev]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3016
Available space (kbytes) : 259104
ID : 1236405787
Device/File Name : +CRSVOTEDISK
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
14：Restore the Voting Disk:
[root@rac1 dev]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. OFFLINE 2b1bd0c122584f5abf72033b2b2d26bd () []
2. OFFLINE 2bc03776cdd94f5cbfb9165c473fdb0e () []
3. ONLINE 3b43c39513a64f2dbf7083a9510ada89 (/dev/asm-d_crs) [CRSDATA]
Located 3 voting disk(s).
[root@rac1 dev]# crsctl replace votedisk +CRSVOTEDISK
CRS-4602: Failed 27 to add voting file 5818c2c531394f45bff13c5a7532c8d4.
CRS-4602: Failed 27 to add voting file 1ce0436528624faabf7d4a1dd8dc978a.
CRS-4602: Failed 27 to add voting file 09def2b244af4f42bf13679a8aa0ff73.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failure 27 with Cluster Synchronization Services while deleting voting disk.
Failed to replace voting disk group with +CRSVOTEDISK.
CRS-4000: Command Replace failed, or completed with errors.
這里報錯是一開始asm-d_crs沒清除磁盤頭信息導致的
======================================== 到這里恢復voting disk失敗了 ，下面重新開始再次嘗試恢復============
下面恢復時要注意：
crsctl start crs -excl -nocrs 啟動后，馬上關閉ASM,不要立刻創(chuàng)建create diskgroup CRSVOTEDISK磁盤組，再使用參數(shù)啟動ASM
不然創(chuàng)建磁盤組時可能會收入如下報錯：
例如：（下面的操作看看就好了，直到 ：下面開始再次恢復操作）
[grid@rac1 ~]$ sqlplus / as sysasm
SQL> create diskgroup CRSVOTEDISK normal redundancy disk '/dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'
2 attribute 'compatible.asm'='11.2.0.0.0', 'compatible.rdbms'='11.2.0.0.0';
create diskgroup CRSVOTEDISK normal redundancy disk '/dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
ORA-15031: disk specification '/dev/asm-d_crs' matches no disks
ORA-15014: path '/dev/asm-d_crs' is not in the discovery set
ORA-15031: disk specification '/dev/asm-c_crs' matches no disks
ORA-15014: path '/dev/asm-c_crs' is not in the discovery set
ORA-15031: disk specification '/dev/asm-b_crs' matches no disks
ORA-15014: path '/dev/asm-b_crs' is not in the discovery set --這里找不到設備應該也是和下面的情況是一樣的，沒指定掃描的路徑
SQL> col PATH for a50
SQL> select group_number, disk_number, mount_status, header_status, path from v$asm_disk;
no rows selected
說明沒識別出磁盤 ，這里為什么沒磁盤現(xiàn)在是搞明白了，因為參數(shù)里面根本沒設置
SQL> show parameter asm
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups stringDATA ---這里使用默認參數(shù)文件啟動時是空的，正常情況也不會顯示保存OCR磁盤組名的
asm_diskstring string/dev/asm* ---這里使用默認參數(shù)文件啟動時是空的，沒指定掃描的路徑
asm_power_limit integer1
asm_preferred_read_failure_groups string
所以為了保險起見，應該crsctl start crs -excl -nocrs 啟動后，馬上關閉ASM,不要立刻創(chuàng)建create diskgroup CRSVOTEDISK磁盤組，再使用參數(shù)啟動ASM
SQL> startup pfile='/tmp/asm_pfile_130717.txt';
[grid@rac1 ~]$ cat /tmp/asm_pfile_130717.txt
+ASM1.__oracle_base='/u01/app/grid'#ORACLE_BASE set from in memory value
+ASM1.asm_diskgroups='DATA'#Manual Mount
+ASM2.asm_diskgroups='DATA'#Manual Mount
*.asm_diskstring='/dev/asm*'
*.asm_power_limit=1
*.diagnostic_dest='/u01/app/grid'
*.instance_type='asm'
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'
這里再查下v$asm_disk就可以查詢到磁盤，也可以順利的創(chuàng)建磁盤組了。。。。。。。。。。，就是因為沒立刻關閉ASM,使用修改好的參數(shù)文件，創(chuàng)建磁盤組時一直提示找不到磁盤，耽誤了半天時間
下面開始再次恢復操作：
關閉crs后再啟動
[root@rac1 dev]# crsctl stop crs
[root@rac1 dev]# crsctl start crs -excl -nocrs
[root@rac1 dev]# crsctl query css votedisk
Located 0 voting disk(s).
關閉rac1上的ASM，再使用參數(shù)文件啟動ASM，創(chuàng)建CRS磁盤組，創(chuàng)建spfile
[grid@rac1 ~]$ sqlplus / as sysasm
SQL>shutdown immediate
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup pfile='/tmp/asm_pfile_130717.txt';
ASM instance started
SQL> col path for a50
SQL> set linesize 130
SQL> select group_number, disk_number, mount_status, header_status, path from v$asm_disk;
GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU PATH
-------------------------------------- ------------ ---------------- ------------------------------
0 0 CLOSED MEMBER /dev/asm-e_data
0 3 CLOSED CANDIDATE /dev/asm-b_crs
0 2 CLOSED CANDIDATE /dev/asm-c_crs
0 1 CLOSED CANDIDATE /dev/asm-d_crs
SQL> create diskgroup CRSVOTEDISK normal redundancy disk '/dev/asm-b_crs','/dev/asm-c_crs','/dev/asm-d_crs'
2 attribute 'compatible.asm'='11.2.0.0.0', 'compatible.rdbms'='11.2.0.0.0';
Diskgroup created.
SQL> create spfile='+CRSVOTEDISK ' from pfile='/tmp/asm_pfile_130717.txt';
File created.
SQL> quit
恢復crs
[root@rac1 dev]# ocrconfig -restore /u01/app/11.2.0.3/grid/cdata/ad-cluster/backup00.ocr
恢復voting disk
[root@rac1 dev]# crsctl replace votedisk +CRSVOTEDISK
Successful addition of voting disk 1b00b0ec4e504f7fbf1f8d20fbbfaa4b.
Successful addition of voting disk 5a3b646433124fdcbf23c3c290de7fe3.
Successful addition of voting disk 5d27d80b96d74f09bf1756be6dee387f.
Successfully replaced voting disk group with +CRSVOTEDISK .
CRS-4266: Voting file(s) successfully replaced
檢測
[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3016
Available space (kbytes) : 259104
ID : 1236405787
Device/File Name : +CRSVOTEDISK
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 1b00b0ec4e504f7fbf1f8d20fbbfaa4b (/dev/asm-b_crs) [CRSVOTEDISK ]
2. ONLINE 5a3b646433124fdcbf23c3c290de7fe3 (/dev/asm-c_crs) [CRSVOTEDISK ]
3. ONLINE 5d27d80b96d74f09bf1756be6dee387f (/dev/asm-d_crs) [CRSVOTEDISK ]
Located 3 voting disk(s).
停止crs以正常方式啟動：
[root@rac1 ~]# crsctl stop crs
[root@rac1 ~]# crsctl start crs
此時，crs和voting disk已經(jīng)完成恢復，但要注意修改rac2上的/etc/oracle/ocr.loc里面的ocrconfig_loc=+CRSVOTEDISK ，不然啟動報錯：
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(19510)]CRS-5019:All OCR locations are on ASM disk groups [CRSDATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0.3/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2013-07-18 00:10:33.678
[/u01/app/11.2.0.3/grid/bin/oraagent.bin(19510)]CRS-5019:All OCR locations are on ASM disk groups [CRSDATA], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0.3/grid/log/rac1/agent/ohasd/oraagent_grid/oraagent_grid.log".
2013-07-18 00:11:03.614
[root@rac2 ~]# crsctl start crs
[root@rac1 ~]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.CRSDATA.dg ora....up.type ONLINE OFFLINE
ora.DATA.dg ora....up.type ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac1
ora.asm ora.asm.type ONLINE ONLINE rac1
ora.chris.db ora....se.type ONLINE ONLINE rac1
ora.cvu ora.cvu.type ONLINE ONLINE rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type ONLINE ONLINE rac1
ora.ons ora.ons.type ONLINE ONLINE rac1
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type ONLINE ONLINE rac1
免責聲明：本站發(fā)布的內容（圖片、視頻和文字）以原創(chuàng)、轉載和分享為主，文章觀點不代表本網(wǎng)站立場，如果涉及侵權請聯(lián)系站長郵箱：is@yisu.com進行舉報，并提供相關證據(jù)，一經(jīng)查實，將立刻刪除涉嫌侵權內容。
