oracle11.2.0.4 rac搭建中的crs-4000錯(cuò)誤解析

   在系統(tǒng)環(huán)境為rhel6.5的服務(wù)器上，搭建數(shù)據(jù)庫(kù)版本為oracle11.2.0.4的兩節(jié)點(diǎn)的RAC，安裝GRID時(shí)遭遇如下錯(cuò)誤：

（注：vote、data和閃回盤(pán)都在存儲(chǔ)設(shè)備上）

ASM created and started successfully.

Disk Group VOTE mounted successfully.

clscfg: -install mode specified

clscfg: EXISTING configuration version 5 detected.

clscfg: version 5 is 11g Release 2.

Successfully accumulated necessary OCR keys.

clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Failed to create voting files on disk group VOTE.

Change to configuration failed, but was successfully rolled back.

CRS-4000: Command Replace failed, or completed with errors.

Voting file add failed

Failed to add voting disks at /g01/11ggrid/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6930.

/g01/11ggrid/app/11.2.0/grid/perl/bin/perl -I/g01/11ggrid/app/11.2.0/grid/perl/lib -I/g01/11ggrid/app/11.2.0/grid/crs/install /g01/11ggrid/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

這個(gè)錯(cuò)誤是說(shuō)無(wú)法對(duì)磁盤(pán)組VOTE創(chuàng)建投票文件。改變配置失敗，但成功回滾。

投票文件添加失敗，未能在/g01/11ggrid/app/11.2.0/grid/crs/install添加表決磁盤(pán)。

碰到這個(gè)錯(cuò)誤還是頭一遭，很是詭異。因?yàn)橹笆褂肁SMLIB掛盤(pán)都沒(méi)有出現(xiàn)任何問(wèn)題，但安裝卻出現(xiàn)這個(gè)問(wèn)題，沒(méi)法繼續(xù)進(jìn)行下去。

檢查ASM的 alert.log日志發(fā)現(xiàn)：

ERROR: Could not create voting files. It spans across 161 AUs (max supported is 64 AUs)

ERROR: Voting file allocation failed for group VOTE

Errors in file /g01/11ggrid/app/11.2.0/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7321.trc:

ORA-15303: Voting files could not be created in diskgroup VOTE due to small Allocation Unit size

查看對(duì)應(yīng)GRID的仲裁盤(pán)對(duì)應(yīng)的DM

#multipath -ll

sc_vote (360a980003830302d712b46586276736b) dm-11  NETAPP,LUN

size=5.0G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=2 status=active

  `- 17:0:0:0 sdax 67:16  active ready  running

繼續(xù)檢查一下DM-11塊的大小問(wèn)題，發(fā)現(xiàn)一些信息：

# cd /sys/block/dm-11/queue

#  more *block_size

::::::::::::::

logical_block_size

::::::::::::::

512

::::::::::::::

physical_block_size

::::::::::::::

4096

究其原因是它表現(xiàn)出不同的邏輯和物理塊大小。對(duì)于大多數(shù)的情況下，物理和邏輯塊的大小是相同的“512”。

那么如何解決這個(gè)問(wèn)題呢？查詢(xún)ORACLE的官方網(wǎng)站，給出的解決方案是：

1. Oracle bug 11780656 is already fixed in 11.2.0.3 with compatible.asm = 11.2.0.3 but OUI does not allow to specify "compatible.asm" attribute to create OCRVOTE diskgroup.

   - Bug 11780656  - ASM MANAGED VOTING FILES CANNOT BE MORE THAN 64X AU SIZE

2. Oracle bug 13999609 indicates that ASMLib driver only works with the expectation that logical block size and physical block size are 512/512 bytes.   

   - Bug 13999609  - PHYSICAL BLOCK SIZE REPORTED CAN CAUSE ISSUES WITH 10G DATABASES

SOLUTION

1] Possible workaround is to use '/dev/oracleasm/disks/*' path instead of "ORCL:*" when creating OCRVOTE diskgroup in OUI.

OR

2] Install the new “oracleasm-support-2.1.8-1” ASMLIB RPM package (which contains the permanent fix) as described in note 1500460.1

根據(jù)以上提示，我們可以在安裝oracleasm-support-2.1.8-1的包，或者也可以在選擇ASM盤(pán)時(shí)，改用

/dev/oracleasm/disks/*的路徑來(lái)解決此問(wèn)題。根據(jù)以上提示進(jìn)行解決，發(fā)現(xiàn)還是無(wú)法成功裝上GRID.

反復(fù)幾次，沒(méi)有找到解決的辦法。由于后端使用的是NETAPP存儲(chǔ)，所以就去NETAPP的官網(wǎng)查找此問(wèn)題的CASE案例。發(fā)現(xiàn)其給出如下解決方案：

For details and caveats regarding this workaround, see the Oracle Alert.

Additionally, Oracle has provided a patch and configuration parameter to enable ASMlib to continue to function using the correct logical block size.

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'

(oracle 設(shè)置) 

NetApp has also provided a workaround in versions 8.0.5, 8.1.3 and 8.2 of Data ONTAP 7-Mode. The workaround allows specified LUNs to continue to not report the logical blocks per physical block value. This work around should only be applied to LUNs used by Oracle ASMlib with the symptoms described in this article.

From the Data ONTAP 7-Mode CLI, enter the following commands:

> lun set report-physical-size <path> disable（netapp存儲(chǔ)設(shè)置）

根據(jù)此提示，然后進(jìn)入/etc/sysconfig目錄，修改其下的oracleasm文件，將其中的

[oracle@rac1 ~]$ cat /etc/sysconfig/oracleasm

#

# This is a configuration file for automatic loading of the Oracle

# Automatic Storage Management library kernel driver.  It is generated

# By running /etc/init.d/oracleasm configure.  Please use that method

# to modify this file

#

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=asmadmin

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER=""

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE=""

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size

# reported by the underlying disk instead of the physical. The default

# is 'false'

ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

將ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false改為true

并對(duì)應(yīng)修改NETAPP上的存儲(chǔ)設(shè)置參數(shù)

（注：Oracle提供了配置參數(shù)來(lái)啟用ASMLib繼續(xù)使用正確的邏輯塊的大小的功能來(lái)避免這個(gè)問(wèn)題。）

通過(guò)如上的處理，該問(wèn)題得以解決，GRID最后安裝成功。