Views expressed here are solely that of my own. Please make sure that you test the code/queries that appear on my blog before applying them to the production environment.

Thursday, March 22, 2012

How to replace Oracle 10g RAC OCR, Voting and ASM spfile disks

Below is the procedure I followed and the errors when replacing Oracle 10g RAC OCR, Voting and ASM spfile disks with the new disks from the new storage system. The database version is Oracle 10gR2 (10.2.0.4) RAC with two nodes and it is running on IBM servers with IBM AIX v6.1 OS.

I followed the procedure explained in the below Metalink Oracle Support (MOS) note.

OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

First thing you should do is analyze the current configuration and find what is the current disk sizes and the permissions of their defined devices.

You need to use root Unix user for the below operations.

Check ASM spfile device and hdisk, use the major and minor numbers of the device to find the corresponding hdisk which is "23, 3" below.
[root@srvdb01]:/home/root \> ls -l /dev/asmspf*
crw-rw----    1 oracle   dba          23,  3 Jul 28 2011  /dev/asmspf_disk
[root@srvdb01]:/home/root \> ls -l /dev/hdisk*|grep "23,  3"
brw-------    1 root     system       23,  3 Jul 28 2011  /dev/hdisk1

Check the corresponding hdisk size which is 100 MB below
[root@srvdb01]:/home/root \> bootinfo -s hdisk1
102


Check OCR disks, they are 300 MB each
[root@srvdb01]:/home/root \> ls -l /dev/ocr*
crw-r-----    1 root     dba          23,  4 Jul 28 2011  /dev/ocr_disk1
crw-r-----    1 root     dba          23,  5 Jul 28 2011  /dev/ocr_disk2
[root@srvdb01]:/home/root \> ls -l /dev/hdisk*|grep "23,  4"
brw-------    1 root     system       23,  4 Jul 28 2011  /dev/hdisk2
[root@srvdb01]:/home/root \> ls -l /dev/hdisk*|grep "23,  5"
brw-------    1 root     system       23,  5 Jul 28 2011  /dev/hdisk3
[root@srvdb01]:/home/root \> bootinfo -s hdisk2
307
[root@srvdb01]:/home/root \> bootinfo -s hdisk3
307


Check Voting disks, they are 300 MB each
[root@srvdb01]:/home/root \> ls -l /dev/vot*
crw-r--r--    1 oracle   dba          23,  6 Mar 20 10:55 /dev/voting_disk1
crw-r--r--    1 oracle   dba          23,  7 Mar 20 10:55 /dev/voting_disk2
crw-r--r--    1 oracle   dba          23,  8 Mar 20 10:55 /dev/voting_disk3
[root@srvdb01]:/home/root \> ls -l /dev/hdisk*|grep "23,  6"
brw-------    1 root     system       23,  6 Jul 28 2011  /dev/hdisk4
[root@srvdb01]:/home/root \> ls -l /dev/hdisk*|grep "23,  7"
brw-------    1 root     system       23,  7 Jul 28 2011  /dev/hdisk5
[root@srvdb01]:/home/root \> ls -l /dev/hdisk*|grep "23,  8"
brw-------    1 root     system       23,  8 Jul 28 2011  /dev/hdisk6
[root@srvdb01]:/home/root \> bootinfo -s hdisk4
307
[root@srvdb01]:/home/root \> bootinfo -s hdisk5
307
[root@srvdb01]:/home/root \> bootinfo -s hdisk6
307
[root@srvdb01]:/home/root \> 


Do the same check on the second node as root user
[root@srvdb02]:/home/root \> ls -l /dev/asmspf*
crw-rw----    1 oracle   dba          23,  3 Jun 29 2011  /dev/asmspf_disk
[root@srvdb02]:/home/root \> ls -l /dev/hdisk*|grep "23,  3"
brw-------    1 root     system       23,  3 Jun 30 2011  /dev/hdisk1
[root@srvdb02]:/home/root \> bootinfo -s hdisk1
102 

[root@srvdb02]:/home/root \> ls -l /dev/ocr*
crw-r-----    1 root     dba          23,  6 Mar 20 11:00 /dev/ocr_disk1
crw-r-----    1 root     dba          23,  7 Mar 20 11:00 /dev/ocr_disk2
[root@srvdb02]:/home/root \> ls -l /dev/hdisk*|grep "23,  6"
brw-------    1 root     system       23,  6 Jul 04 2011  /dev/hdisk2
[root@srvdb02]:/home/root \> ls -l /dev/hdisk*|grep "23,  7"
brw-------    1 root     system       23,  7 Jul 04 2011  /dev/hdisk3
[root@srvdb02]:/home/root \> bootinfo -s hdisk2
307
[root@srvdb02]:/home/root \> bootinfo -s hdisk3
307

[root@srvdb02]:/home/root \> ls -l /dev/vot*
crw-r--r--    1 oracle   dba          23,  8 Mar 20 11:01 /dev/voting_disk1
crw-r--r--    1 oracle   dba          23,  9 Mar 20 11:01 /dev/voting_disk2
crw-r--r--    1 oracle   dba          23, 10 Mar 20 11:01 /dev/voting_disk3
[root@srvdb02]:/home/root \> ls -l /dev/hdisk*|grep "23,  8"
brw-------    1 root     system       23,  8 Jul 04 2011  /dev/hdisk4
[root@srvdb02]:/home/root \> ls -l /dev/hdisk*|grep "23,  9"
brw-------    1 root     system       23,  9 Jul 04 2011  /dev/hdisk5
[root@srvdb02]:/home/root \> ls -l /dev/hdisk*|grep "23, 10"
brw-------    1 root     system       23, 10 Jul 04 2011  /dev/hdisk6
[root@srvdb02]:/home/root \> bootinfo -s hdisk4
307
[root@srvdb02]:/home/root \> bootinfo -s hdisk5
307
[root@srvdb02]:/home/root \> bootinfo -s hdisk6
307
[root@srvdb02]:/home/root \> 

New disks to be assigned as OCR, Voting and ASM spfile are below
brw------- 1 root system 21, 6 Mar 20 10:34 hdisk43 spfile
brw------- 1 root system 21, 2 Mar 20 10:34 hdisk44 ocr1
brw------- 1 root system 21, 1 Mar 20 10:34 hdisk45 ocr2
brw------- 1 root system 21, 3 Mar 20 10:34 hdisk46 vote1
brw------- 1 root system 21, 5 Mar 20 10:34 hdisk47 vote2
brw------- 1 root system 21, 4 Mar 20 10:34 hdisk48 vote3


Now start to configure the new hdisks

First check the sizes of the new disks by running the following command on each RAC node, as you can see from the below output we assigned 1 GB disks for each component.

for i in 43 44 45 46 47 48
do
bootinfo -s hdisk$i
done

[root@srvdb01]:/home/root \> for i in 43 44 45 46 47 48  
> do
> bootinfo -s hdisk$i
> done 
1024
1024
1024
1024
1024
1024
[root@srvdb01]:/home/root \> 

[root@srvdb02]:/home/root \> for i in 43 44 45 46 47 48  
> do
> bootinfo -s hdisk$i
> done
1024
1024
1024
1024
1024
1024
[root@srvdb02]:/home/root \> 

Now use the following command to set the "reserve_policy" of the new disks to the value "no_reserve", otherwise these disks can not be used in a RAC configuration. Also use the following second command to make sure that this operation is successful.
for i in 43 44 45 46 47 48
do
chdev -l hdisk$i -a reserve_policy=no_reserve
done

for i in 43 44 45 46 47 48
do
lsattr -El hdisk$i | grep reserve
done

[root@srvdb01]:/home/root \> for i in 43 44 45 46 47 48 
> do
> chdev -l hdisk$i -a reserve_policy=no_reserve
> done
hdisk43 changed
hdisk44 changed
hdisk45 changed
hdisk46 changed
hdisk47 changed
hdisk48 changed


As we can see they are set accordingly.
[root@srvdb01]:/home/root \> for i in 43 44 45 46 47 48  
> do
> lsattr -El hdisk$i | grep reserve
> done
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
[root@srvdb01]:/home/root \> 

Do the same thing in the second node
[root@srvdb02]:/home/root \> for i in 43 44 45 46 47 48 
> do
> chdev -l hdisk$i -a reserve_policy=no_reserve
> done
hdisk43 changed
hdisk44 changed
hdisk45 changed
hdisk46 changed
hdisk47 changed
hdisk48 changed
[root@srvdb02]:/home/root \> for i in 43 44 45 46 47 48  
> do
> lsattr -El hdisk$i | grep reserve
> done
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
reserve_policy  no_reserve                                          Reserve Policy                   True
[root@srvdb02]:/home/root \> 

To find the hdisk major and minor numbers, we use the following command on each node, since the major and minor numbers of the new assigned hdisk can be different on different RAC nodes.

for i in 43 44 45 46 47 48
do
ls -la /dev/hdisk$i
done

[root@srvdb01]:/home/root \> for i in 43 44 45 46 47 48 
> do
> ls -la /dev/hdisk$i
> done

brw-------    1 root     system       21,  1 Mar 20 10:34 /dev/hdisk43
brw-------    1 root     system       21,  2 Mar 20 10:34 /dev/hdisk44
brw-------    1 root     system       21,  4 Mar 20 10:34 /dev/hdisk45
brw-------    1 root     system       21,  5 Mar 20 10:34 /dev/hdisk46
brw-------    1 root     system       21,  3 Mar 20 10:34 /dev/hdisk47
brw-------    1 root     system       21,  6 Mar 20 10:34 /dev/hdisk48
[root@srvdb01]:/home/root \> 

[root@srvdb02]:/home/root \> for i in 43 44 45 46 47 48 
> do
> ls -la /dev/hdisk$i
> done

brw-------    1 root     system       21,  6 Mar 20 10:34 /dev/hdisk43
brw-------    1 root     system       21,  2 Mar 20 10:34 /dev/hdisk44
brw-------    1 root     system       21,  1 Mar 20 10:34 /dev/hdisk45
brw-------    1 root     system       21,  3 Mar 20 10:34 /dev/hdisk46
brw-------    1 root     system       21,  5 Mar 20 10:34 /dev/hdisk47
brw-------    1 root     system       21,  4 Mar 20 10:34 /dev/hdisk48
[root@srvdb02]:/home/root \> 

Start to create the virtual devices to be used that corresponds to the actual hdisk as below. As you can see above the major and minor numbers of the same hdisks can be different on each RAC node.

We prepare the following commands for each node

For the first node
mknod /dev/asmspf_disk_01 c 21 1

mknod /dev/ocr_disk_01 c 21 2
mknod /dev/ocr_disk_02 c 21 4

mknod /dev/voting_disk_01 c 21 5
mknod /dev/voting_disk_02 c 21 3
mknod /dev/voting_disk_03 c 21 6


For the second node
mknod /dev/asmspf_disk_01 c 21 6

mknod /dev/ocr_disk_01 c 21 2
mknod /dev/ocr_disk_02 c 21 1

mknod /dev/voting_disk_01 c 21 3
mknod /dev/voting_disk_02 c 21 5
mknod /dev/voting_disk_03 c 21 4

Run those commands on each corresponding RAC node
First node
[root@srvdb01]:/home/root \> mknod /dev/asmspf_disk_01 c 21 1
[root@srvdb01]:/home/root \> mknod /dev/ocr_disk_01 c 21 2
[root@srvdb01]:/home/root \> mknod /dev/ocr_disk_02 c 21 4
[root@srvdb01]:/home/root \> mknod /dev/voting_disk_01 c 21 5
[root@srvdb01]:/home/root \> mknod /dev/voting_disk_02 c 21 3
[root@srvdb01]:/home/root \> mknod /dev/voting_disk_03 c 21 6


Second node
[root@srvdb02]:/home/root \> mknod /dev/asmspf_disk_01 c 21 6
[root@srvdb02]:/home/root \> mknod /dev/ocr_disk_01 c 21 2
[root@srvdb02]:/home/root \> mknod /dev/ocr_disk_02 c 21 1
[root@srvdb02]:/home/root \> mknod /dev/voting_disk_01 c 21 3
[root@srvdb02]:/home/root \> mknod /dev/voting_disk_02 c 21 5
[root@srvdb02]:/home/root \> mknod /dev/voting_disk_03 c 21 4


Now check the new situation, the devices having major number 21 are the new disks configured recently

First node
[root@srvdb01]:/home/root \> ls -l /dev/asmspf*
crw-rw----    1 oracle   dba          23,  3 Jul 28 2011  /dev/asmspf_disk
crw-------    1 root     system       21,  1 Mar 20 11:26 /dev/asmspf_disk_01
[root@srvdb01]:/home/root \> ls -l /dev/ocr*
crw-r-----    1 root     dba          23,  4 Jul 28 2011  /dev/ocr_disk1
crw-r-----    1 root     dba          23,  5 Jul 28 2011  /dev/ocr_disk2
crw-------    1 root     system       21,  2 Mar 20 11:26 /dev/ocr_disk_01
crw-------    1 root     system       21,  4 Mar 20 11:26 /dev/ocr_disk_02
[root@srvdb01]:/home/root \> ls -l /dev/vot*
crw-r--r--    1 oracle   dba          23,  6 Mar 20 11:27 /dev/voting_disk1
crw-r--r--    1 oracle   dba          23,  7 Mar 20 11:27 /dev/voting_disk2
crw-r--r--    1 oracle   dba          23,  8 Mar 20 11:27 /dev/voting_disk3
crw-------    1 root     system       21,  5 Mar 20 11:26 /dev/voting_disk_01
crw-------    1 root     system       21,  3 Mar 20 11:26 /dev/voting_disk_02
crw-------    1 root     system       21,  6 Mar 20 11:26 /dev/voting_disk_03
[root@srvdb01]:/home/root \>

Second node
[root@srvdb02]:/home/root \> ls -l /dev/asmspf*
crw-rw----    1 oracle   dba          23,  3 Jun 29 2011  /dev/asmspf_disk
crw-------    1 root     system       21,  6 Mar 20 11:26 /dev/asmspf_disk_01
[root@srvdb02]:/home/root \> ls -l /dev/ocr*
crw-r-----    1 root     dba          23,  6 Mar 20 11:27 /dev/ocr_disk1
crw-r-----    1 root     dba          23,  7 Mar 20 11:27 /dev/ocr_disk2
crw-------    1 root     system       21,  2 Mar 20 11:26 /dev/ocr_disk_01
crw-------    1 root     system       21,  1 Mar 20 11:26 /dev/ocr_disk_02
[root@srvdb02]:/home/root \> ls -l /dev/vot*
crw-r--r--    1 oracle   dba          23,  8 Mar 20 11:27 /dev/voting_disk1
crw-r--r--    1 oracle   dba          23,  9 Mar 20 11:27 /dev/voting_disk2
crw-r--r--    1 oracle   dba          23, 10 Mar 20 11:27 /dev/voting_disk3
crw-------    1 root     system       21,  3 Mar 20 11:26 /dev/voting_disk_01
crw-------    1 root     system       21,  5 Mar 20 11:26 /dev/voting_disk_02
crw-------    1 root     system       21,  4 Mar 20 11:26 /dev/voting_disk_03
[root@srvdb02]:/home/root \> 

Pay attention in this step : File ownership and device permissions should be the same as the old disks !!!!!!
Use the following commands on each RAC node

for i in 01 02
do
chown root:dba /dev/ocr_disk_$i
chmod 640 /dev/ocr_disk_$i
done

for i in 01 02 03
do
chown oracle:dba /dev/voting_disk_$i
chmod 644 /dev/voting_disk_$i
done

chown oracle:dba /dev/asmspf_disk_01
chmod 660 /dev/asmspf_disk_01

First node
[root@srvdb01]:/home/root \> for i in 01 02 
> do
> chown root:dba /dev/ocr_disk_$i
> chmod 640 /dev/ocr_disk_$i
> done
[root@srvdb01]:/home/root \> for i in 01 02 03  
> do
> chown oracle:dba /dev/voting_disk_$i
> chmod 644 /dev/voting_disk_$i
> done
[root@srvdb01]:/home/root \> chown oracle:dba /dev/asmspf_disk_01
[root@srvdb01]:/home/root \> chmod 660 /dev/asmspf_disk_01

Second node
[root@srvdb02]:/home/root \> for i in 01 02 
> do
> chown root:dba /dev/ocr_disk_$i
> chmod 640 /dev/ocr_disk_$i
> done
[root@srvdb02]:/home/root \> for i in 01 02 03  
> do
> chown oracle:dba /dev/voting_disk_$i
> chmod 644 /dev/voting_disk_$i
> done
[root@srvdb02]:/home/root \> chown oracle:dba /dev/asmspf_disk_01
[root@srvdb02]:/home/root \> chmod 660 /dev/asmspf_disk_01


Now check the file ownerships and the permissions of the new disks, they are the ones with major number 21, and the ownership and permissions should be the same as the old disks having the major number 23 below.

First node
[root@srvdb01]:/home/root \> ls -l /dev/asmspf*
crw-rw----    1 oracle   dba          23,  3 Jul 28 2011  /dev/asmspf_disk
crw-rw----    1 oracle   dba          21,  1 Mar 20 11:26 /dev/asmspf_disk_01
[root@srvdb01]:/home/root \> ls -l /dev/ocr*
crw-r-----    1 root     dba          23,  4 Jul 28 2011  /dev/ocr_disk1
crw-r-----    1 root     dba          23,  5 Jul 28 2011  /dev/ocr_disk2
crw-r-----    1 root     dba          21,  2 Mar 20 11:26 /dev/ocr_disk_01
crw-r-----    1 root     dba          21,  4 Mar 20 11:26 /dev/ocr_disk_02
[root@srvdb01]:/home/root \> ls -l /dev/vot*
crw-r--r--    1 oracle   dba          23,  6 Mar 20 11:37 /dev/voting_disk1
crw-r--r--    1 oracle   dba          23,  7 Mar 20 11:37 /dev/voting_disk2
crw-r--r--    1 oracle   dba          23,  8 Mar 20 11:37 /dev/voting_disk3
crw-r--r--    1 oracle   dba          21,  5 Mar 20 11:26 /dev/voting_disk_01
crw-r--r--    1 oracle   dba          21,  3 Mar 20 11:26 /dev/voting_disk_02
crw-r--r--    1 oracle   dba          21,  6 Mar 20 11:26 /dev/voting_disk_03
[root@srvdb01]:/home/root \> 

Second node
[root@srvdb02]:/home/root \> ls -l /dev/asmspf*
crw-rw----    1 oracle   dba          23,  3 Jun 29 2011  /dev/asmspf_disk
crw-rw----    1 oracle   dba          21,  6 Mar 20 11:26 /dev/asmspf_disk_01
[root@srvdb02]:/home/root \> ls -l /dev/ocr*
crw-r-----    1 root     dba          23,  6 Mar 20 11:37 /dev/ocr_disk1
crw-r-----    1 root     dba          23,  7 Mar 20 11:37 /dev/ocr_disk2
crw-r-----    1 root     dba          21,  2 Mar 20 11:26 /dev/ocr_disk_01
crw-r-----    1 root     dba          21,  1 Mar 20 11:26 /dev/ocr_disk_02
[root@srvdb02]:/home/root \> ls -l /dev/vot*
crw-r--r--    1 oracle   dba          23,  8 Mar 20 11:38 /dev/voting_disk1
crw-r--r--    1 oracle   dba          23,  9 Mar 20 11:38 /dev/voting_disk2
crw-r--r--    1 oracle   dba          23, 10 Mar 20 11:38 /dev/voting_disk3
crw-r--r--    1 oracle   dba          21,  3 Mar 20 11:26 /dev/voting_disk_01
crw-r--r--    1 oracle   dba          21,  5 Mar 20 11:26 /dev/voting_disk_02
crw-r--r--    1 oracle   dba          21,  4 Mar 20 11:26 /dev/voting_disk_03
[root@srvdb02]:/home/root \> 

Now you need to use the following command to set the "pv" parameter of each new hdisk to the value "clear" to sign them as new untouched devices in the system.

for i in 43 44 45 46 47 48
do
chdev -l hdisk$i -a pv=clear
done


First node
[root@srvdb01]:/home/root \> for i in 43 44 45 46 47 48
> do
> chdev -l hdisk$i -a pv=clear
> done

hdisk43 changed
hdisk44 changed
hdisk45 changed
hdisk46 changed
hdisk47 changed
hdisk48 changed
[root@srvdb01]:/home/root \> 

Second node
[root@srvdb02]:/home/root \> for i in 43 44 45 46 47 48
> do
> chdev -l hdisk$i -a pv=clear
> done

hdisk43 changed
hdisk44 changed
hdisk45 changed
hdisk46 changed
hdisk47 changed
hdisk48 changed
[root@srvdb02]:/home/root \> 

In this step you need to format the new virtual devices by running the following commands on each RAC node. You need to wait until the "Done" message displayed for each command that indicates that formatting is finished. Run those commands first in the first node wait until they complete ("Done" message) and then run them in the second node and wait for them to complete, this is ver important.

for i in 01 02
do
dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000 &
done

for i in 01 02 03
do
dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000 &
done

dd if=/dev/zero of=/dev/asmspf_disk_01 bs=8192 count=25000 &

First node
[root@srvdb01]:/home/root \> for i in 01 02 
> 
> do
> dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000 &
> done
[1] 852170
[2] 848092
[root@srvdb01]:/home/root \> for i in 01 02 03 
> 
> do
> dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000 &
> done
[3] 1274024
[4] 532694
[5] 1503258
[root@srvdb01]:/home/root \> 25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.
dd if=/dev/zero of=/dev/asmspf_disk_01 bs=8192 count=25000 &
[6] 852178
[1]   Done                    dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000
[2]   Done                    dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000
[root@srvdb01]:/home/root \> 
[root@srvdb01]:/home/root \> 25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.

[3]   Done                    dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000
[4]   Done                    dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000
[5]-  Done                    dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000
[6]+  Done                    dd if=/dev/zero of=/dev/asmspf_disk_01 bs=8192 count=25000
[root@srvdb01]:/home/root \> 

Second node
[root@srvdb02]:/home/root \> for i in 01 02 
> 
> do
> dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000 &
> done
[1] 1040494
[2] 1507446
[root@srvdb02]:/home/root \> 
[root@srvdb02]:/home/root \> for i in 01 02 03 
> 
> do
> dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000 &
> done
[3] 823530
[4] 975062
[5] 520304
[root@srvdb02]:/home/root \> 25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.
dd if=/dev/zero of=/dev/asmspf_disk_01 bs=8192 count=25000 &
[6] 581850
[1]   Done                    dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000
[2]   Done                    dd if=/dev/zero of=/dev/ocr_disk_$i bs=8192 count=25000
[root@srvdb02]:/home/root \> 
[root@srvdb02]:/home/root \> 
[root@srvdb02]:/home/root \> 25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.
25000+0 records in.
25000+0 records out.

[3]   Done                    dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000
[4]   Done                    dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000
[5]-  Done                    dd if=/dev/zero of=/dev/voting_disk_$i bs=8192 count=25000
[root@srvdb02]:/home/root \> 25000+0 records in.
25000+0 records out.

[6]+  Done                    dd if=/dev/zero of=/dev/asmspf_disk_01 bs=8192 count=25000
[root@srvdb02]:/home/root \> 

Now that the configuration of the new disks is completed.

---------------------------
1-) REPLACE OCR DISKS

We can now continue with the replacing the old OCR, Voting and ASM spfile disks with the new ones.

According to the MOS note id 428681.1, in Oracle 10g RAC, OCR disks can be replaced while the CRS is open, so we tried to do it without closing the CRS.

First we need to check if there is any OCR backup, normally there is a scheduled OCR backup, backing up the OCR disk automatically, so most probably you will see an output like below.

Check the OCR backup, there are some backups taken recently, so there is no problem if something goes wrong we can use the most recent backup before the operation to recover the OCR.
[root@srvdb01]:/home/root \> ocrconfig -showbackup

srvdb02     2012/03/20 08:57:32     /oracle/crshome1/cdata/crs

srvdb02     2012/03/20 04:57:32     /oracle/crshome1/cdata/crs

srvdb02     2012/03/20 00:57:31     /oracle/crshome1/cdata/crs

srvdb02     2012/03/18 20:57:30     /oracle/crshome1/cdata/crs

srvdb02     2012/03/07 16:57:14     /oracle/crshome1/cdata/crs
[root@srvdb02]:/home/root \> 

Check the current OCR info, we have two OCR disks configured already, first one is primary and the second one is just the OCR mirror.
[root@srvdb01]:/home/root \> ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     314344
         Used space (kbytes)      :       3876
         Available space (kbytes) :     310468
         ID                       : 1673980957
         Device/File Name         : /dev/ocr_disk1
                                    Device/File integrity check succeeded
         Device/File Name         : /dev/ocr_disk2
                                    Device/File integrity check succeeded

         Cluster registry integrity check succeeded

[root@srvdb01]:/home/root \> 

Now we try to replace the primary OCR disk, but we got "PROT-16" error, this can be because of bug I do not know the reason but I was unable to do it.
Run these commands only on one node.
[root@srvdb01]:/home/root \> ocrconfig -replace ocr /dev/ocr_disk_01
PROT-16: Internal Error


Then I tried to replace the ocrmirror but it gave another error
[root@srvdb01]:/home/root \> ocrconfig -replace ocrmirror /dev/ocr_disk_01
PROT-21: Invalid parameter


When I checked the log file it gives the following error.
[root@srvdb01]:/oracle/crshome1/log/srvdb01/client \> vi ocrconfig_2027594.log
"ocrconfig_2027594.log" 5 lines, 442 characters 
Oracle Database 10g CRS Release 10.2.0.4.0 Production Copyright 1996, 2008 Oracle.  All rights reserved.
2012-03-20 12:10:20.964: [ OCRCONF][1]ocrconfig starts...
2012-03-20 12:10:21.019: [  OCRCLI][1]proac_replace_dev:[/dev/ocr_disk_01]: Failed. Retval [8]
2012-03-20 12:10:21.019: [ OCRCONF][1]The input OCR device either is identical to the other device or cannot be opened
2012-03-20 12:10:21.019: [ OCRCONF][1]Exiting [status=failed]...


Not to loose any time I left this error aside to search for a solution later and I decided to go on with the "to replace the OCR disk offline" procedure, since at least I needed to close the CRS stack in Oracle 10g RAC for replacing the voting this it is not so much important to replace the OCR disks while the CRS is open or not.

Just continue with the Offline replacement procedure.
First we need to stop the CRS on all nodes for this operation.

First node
[root@srvdb01]:/home/root \> crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.


Check the CRS if it's closed successfuly
[root@srvdb01]:/home/root \> crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM 


Second node
[root@srvdb02]:/home/root \> crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.

[root@srvdb02]:/home/root \> crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM 


Run the following commands on all the RAC nodes
[root@srvdb01]:/home/root \> ocrconfig -repair ocr /dev/ocr_disk_01
[root@srvdb01]:/home/root \> ocrconfig -repair ocrmirror /dev/ocr_disk_02

[root@srvdb02]:/home/root \> ocrconfig -repair ocr /dev/ocr_disk_01
[root@srvdb02]:/home/root \> ocrconfig -repair ocrmirror /dev/ocr_disk_02

Run this command only on one node
[root@srvdb01]:/home/root \> ocrconfig -overwrite


Check the OCR for the latest state, as we can see the nold disks are replaced by the new ones
[root@srvdb01]:/home/root \> ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     314344
         Used space (kbytes)      :       3884
         Available space (kbytes) :     310460
         ID                       : 1673980957
         Device/File Name         : /dev/ocr_disk_01
                                    Device/File integrity check succeeded
         Device/File Name         : /dev/ocr_disk_02
                                    Device/File needs to be synchronized with the other device 

         Cluster registry integrity check succeeded

[root@srvdb01]:/home/root \> 

[root@srvdb02]:/home/root \> ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :     314344
         Used space (kbytes)      :       3884
         Available space (kbytes) :     310460
         ID                       : 1673980957
         Device/File Name         : /dev/ocr_disk_01
                                    Device/File integrity check succeeded
         Device/File Name         : /dev/ocr_disk_02
                                    Device/File needs to be synchronized with the other device

         Cluster registry integrity check succeeded

[root@srvdb02]:/home/root \>

Now we can remove the virtual devices of old OCR disks on each node.
[root@srvdb01]:/home/root \> rm /dev/ocr_disk1
[root@srvdb01]:/home/root \> rm /dev/ocr_disk2

[root@srvdb02]:/home/root \> rm /dev/ocr_disk1
[root@srvdb02]:/home/root \> rm /dev/ocr_disk2

---------------------------
2-) REPLACE VOTING DISKS

Now we can continue replacing the old voting disks with the new ones.

Check current voting disk info
[root@srvdb01]:/home/root \> crsctl query css votedisk
 0.     0    /dev/voting_disk1
 1.     0    /dev/voting_disk2
 2.     0    /dev/voting_disk3


located 3 votedisk(s).


Now add the new voting disks, do this operation only on first node
[root@srvdb01]:/home/root \> crsctl add css votedisk /dev/voting_disk_01 -force
Now formatting voting disk: /dev/voting_disk_01
successful addition of votedisk /dev/voting_disk_01.
[root@srvdb01]:/home/root \> crsctl add css votedisk /dev/voting_disk_02 -force
Now formatting voting disk: /dev/voting_disk_02
successful addition of votedisk /dev/voting_disk_02.
[root@srvdb01]:/home/root \> crsctl add css votedisk /dev/voting_disk_03 -force
Now formatting voting disk: /dev/voting_disk_03
successful addition of votedisk /dev/voting_disk_03.


Check the current voting disk info
[root@srvdb01]:/home/root \> crsctl query css votedisk
 0.     0    /dev/voting_disk1
 1.     0    /dev/voting_disk2
 2.     0    /dev/voting_disk3
 3.     0    /dev/voting_disk_01
 4.     0    /dev/voting_disk_02
 5.     0    /dev/voting_disk_03

located 6 votedisk(s).


Delete the old voting disks.
[root@srvdb01]:/home/root \> crsctl delete css votedisk /dev/voting_disk1 -force 
successful deletion of votedisk /dev/voting_disk1.
[root@srvdb01]:/home/root \> crsctl delete css votedisk /dev/voting_disk2 -force 
successful deletion of votedisk /dev/voting_disk2.
[root@srvdb01]:/home/root \> crsctl delete css votedisk /dev/voting_disk3 -force
successful deletion of votedisk /dev/voting_disk3.


Check the current voting disk info
[root@srvdb01]:/home/root \> crsctl query css votedisk
 0.     0    /dev/voting_disk_01
 1.     0    /dev/voting_disk_02
 2.     0    /dev/voting_disk_03

located 3 votedisk(s).

Now we successfully replaced the old voting disks with the new ones.

Now we can remove the virtual devices of old OCR disks on each node.
[root@srvdb01]:/home/root \> rm /dev/voting_disk1
[root@srvdb01]:/home/root \> rm /dev/voting_disk2
[root@srvdb01]:/home/root \> rm /dev/voting_disk3

[root@srvdb02]:/home/root \> rm /dev/voting_disk1
[root@srvdb02]:/home/root \> rm /dev/voting_disk2
[root@srvdb02]:/home/root \> rm /dev/voting_disk3


---------------------------
3-) REPLACE ASM SPFILE DISK

We can continue with replacing the ASM spfile disk now.

Following operations should be performed on each RAC node.

Switch to the the ASM profile.

[oracle@srvdb01]:/oracle > . .profile_asm
[oracle@srvdb01]:/oracle > env|grep ORA
ORACLE_BASE=/oracle
ORACLE_SID=+ASM1
ORA_ASM_HOME=/oracle/asmhome1
ORA_CRS_HOME=/oracle/crshome1
ORACLE_HOME=/oracle/asmhome1

Go to init.ora file destination
[oracle@srvdb01]:/oracle > cd $ORACLE_HOME/dbs

Check this file for the current spfile setting
[oracle@srvdb01]:/oracle/asmhome1/dbs > more init+ASM1.ora
SPFILE='/dev/asmspf_disk'

Also before you close the CRS the ASM database should show the below output.
SYS@+ASM1 AS SYSDBA> sho parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /dev/asmspf_disk


We need to backup this file, create a new pfile form current spfile, create a new spfile from this pfile in the new ASM spfile disk, and arrange the new init.ora file to point to the new ASm spfile location.

Backup the old pfile
[oracle@srvdb01]:/oracle/asmhome1/dbs > cp init+ASM1.ora init+ASM1.ora-201203201041


In sqlplus, create a new pfile from spfile
SYS@+ASM1 AS SYSDBA> create pfile from spfile;

File created.

Check the new pfile
[oracle@srvdb01]:/oracle/asmhome1/dbs > more init+ASM1.ora
+ASM1.asm_diskgroups='DG_DB_ASM','DG_FRA_ASM'#Manual Mount
*.asm_diskgroups='DG_DB_ASM','DG_FRA_ASM'
*.asm_diskstring='/dev/ASM*'
*.background_dump_dest='/oracle/admin/+ASM/bdump'
*.cluster_database=true
*.core_dump_dest='/oracle/admin/+ASM/cdump'
+ASM2.instance_number=2
+ASM1.instance_number=1
*.instance_type='asm'
*.large_pool_size=12M
*.processes=150
*.remote_login_passwordfile='exclusive'
*.user_dump_dest='/oracle/admin/+ASM/udump'


In sqlplus, create the new spfile in the new ASM spfile disk location
SYS@+ASM1 AS SYSDBA> create spfile='/dev/asmspf_disk_01' from pfile;

File created.

Change the new init.ora file to point to the new ASm spfile location
[oracle@srvdb01]:/oracle/asmhome1/dbs > more init+ASM1.ora
SPFILE='/dev/asmspf_disk_01'

Do all the above operations in the second node also

On second node
Switch to the the ASM profile.

[oracle@srvdb02]:/oracle > . .profile_asm
[oracle@srvdb02]:/oracle > env|grep ORA
ORACLE_BASE=/oracle
ORACLE_SID=+ASM2
ORA_ASM_HOME=/oracle/asmhome1
ORA_CRS_HOME=/oracle/crshome1
ORACLE_HOME=/oracle/asmhome1

Go to init.ora file destination
[oracle@srvdb02]:/oracle > cd $ORACLE_HOME/dbs

Check this file for the current spfile setting
[oracle@srvdb02]:/oracle/asmhome1/dbs > more init+ASM2.ora
SPFILE='/dev/asmspf_disk'

Also before you close the CRS the ASM database should show the below output.
SYS@+ASM1 AS SYSDBA> sho parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /dev/asmspf_disk


We need to backup this file, create a new pfile form current spfile, create a new spfile from this pfile in the new ASM spfile disk, and arrange the new init.ora file to point to the new ASm spfile location.

Backup the old pfile
[oracle@srvdb02]:/oracle/asmhome1/dbs > cp init+ASM2.ora init+ASM2.ora-201203201041


In sqlplus, create a new pfile from spfile
SYS@+ASM1 AS SYSDBA> create pfile from spfile;

File created.

Check the new pfile
[oracle@srvdb02]:/oracle/asmhome1/dbs > more init+ASM2.ora
+ASM1.asm_diskgroups='DG_DB_ASM','DG_FRA_ASM'#Manual Mount
*.asm_diskgroups='DG_DB_ASM','DG_FRA_ASM'
*.asm_diskstring='/dev/ASM*'
*.background_dump_dest='/oracle/admin/+ASM/bdump'
*.cluster_database=true
*.core_dump_dest='/oracle/admin/+ASM/cdump'
+ASM2.instance_number=2
+ASM1.instance_number=1
*.instance_type='asm'
*.large_pool_size=12M
*.processes=150
*.remote_login_passwordfile='exclusive'
*.user_dump_dest='/oracle/admin/+ASM/udump'


In sqlplus, create the new spfile in the new ASM spfile disk location
SYS@+ASM1 AS SYSDBA> create spfile='/dev/asmspf_disk_01' from pfile;

File created.

Change the new init.ora file to point to the new ASm spfile location
[oracle@srvdb02]:/oracle/asmhome1/dbs > more init+ASM2.ora
SPFILE='/dev/asmspf_disk_01'

---------------------------

Now that all the configurations are finished and we can try to open the CRS and see the result.

[root@srvdb01]:/home/root \> crsctl start crs
Attempting to start CRS stack 
The CRS stack will be started shortly
[root@srvdb01]:/home/root \> 

[root@srvdb02]:/home/root \> crsctl start crs
Attempting to start CRS stack 
The CRS stack will be started shortly
[root@srvdb02]:/home/root \> 

[root@srvdb01]:/home/root \> crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[root@srvdb02]:/home/root \> crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

[root@srvdb01]:/home/root \> crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    srvdb01   
ora....T2.inst application    ONLINE    ONLINE    srvdb02   
ora.CORET.db   application    ONLINE    ONLINE    srvdb02   
ora....SM1.asm application    ONLINE    ONLINE    srvdb01   
ora....01.lsnr application    ONLINE    ONLINE    srvdb01   
ora....b01.gsd application    ONLINE    ONLINE    srvdb01   
ora....b01.ons application    ONLINE    ONLINE    srvdb01   
ora....b01.vip application    ONLINE    ONLINE    srvdb01   
ora....SM2.asm application    ONLINE    ONLINE    srvdb02   
ora....02.lsnr application    ONLINE    ONLINE    srvdb02   
ora....b02.gsd application    ONLINE    ONLINE    srvdb02   
ora....b02.ons application    ONLINE    ONLINE    srvdb02   
ora....b02.vip application    ONLINE    ONLINE    srvdb02   
[root@srvdb01]:/home/root \> 

[root@srvdb02]:/home/root \> crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora....T1.inst application    ONLINE    ONLINE    srvdb01   
ora....T2.inst application    ONLINE    ONLINE    srvdb02   
ora.CORET.db   application    ONLINE    ONLINE    srvdb02   
ora....SM1.asm application    ONLINE    ONLINE    srvdb01   
ora....01.lsnr application    ONLINE    ONLINE    srvdb01   
ora....b01.gsd application    ONLINE    ONLINE    srvdb01   
ora....b01.ons application    ONLINE    ONLINE    srvdb01   
ora....b01.vip application    ONLINE    ONLINE    srvdb01   
ora....SM2.asm application    ONLINE    ONLINE    srvdb02   
ora....02.lsnr application    ONLINE    ONLINE    srvdb02   
ora....b02.gsd application    ONLINE    ONLINE    srvdb02   
ora....b02.ons application    ONLINE    ONLINE    srvdb02   
ora....b02.vip application    ONLINE    ONLINE    srvdb02   
[root@srvdb02]:/home/root \> 

As a result everything looks good.
Lets check if the ASM spfile is also updated in the ASM instances.

First node
[oracle@srvdb01]:/oracle > . .profile_asm
[YOU HAVE NEW MAIL]
[oracle@srvdb01]:/oracle > env|grep ORA
ORACLE_BASE=/oracle
ORACLE_SID=+ASM1
ORA_ASM_HOME=/oracle/asmhome1
ORA_CRS_HOME=/oracle/crshome1
ORACLE_HOME=/oracle/asmhome1


In sqlplus
SYS@+ASM1 > sho parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /dev/asmspf_disk_01
SYS@+ASM1 > 

Second node
[oracle@srvdb02]:/oracle > . .profile_asm
[YOU HAVE NEW MAIL]
[oracle@srvdb02]:/oracle > env|grep ORA
ORACLE_BASE=/oracle
ORACLE_SID=+ASM2
ORA_ASM_HOME=/oracle/asmhome1
ORA_CRS_HOME=/oracle/crshome1
ORACLE_HOME=/oracle/asmhome1


In sqlplus
SYS@+ASM2 > sho parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      /dev/asmspf_disk_01
SYS@+ASM2 > 

ASM spfile replacement is also successfull, now we can rm the old ASm spfile virtual device on both nodes

[root@srvdb01]:/home/root \> rm /dev/asmspf_disk
[root@srvdb02]:/home/root \> rm /dev/asmspf_disk


Now this is the end of the operation which was successfull. I hope everything will go fine if you will need to replace OCR, Voting and ASM spfile disks in your Oracle RAC systems one day.

2 comments:

Path Infotech said...

Thanks for sharing the information

For more info : Oracle Certification Courses in Noida

Ural Ural said...

You're all welcome, learn more by sharing your knowledge.

Cheers,
Ural