Oracle Vote Disk Fun

Losing Oracle RAC vote disks can cause catastrophic failure, so one would generally want to spread their disks across multiple storage devices to prevent this. In my career I have worked for both small and large companies. The biggest difference here is generally resources, and at a small company I ran across a setup where they had only two network storage arrays and thus only two separate locations for vote disk retention.

The fix for this is quite novel. When a voting file disk is lost, ASM will move the voting file to the next available disk in the group IF one is available, and it does this automagically, however if there is not an extra disk for asm to use and the cluster was brought down by an outage in the past as a result.

Thus the solution to this problem is as easy as adding a 4th disk, dropping the problematic disk, and the new voting file gets instantly created on the 4th disk without manual involvement. Of course when I explained this I was scoffed at and told that I just did not understand Oracle RAC, so I ended up walking them through it and these are my notes.

Moving forward, the suggestion is to add a 4th disk to all voting disk groups to add an extra level of redundancy. With an extra disk already added to the disk group in use, anytime a voting disk(s) becomes unavailable it will automatically be replaced by the asm instance and the result will be 0% down time.

WORK FLOW DURING TESTING:

crw-rw—-   1 oracle     dba        135 0x000021 Oct 28  2009 ora_asm_disk1_eva4

crw-rw—-   1 oracle     dba        135 0x000022 Oct 28  2009 ora_asm_disk2_eva4

crw-rw—-   1 oracle     dba        135 0x000023 Oct 28  2009 ora_asm_disk3_eva4

crw-rw—-   1 oracle     dba        135 0x00000b Sep 24 14:06 ora_asm_disk4_eva4

crw-rw—-   1 oracle     dba        135 0x000004 Oct 27 11:33 ora_data_disk01_sa21

crw-rw—-   1 oracle     dba        135 0x000007 Oct 27 14:33 ora_data_disk02_sa21

crw-rw—-   1 oracle     dba        135 0x000008 Oct 27 13:12 ora_data_disk03_sa21

crw-rw—-   1 oracle     dba        135 0x00000c Oct 27 14:10 ora_data_disk04_sa21

crw-rw—-   1 oracle     dba        135 0x000020 Oct 28  2009 ora_ocr_disk1_eva2

crw-rw—-   1 oracle     dba        135 0x000005 Sep 20 16:25 ora_vote_disk01_sa21

crw-rw—-   1 oracle     dba        135 0x000009 Sep 20 16:25 ora_vote_disk02_sa21

crw-rw—-   1 oracle     dba        135 0x00000a Sep 20 16:25 ora_vote_disk03_sa21

crw-rw—-   1 oracle     dba        135 0x00000d Sep 27 14:07 ora_vote_disk04_sa21

crw-rw—-   1 oracle     dba        135 0x00000e Sep 27 14:07 ora_vote_disk05_sa21

crw-rw—-   1 oracle     dba        135 0x00001c Oct 28  2009 ora_vote_disk1_eva4

crw-rw—-   1 oracle     dba        135 0x00001f Oct 28  2009 ora_vote_disk2_eva2

GROUP_NUMBER NAME                 STATE    V PATH

———— ——————– ——– – ————————————————–

3 DEV_VOTE_TEMP_0001   NORMAL   N /dev/rdisk/ora_asm_disk2_eva4

3 DEV_VOTE_TEMP_0000   NORMAL   N /dev/rdisk/ora_asm_disk1_eva4

3 DEV_VOTE_TEMP_0002   NORMAL   N /dev/rdisk/ora_asm_disk3_eva4

1 DEV_VOTE_FG_0000     NORMAL   Y /dev/rdisk/ora_vote_disk01_sa21

1 DEV_VOTE_FG_0001     NORMAL   Y /dev/rdisk/ora_vote_disk02_sa21

1 DEV_VOTE_FG_0002     NORMAL   Y /dev/rdisk/ora_vote_disk03_sa21

##  STATE    File Universal Id                File Name Disk group

—  —–    —————–                ——— ———

1. ONLINE   571a4bdd1fb34fb7bf1c7433b2cf0e07 (/dev/rdisk/ora_vote_disk01_sa21) [DEV_VOTE_FG]

2. ONLINE   3fc520d1f9794f5dbfca4272363edfe1 (/dev/rdisk/ora_vote_disk02_sa21) [DEV_VOTE_FG]

3. ONLINE   368387fd5a294f08bfa0985cb12a52dd (/dev/rdisk/ora_vote_disk03_sa21) [DEV_VOTE_FG]

GROUP_NUMBER DISK_NUMBER NAME                 PATH

———— ———– ——————– ————————————————–

0           1                      /dev/rdisk/ora_ocr_disk1_eva2

0           2                      /dev/rdisk/ora_vote_disk04_sa21

0          12                      /dev/rdisk/ora_asm_disk4_eva4

0          15                      /dev/rdisk/ora_vote_disk05_sa21

3           1 DEV_VOTE_TEMP_0001   /dev/rdisk/ora_asm_disk2_eva4

2           4 DEV_DATA_0004        /dev/rdisk/ora_vote_disk2_eva2

3           0 DEV_VOTE_TEMP_0000   /dev/rdisk/ora_asm_disk1_eva4

3           2 DEV_VOTE_TEMP_0002   /dev/rdisk/ora_asm_disk3_eva4

2           3 DEV_DATA_0003        /dev/rdisk/ora_vote_disk1_eva4

2           5 DEV_DATA_0005        /dev/rdisk/ora_data_disk01_sa21

2           6 DEV_DATA_0006        /dev/rdisk/ora_data_disk02_sa21

2           7 DEV_DATA_0007        /dev/rdisk/ora_data_disk03_sa21

1           0 DEV_VOTE_FG_0000     /dev/rdisk/ora_vote_disk01_sa21

1           1 DEV_VOTE_FG_0001     /dev/rdisk/ora_vote_disk02_sa21

1           2 DEV_VOTE_FG_0002     /dev/rdisk/ora_vote_disk03_sa21

2           8 DEV_DATA_0008        /dev/rdisk/ora_data_disk04_sa21

added /dev/rdisk/ora_vote_disk04_sa21 to diskgroup DEV_VOTE_FG:

GROUP_NUMBER DISK_NUMBER NAME                           PATH

———— ———– —————————— ————————————————–

0           1                                /dev/rdisk/ora_ocr_disk1_eva2

0          12                                /dev/rdisk/ora_asm_disk4_eva4

0          15                                /dev/rdisk/ora_vote_disk05_sa21

3           1 DEV_VOTE_TEMP_0001             /dev/rdisk/ora_asm_disk2_eva4

2           4 DEV_DATA_0004                  /dev/rdisk/ora_vote_disk2_eva2

3           0 DEV_VOTE_TEMP_0000             /dev/rdisk/ora_asm_disk1_eva4

3           2 DEV_VOTE_TEMP_0002             /dev/rdisk/ora_asm_disk3_eva4

2           3 DEV_DATA_0003                  /dev/rdisk/ora_vote_disk1_eva4

2           5 DEV_DATA_0005                  /dev/rdisk/ora_data_disk01_sa21

2           6 DEV_DATA_0006                  /dev/rdisk/ora_data_disk02_sa21

2           7 DEV_DATA_0007                  /dev/rdisk/ora_data_disk03_sa21

1           0 DEV_VOTE_FG_0000               /dev/rdisk/ora_vote_disk01_sa21

1           1 DEV_VOTE_FG_0001               /dev/rdisk/ora_vote_disk02_sa21

1           2 DEV_VOTE_FG_0002               /dev/rdisk/ora_vote_disk03_sa21

2           8 DEV_DATA_0008                  /dev/rdisk/ora_data_disk04_sa21

1           3 DEV_VOTE_FG_0003               /dev/rdisk/ora_vote_disk04_sa21

##  STATE    File Universal Id                File Name Disk group

—  —–    —————–                ——— ———

1. ONLINE   571a4bdd1fb34fb7bf1c7433b2cf0e07 (/dev/rdisk/ora_vote_disk01_sa21) [DEV_VOTE_FG]

2. ONLINE   3fc520d1f9794f5dbfca4272363edfe1 (/dev/rdisk/ora_vote_disk02_sa21) [DEV_VOTE_FG]

3. ONLINE   368387fd5a294f08bfa0985cb12a52dd (/dev/rdisk/ora_vote_disk03_sa21) [DEV_VOTE_FG]

dropped:

1           2 DEV_VOTE_FG_0002               /dev/rdisk/ora_vote_disk03_sa21

##  STATE    File Universal Id                File Name Disk group

—  —–    —————–                ——— ———

1. ONLINE   571a4bdd1fb34fb7bf1c7433b2cf0e07 (/dev/rdisk/ora_vote_disk01_sa21) [DEV_VOTE_FG]

2. ONLINE   3fc520d1f9794f5dbfca4272363edfe1 (/dev/rdisk/ora_vote_disk02_sa21) [DEV_VOTE_FG]

3. ONLINE   60986046fc474f84bfcef38b66bba514 (/dev/rdisk/ora_vote_disk04_sa21) [DEV_VOTE_FG]

//undrop does not work

alter diskgroup DEV_VOTE_FG add disk ‘/dev/rdisk/ora_vote_disk03_sa21’ rebalance power 5 wait

select group_number,disk_number,name,path from v$asm_disk where name like ‘%VOTE%’;

GROUP_NUMBER DISK_NUMBER NAME                           PATH

———— ———– —————————— ————————————————–

3           1 DEV_VOTE_TEMP_0001             /dev/rdisk/ora_asm_disk2_eva4

3           0 DEV_VOTE_TEMP_0000             /dev/rdisk/ora_asm_disk1_eva4

3           2 DEV_VOTE_TEMP_0002             /dev/rdisk/ora_asm_disk3_eva4

1           0 DEV_VOTE_FG_0000               /dev/rdisk/ora_vote_disk01_sa21

1           1 DEV_VOTE_FG_0001               /dev/rdisk/ora_vote_disk02_sa21

1           2 DEV_VOTE_FG_0002               /dev/rdisk/ora_vote_disk03_sa21

1           3 DEV_VOTE_FG_0003               /dev/rdisk/ora_vote_disk04_sa21

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s