Losing Oracle RAC vote disks can cause catastrophic failure, so one would generally want to spread their disks across multiple storage devices to prevent this. In my career I have worked for both small and large companies. The biggest difference here is generally resources, and at a small company I ran across a setup where they had only two network storage arrays and thus only two separate locations for vote disk retention.
The fix for this is quite novel. When a voting file disk is lost, ASM will move the voting file to the next available disk in the group IF one is available, and it does this automagically, however if there is not an extra disk for asm to use and the cluster was brought down by an outage in the past as a result.
Thus the solution to this problem is as easy as adding a 4th disk, dropping the problematic disk, and the new voting file gets instantly created on the 4th disk without manual involvement. Of course when I explained this I was scoffed at and told that I just did not understand Oracle RAC, so I ended up walking them through it and these are my notes.
Moving forward, the suggestion is to add a 4th disk to all voting disk groups to add an extra level of redundancy. With an extra disk already added to the disk group in use, anytime a voting disk(s) becomes unavailable it will automatically be replaced by the asm instance and the result will be 0% down time.
WORK FLOW DURING TESTING:
crw-rw—- 1 oracle dba 135 0x000021 Oct 28 2009 ora_asm_disk1_eva4
crw-rw—- 1 oracle dba 135 0x000022 Oct 28 2009 ora_asm_disk2_eva4
crw-rw—- 1 oracle dba 135 0x000023 Oct 28 2009 ora_asm_disk3_eva4
crw-rw—- 1 oracle dba 135 0x00000b Sep 24 14:06 ora_asm_disk4_eva4
crw-rw—- 1 oracle dba 135 0x000004 Oct 27 11:33 ora_data_disk01_sa21
crw-rw—- 1 oracle dba 135 0x000007 Oct 27 14:33 ora_data_disk02_sa21
crw-rw—- 1 oracle dba 135 0x000008 Oct 27 13:12 ora_data_disk03_sa21
crw-rw—- 1 oracle dba 135 0x00000c Oct 27 14:10 ora_data_disk04_sa21
crw-rw—- 1 oracle dba 135 0x000020 Oct 28 2009 ora_ocr_disk1_eva2
crw-rw—- 1 oracle dba 135 0x000005 Sep 20 16:25 ora_vote_disk01_sa21
crw-rw—- 1 oracle dba 135 0x000009 Sep 20 16:25 ora_vote_disk02_sa21
crw-rw—- 1 oracle dba 135 0x00000a Sep 20 16:25 ora_vote_disk03_sa21
crw-rw—- 1 oracle dba 135 0x00000d Sep 27 14:07 ora_vote_disk04_sa21
crw-rw—- 1 oracle dba 135 0x00000e Sep 27 14:07 ora_vote_disk05_sa21
crw-rw—- 1 oracle dba 135 0x00001c Oct 28 2009 ora_vote_disk1_eva4
crw-rw—- 1 oracle dba 135 0x00001f Oct 28 2009 ora_vote_disk2_eva2
GROUP_NUMBER NAME STATE V PATH
———— ——————– ——– – ————————————————–
3 DEV_VOTE_TEMP_0001 NORMAL N /dev/rdisk/ora_asm_disk2_eva4
3 DEV_VOTE_TEMP_0000 NORMAL N /dev/rdisk/ora_asm_disk1_eva4
3 DEV_VOTE_TEMP_0002 NORMAL N /dev/rdisk/ora_asm_disk3_eva4
1 DEV_VOTE_FG_0000 NORMAL Y /dev/rdisk/ora_vote_disk01_sa21
1 DEV_VOTE_FG_0001 NORMAL Y /dev/rdisk/ora_vote_disk02_sa21
1 DEV_VOTE_FG_0002 NORMAL Y /dev/rdisk/ora_vote_disk03_sa21
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 571a4bdd1fb34fb7bf1c7433b2cf0e07 (/dev/rdisk/ora_vote_disk01_sa21) [DEV_VOTE_FG]
2. ONLINE 3fc520d1f9794f5dbfca4272363edfe1 (/dev/rdisk/ora_vote_disk02_sa21) [DEV_VOTE_FG]
3. ONLINE 368387fd5a294f08bfa0985cb12a52dd (/dev/rdisk/ora_vote_disk03_sa21) [DEV_VOTE_FG]
GROUP_NUMBER DISK_NUMBER NAME PATH
———— ———– ——————– ————————————————–
0 1 /dev/rdisk/ora_ocr_disk1_eva2
0 2 /dev/rdisk/ora_vote_disk04_sa21
0 12 /dev/rdisk/ora_asm_disk4_eva4
0 15 /dev/rdisk/ora_vote_disk05_sa21
3 1 DEV_VOTE_TEMP_0001 /dev/rdisk/ora_asm_disk2_eva4
2 4 DEV_DATA_0004 /dev/rdisk/ora_vote_disk2_eva2
3 0 DEV_VOTE_TEMP_0000 /dev/rdisk/ora_asm_disk1_eva4
3 2 DEV_VOTE_TEMP_0002 /dev/rdisk/ora_asm_disk3_eva4
2 3 DEV_DATA_0003 /dev/rdisk/ora_vote_disk1_eva4
2 5 DEV_DATA_0005 /dev/rdisk/ora_data_disk01_sa21
2 6 DEV_DATA_0006 /dev/rdisk/ora_data_disk02_sa21
2 7 DEV_DATA_0007 /dev/rdisk/ora_data_disk03_sa21
1 0 DEV_VOTE_FG_0000 /dev/rdisk/ora_vote_disk01_sa21
1 1 DEV_VOTE_FG_0001 /dev/rdisk/ora_vote_disk02_sa21
1 2 DEV_VOTE_FG_0002 /dev/rdisk/ora_vote_disk03_sa21
2 8 DEV_DATA_0008 /dev/rdisk/ora_data_disk04_sa21
added /dev/rdisk/ora_vote_disk04_sa21 to diskgroup DEV_VOTE_FG:
GROUP_NUMBER DISK_NUMBER NAME PATH
———— ———– —————————— ————————————————–
0 1 /dev/rdisk/ora_ocr_disk1_eva2
0 12 /dev/rdisk/ora_asm_disk4_eva4
0 15 /dev/rdisk/ora_vote_disk05_sa21
3 1 DEV_VOTE_TEMP_0001 /dev/rdisk/ora_asm_disk2_eva4
2 4 DEV_DATA_0004 /dev/rdisk/ora_vote_disk2_eva2
3 0 DEV_VOTE_TEMP_0000 /dev/rdisk/ora_asm_disk1_eva4
3 2 DEV_VOTE_TEMP_0002 /dev/rdisk/ora_asm_disk3_eva4
2 3 DEV_DATA_0003 /dev/rdisk/ora_vote_disk1_eva4
2 5 DEV_DATA_0005 /dev/rdisk/ora_data_disk01_sa21
2 6 DEV_DATA_0006 /dev/rdisk/ora_data_disk02_sa21
2 7 DEV_DATA_0007 /dev/rdisk/ora_data_disk03_sa21
1 0 DEV_VOTE_FG_0000 /dev/rdisk/ora_vote_disk01_sa21
1 1 DEV_VOTE_FG_0001 /dev/rdisk/ora_vote_disk02_sa21
1 2 DEV_VOTE_FG_0002 /dev/rdisk/ora_vote_disk03_sa21
2 8 DEV_DATA_0008 /dev/rdisk/ora_data_disk04_sa21
1 3 DEV_VOTE_FG_0003 /dev/rdisk/ora_vote_disk04_sa21
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 571a4bdd1fb34fb7bf1c7433b2cf0e07 (/dev/rdisk/ora_vote_disk01_sa21) [DEV_VOTE_FG]
2. ONLINE 3fc520d1f9794f5dbfca4272363edfe1 (/dev/rdisk/ora_vote_disk02_sa21) [DEV_VOTE_FG]
3. ONLINE 368387fd5a294f08bfa0985cb12a52dd (/dev/rdisk/ora_vote_disk03_sa21) [DEV_VOTE_FG]
dropped:
1 2 DEV_VOTE_FG_0002 /dev/rdisk/ora_vote_disk03_sa21
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 571a4bdd1fb34fb7bf1c7433b2cf0e07 (/dev/rdisk/ora_vote_disk01_sa21) [DEV_VOTE_FG]
2. ONLINE 3fc520d1f9794f5dbfca4272363edfe1 (/dev/rdisk/ora_vote_disk02_sa21) [DEV_VOTE_FG]
3. ONLINE 60986046fc474f84bfcef38b66bba514 (/dev/rdisk/ora_vote_disk04_sa21) [DEV_VOTE_FG]
//undrop does not work
alter diskgroup DEV_VOTE_FG add disk ‘/dev/rdisk/ora_vote_disk03_sa21’ rebalance power 5 wait
select group_number,disk_number,name,path from v$asm_disk where name like ‘%VOTE%’;
GROUP_NUMBER DISK_NUMBER NAME PATH
———— ———– —————————— ————————————————–
3 1 DEV_VOTE_TEMP_0001 /dev/rdisk/ora_asm_disk2_eva4
3 0 DEV_VOTE_TEMP_0000 /dev/rdisk/ora_asm_disk1_eva4
3 2 DEV_VOTE_TEMP_0002 /dev/rdisk/ora_asm_disk3_eva4
1 0 DEV_VOTE_FG_0000 /dev/rdisk/ora_vote_disk01_sa21
1 1 DEV_VOTE_FG_0001 /dev/rdisk/ora_vote_disk02_sa21
1 2 DEV_VOTE_FG_0002 /dev/rdisk/ora_vote_disk03_sa21
1 3 DEV_VOTE_FG_0003 /dev/rdisk/ora_vote_disk04_sa21
Missouri dBA 