ORA-16751: failed to switchover to physical standby database and Switchover Ends with Two Physical Standby Databases

You might find yourself in a situation where your switchover fails and left your environment with two physical standby databases.
From the broker your might see the following error:
ORA-16816: incorrect database role
Solution:
!!!!!!!!!!!!!!!!! First of all you need to stop database instances on all servers.
1. Logon (as sysdba) to the instance that was your Primary database instance before the switchover.

2. Confirm the database role.

SQL> select database_role from v$database;

DATABASE_ROLE

-------------------------------

PHYSICAL STANDBY

3. Shutdown the instance.

SQL> shutdown immediate;

Oracle Instance Shutdown.

4. Mount the database.

SQL> startup mount;

Database opened.

5. Cancel the MRP process. You will receive "ORA-16136: Managed Standby Recovery not active" if it is not running, but you can ignore.

SQL> alter database recover managed standby database cancel;

6. Terminate the current switchover to Standby that never completed fully.

SQL> alter database recover managed standby database finish;

Database altered

7. Now switchover to Primary.

SQL> alter database commit to switchover to primary with session shutdown;

Database altered

8. Open the database.

SQL> alter database open;

Database altered.

9. Confirm the database role.

SQL> select database_role from v$database;

DATABASE_ROLE

------------------------------

PRIMARY

Additional Steps
When attempting to open the Primary Database you may suffer the following error:

SQL> alter database open

*

ERROR at line 1:

ORA-16649: possible failover to another database prevents this database being opened

In this case, before you can open the database, you must disable Data Guard Broker as follows:

 SQL> alter system set dg_broker_start=false scope=both;

 System altered.

SQL> alter database open;

 Database altered.

Now re-install Data Guard Broker.

Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated

In Oracle Database 12c command

SQL> alter database recover managed standby database disconnect from session using current logfile;

is deprecated and if you use it in alert log file you'll see warnings about it:

Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
Warning: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE has been deprecated.
ALTER DATABASE RECOVER  managed standby database disconnect from session using current logfile  

Instead of this command we can use next command in oracle database 12c

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT NODELAY;

To start apply services on a physical standby database, ensure the physical standby database is started and mounted and then start Redo Apply as follows:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE;

This also automatically enables real-time apply provided the standby database is configured with a standby redo log and is in ARCHIVELOG mode.
Redo Apply can be run either as a foreground session or as a background process. To start Redo Apply in the foreground, issue the following SQL statement:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE;

If you start a foreground session, control is not returned to the command prompt until recovery is canceled by another session.
To start Redo Apply in the background, include the DISCONNECT keyword on the SQL statement. For example:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

or

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING ARCHIVED LOGFILE DISCONNECT;

This statement starts a detached server process and immediately returns control to the user. While the managed recovery process is performing recovery in the background, the foreground process that issued the RECOVER statement can continue performing other tasks. This command does not disconnect the current SQL session.

WARNING! Recovering data file from a fuzzy .. file ORA-01194: file needs more recovery to be consistent

Oracle Release - 10.2.0.5.0,  platform - HP-UX (64-bit)

problem - Disk damaged on physical standby and dozens of data files were lost.  Instance was down and archive log gap presented between standby and primary databases.

Instead of restore from backup we just copied (fast way) online data files from primary and started applying missing logs.

At the start of apply process we saw warnings as follow in the standby alert log telling that we are supplying incorrect files.

WARNING! Recovering data file 291 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 529 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 530 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command. ...

Later, after all logs applied successfully we tried to open standby in read only mode:

SQL> alter database open read only ;
alter database open read only
*
ERROR at line 1:
ORA-16004: backup database requires recovery
ORA-01194: file 291 needs more recovery to be consistent
ORA-01110: data file 291 ...

Although all logs applied and dbs are synchronized there was a problem with the data file in our warning list - 291.
Of course it indicates that files should have been transferred in backup mode.

Apparently, we found all relevant tablespaces and put them into backup mode:

ALTER TABLESPACE TS1 BEGIN BACKUP;
ALETR TABLESPACE TS2 BEGIN BACKUP;

As transfer finished we took them out from backup mode.

ALTER TABLESPACE TS1 END BACKUP;
ALETR TABLESPACE TS2 END BACKUP;

It is all, after applying all logs we could successfully open standby in read only mode.

Thanks,