No ADR base is set Oracle RAC 19c

When I try connect to adrci appears below error:

adrci

ADRCI: Release 19.0.0.0.0 - Production on Thu Apr 14 14:51:54 2022

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

No ADR base is set
adrci> show homes
No ADR homes are set
adrci> exit

Solution: check to see if there is a file adrci_dir.mif in $ORACLE_HOME/log/diag if the directory does not exist do the following :

mkdir -p $ORACLE_HOME/log/diag
printf "%s" "/u01/app/oracle" > $ORACLE_HOME/log/diag/adrci_dir.mif

"/u01/app/oracle" will be your ADR base, which is actualy oracle_base. After that adrci will work:

adrci

ADRCI: Release 19.0.0.0.0 - Production on Thu Apr 14 15:01:21 2022

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/u01/app/oracle"
adrci> show homes
ADR Homes: 
diag/asmtool/user_oracle/host_578647856_107
diag/asmtool/user_oracle/host_578647856_80
diag/asmtool/user_grid/host_578647856_107
diag/rdbms/testdb2/TESTDB2
diag/rdbms/testdb/TESTDB

Exadata CELL-01514: Connect Error. Verify that Management Server is listening at the specified HTTP port: 8888

 Today we faced below problem while trying to connect cellcli on Exadata Storage Cell

"CELL-01514: Connect Error. Verify that Management Server is listening at the specified HTTP port: 8888"

[root@testcel01 ~]# cellcli -e list physicaldisk detail

CELL-01514: Connect Error. Verify that Management Server is listening at the specified HTTP port: 8888.
[root@testcel01 ~]#

Cell server is not running actually and we need to solve this problem.
To solve this problem, you need to start cell daemon:

[root@testcel01 ~]# service celld start
1: ERROR: Resource temporarily unavailable
rds-ping failed on 192.168.10.27
1: ERROR: Resource temporarily unavailable
rds-ping failed on 192.168.23.16
1: 90 usec
1: 134 usec

Starting the RS, CELLSRV, and MS services...
Getting the state of RS services... running
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Starting MS services...
The STARTUP of MS services was successful.
[root@testcel01 ~]# 
[root@testcel01 ~]#

Cell server daemon was started successful. You can check its status:

[root@testcel01 ~]# service celld status
rsStatus: running
msStatus: running
cellsrvStatus: running
[root@testcel01 ~]#

TFA-00002 : Oracle Trace File Analyzer (TFA) is not running

When I tried to view oratop on database server appears below error:

tfactl oratop -database testdb
TFA-00002 Oracle Trace File Analyzer (TFA) is not running

Solution : As a root user

/etc/init.d/init.tfa  start  
Starting TFA..
start: Job is already running: oracle-tfa
Waiting up to 100 seconds for TFA to be started..
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Successfully started TFA Process..
. . . . .
TFA-00002 : Oracle Trace File Analyzer (TFA) is not running
TFA Failed to start listening for commands

If it is not started successfully than check below : 

run synctfanodes.sh if you have root password . 

if no root password copy below files to TFA Failed node and try to restart TFA .. Please check whether these 4 files exist on all nodes are similar.

TFA_HOME/server.jks 

TFA_HOME/client.jks 

TFA_HOME/internal/ssl.properties 

TFA_HOME/internal/ 

portmapping.txt If not similar just copy these files to TFA Failing node and try TFA Restart ..It Worked for me .

 
./tfactl  -v "debug" start
Starting TFA..
start: Job is already running: oracle-tfa
Waiting up to 100 seconds for TFA to be started..
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Successfully started TFA Process..
. . . . .
TFA Started and listening for commands

tfactl print status

.------------------------------------------------------------------------------------------------.
| Host    | Status of TFA | PID    | Port | Version    | Build ID             | Inventory Status |
+---------+---------------+--------+------+------------+----------------------+------------------+
| x82db01 | RUNNING       | 272939 | 5000 | 21.2.1.0.0 | 21210020210813042249 | COMPLETE         |
'---------+---------------+--------+------+------------+----------------------+------------------'

After that we can use oratop and monitor database with : tfactl oratop -database testdb

ORA-00600 [17147] ORA-48216 When Querying V$DIAG_ALERT_EXT View

When I try to query v$diag_alert_ext appears error:

ORA-00600: internal error code, arguments: [17147], [0x110A09510], [], [], [], [], [], [], [], [], [], []
ORA-48216: Field Len Exceeds Max Field Length [5] [30] [29286] [66]

Here is query which is used for quering alert.log entries

select instance_id, originating_timestamp, message_text, '1'
  from table(gv$(cursor (select instance_id, originating_timestamp, message_text
                    from v$diag_alert_ext a
                   where originating_timestamp > (sysdate - 1)
                     and (message_text like '%ORA-%' or
                         message_text like '%CRS-%'))))
 order by originating_timestamp desc;

As I found on metalink The issue is matching unpublished Bug 21266522 duplicate of unpublished Bug 18909196 - QUERIES INVOLVING X$DBGALERTEXT SOMETIMES FAIL DUE TO ORA-48216.
Solution:

1. Download and install the one-off Patch 18909196, available for your platform (this issue only exists in 11.2.0.4). If a patch does not exist for your platform please open a Service Request for Oracle Support.
-OR-
2. Implement the workaround: remove (i.e. delete) the xml-based alert log within /alert/log.xml (and log_*.xml).

After startup database with srvctl, sqlplus as sysdba results in ORA-12547: TNS:lost contact

We had a very strange situation on one of our test environment: 

1. If use sqlplus to start instance, then use srvctl to stop it will failed with ORA-12547

[oracle@test_host:/home/oracle]$sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Nov 15 12:57:46 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@test_host:/home/oracle]$srvctl stop database -d testdb
PRCD-1334 : failed to stop database testdb
PRCD-1124 : Failed to stop database testdb and its services
PRCD-1131 : Failed to stop database testdb and its services on nodes
PRCR-1133 : Failed to stop database testdb and its running services
PRCR-1132 : Failed to stop resources using a filter
ORA-12547: TNS:lost contact
CRS-2675: Stop of 'ora.testdb.db' on 'test_host' failed

2 . If use srvctl to start instance, sqlplus / as sysdba will get ORA-12547

[oracle@test_host:/home/oracle]$sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Thu Nov 15 12:59:21 2021
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.

ERROR:
ORA-12547: TNS:lost contact

Enter user-name:
ERROR:
ORA-12547: TNS:lost contact

Enter user-name:
ERROR:
ORA-12547: TNS:lost contact

SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus

3. Checked the permission for oracle binary and library and orabasetab, no problem found:

$ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle asmadmin 441253072 Nov 15 12:29 /u01/app/oracle/product/19.0.0/db_1/bin/oracle

( No change after we start instance by using srvctl or sqlplus )

$ls -l $ORACLE_HOME|grep lib
drwxr-xr-x 2 oracle oinstall 4096 Apr 17 2019 jlib
drwxr-xr-x 4 oracle oinstall 12288 Nov 13 12:29 lib

$more orabasetab
#orabasetab file is used to track Oracle Home associated with Oracle Base
/u01/app/oracle/product/19.0.0/db_1:/u01/app/oracle:OraDB19Home1:N:

Solution: The issue is due to incorrect configuration for Oracle home in OCR:

[grid@test_host:/home/grid]$srvctl config database -db testdb
Database unique name: testdb
Database name:
Oracle home: /u01/app/oracle/product/19.0.0/db_1/

So problem is / symbol at the end of oracle_home and we can change it

srvctl stop database -d testdb
srvctl modify database -d testdb -o /u01/app/oracle/product/19.0.0/db_1
srvctl start database -d testdb

This is corresponding oracle support doc: DB started by sqlplus cannot be stopped by srvctl( ORA-12547 ), and vice versa (Doc ID 2612663.1)