OEM stopped working and start failed with webtier could not be started

Suddenly Oracle Enterprise Manager - oem stopped working and when we are trying to start it appears error: 

$emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Could Not Be Started.
Error Occurred: WebTier Could Not Be Started.

Oracle provides a default wallet and self-signed certificate Out-Of-The-Box that is used to test the functionality of SSL.

These Self-Signed Certificates will expire after some time and can prevent OHS from starting. This document is provided

to provide a simple method to recreate this wallet.

Solution: 

1. Stop OHS(and AdminServer if collocated. Next, move ${ORACLE_INSTANCE}/config/fmwconfig/components/${COMPONENT_TYPE}/${COMPONENT_NAME}/keystores/default/cwallet.sso

to cwallet.sso.expired. Basically rename it as it will be recreated.

Example:

$ pwd
$DOMAIN_HOME_PATH/config/fmwconfig/components/OHS/ohs1/keystores/default

$ mv cwallet.sso cwallet.sso.expired

2. Make sure orapki is in the $PATH variable.

Check to see if orapki is in the $PATH Environment Variable:

$ which orapki

Add it to the beginning of the $PATH environmental variable:

$ export PATH=$OMS_HOME/oracle_common/bin:$PATH

4. Execute the following commands from the directory where the old one was renamed. i.e. ${ORACLE_INSTANCE}/config/fmwconfig/components/${COMPONENT_TYPE}/${COMPONENT_NAME}/keystores/default/

$ orapki wallet create -wallet . -auto_login_only (NOTE: The dot (.) is specified here as the Present Working Directory.)
Example output:

Oracle PKI Tool : Version 12.2.1.2.0
Copyright (c) 2004, 2016, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

- $ ls
cwallet.sso cwallet.sso.expired cwallet.sso.lck

5. Now recreate the self-signed certificate using the same DN used in the original wallet.

$ orapki wallet add -wallet . -dn 'CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY' -keysize 2048 -self_signed -validity 3650 -auto_login_only
(NOTE: The dot (.) is specified here as the Present Working Directory.)
Example output:

Oracle PKI Tool : Version 12.2.1.2.0
Copyright (c) 2004, 2016, Oracle and/or its affiliates. All rights reserved.

Operation is successfully completed.

6. Now check the wallet with the display command:

$ orapki wallet display -wallet . (NOTE: The dot (.) is specified here as the Present Working Directory.)
Example output:

Oracle PKI Tool : Version 12.2.1.2.0
Copyright (c) 2004, 2016, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject: CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY
Trusted Certificates:
Subject: CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY

$ openssl x509 -in server.cer -noout -text
Certificate:
Data:
Version: 1 (0x0)
Serial Number:
9a:a0:bb:e2:f5:ff:3b:e8:8d:46:7a:ac:18:f3:89:39
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=FOR TESTING ONLY, OU=FOR TESTING ONLY, CN=localhost
Validity
Not Before: Nov 17 15:45:08 2020 GMT
Not After : Nov 15 15:45:08 2030 GMT

7. Now that a new, self-signed certificate/wallet has been created, in the same directory where the old one was renamed, OHS(and AdminServer if Collocated) can be restarted successfully.

ORA-01000: maximum open cursors exceeded oracle.sysman.emSDK.emd.comm.MetricGetException: ORA-01000: maximum open cursors exceeded

Sometimes we are not getting alerts from OEM(Oracle Enterprise Manager) and if we try to check some metric values it is throwing error:
ORA-01000: maximum open cursors exceeded oracle.sysman.emSDK.emd.comm.MetricGetException: ORA-01000: maximum open cursors exceeded 

I Found interesting Doc about this issue on metalink: 

ORA-01000: MAXIMUM OPEN CURSORS EXCEEDED (Doc ID 2751726.1).

It seems it is Bug 31542357 - Use of setTimeToLiveTimeout causes leak of cursors and ORA-01000 eventually 

Solution: 

For 13.4 Cloud Control the patch will be included in the 13.4 RU7 and newer patches. 

The work around from development is: On agent side add this property in emd.properties and restart the agent. The following property will kill the seesion on db and avoid the cursor leak.

SqlConnectionCache._TimeToLiveTimeout=0

12C Database reports KFOD Error During Oracle Exadata Database Machine Discovery in OEM: "DiscoveryWarning: kfod returns status 126"

During Oracle Exadata Database Machine Discovery in Oracle Enterprise Manager (OEM) appears error:

The following errors are found during discovery. 
Please examine the error messages and the targets being discovered if any. 
DiscoveryWarning: kfod returns status 126. 
Please check the OSSCONF environment variable and make sure that 
cellinit.ora and cellip.ora are readable by the EM agent. 
DiscoveryWarning: /u01/app/oracle/product/12.2.0.1/dbhome_1/bin/kfod: line 22: 
/u01/agent12c/agent12c/sysman/emd/%ORACLE_HOME%/bin/kfod.bin: No such file or directory 
/u01/app/oracle/product/12.2.0.1/dbhome_1/bin/kfod: line 22: 
exec: /u01/agent12c/agent12c/sysman/emd/%ORACLE_HOME%/bin/kfod.bin: 
cannot execute: No such file or directory

cause:This is a known bug in the kfod wrapper script: Unpublished BUG 19682778 - 121021GIPSU: KFOD FILE IN RAC HOME NOT CORRECT AFTER APPLYING GIPSU
Workaround:
- Change the first line in the kfod wrapper script "$OHOME/bin/kfod.bin"
       OHOME=%ORACLE_HOME%
       to
       OHOME=<DB_Home>      Eg. OHOME=/u01/app/oracle/product/12.2.0.1/dbhome_1

Your kfod file must look like this:
cd $ORACLE_HOME
pwd
/u01/app/oracle/product/12.2.0.1/dbhome_1
cd bin/

cat kfod

#!/bin/sh
#
# $Header: opsm/utl/kfod.sbs
#
# kfod
#
# Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.
#
#    NAME
#      kfod - KFOD utility
#
#    DESCRIPTION
#      This is a script which is a wrapper on top of kfod.bin
#      This script is only shipped to the DATABASE home
#
#    MODIFIED   (MM/DD/YY)
#        samjo   05/21/14 - Creation
OHOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
ORACLE_HOME=${OHOME}
export ORACLE_HOME

exec $OHOME/bin/kfod.bin "$@"

Failed To Launch Process: Auth Fail During Infiniband Switch Discovery In OEM

Below error is reported during Infiniband switch discovery in Oracle Enterprise Manager (OEM) 12c Release 4 (12.1.0.4.0)

Failed To Launch process: Auth Fail 

However you are able to ssh to same switch using nm2user, without any errors.
This is caused due to the setting "PasswordAuthentication no" in the IB Switch's
/etc/ssh/sshd_config file, which disables clear text password authentication:

#cat /etc/ssh/sshd_config

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
# NM2 change - set PermitEmptyPasswords to no
PermitEmptyPasswords no
# NM2 change - set PasswordAuthentication to no
PasswordAuthentication no

Solution :
A workaround to complete the discovery is to include (uncomment) "PasswordAuthentication yes" in the sshd_config file and remove (comment) "PasswordAuthentication no", and restart SSHD using /etc/init.d/sshd restart.

Enterprise Manager (OEM) is not able to connect to the database instance. The state of the components are listed below

When started browser for login in enterprise manager, it shows next:

Enterprise Manager is not able to connect to the database instance. The state of the components are listed below.
first of all check enterprise manager status :

emctl status dbconsole

that told Oracle Enterprise Manager 10g is running.
after that i checked log files under $ORACLE_HOME\hostname.domainname_dbname\sysman\log
and found the following error
[SystemThreadGroup-8] ERROR app.SessionObjectManager sessionDestroyed.128 - java.sql.SQLException: ORA-28000: the account is locked
now Solution is simple:

sqlplus  / as sysdba                            

SQL*Plus: Release 10.2.0.4.0 - Production on Thu Oct 10 09:59:32 2013

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select account_status from dba_users where username='SYSMAN';

ACCOUNT_STATUS
--------------------------------
LOCKED(TIMED)

SQL> alter user sysman account unlock;

User altered.

after that run browser and enjoy with enterprise manager.

if that don't help then make next steps:

SQL> alter user sysman account unlock;
SQL> alter user sysman identified by passwd;

emctl setpasswd dbconsole

and this must helps.