remidian.com

Here’s how to overcome ORA-20206 in Oracle GRID environment, agent status unclear…

Scenario

• Removed (Windows) Agent from Grid. (With SQL Server Management plugin)
• Tried to restart this SAME agent, errors below occur.

# $ORACLE_HOME/sysman/log/emagent.log contains
 
2008-01-17 14:13:52 Thread-4032 EMAgent started successfully (00702)
2008-01-17 14:14:21 Thread-2720 <Upload Manager> Unsuccessful Upload attempts for XML file exceeds specified limit=3, 
      Agent will shutdown (00851)
2008-01-17 14:14:21 Thread-4032 EMAgent abnormal terminating (00704)

# $ORACLE_HOME/sysman/log/emdctl.trc contains
 
2008-01-17 14:13:32 Thread-752 WARN  http: snmehl_connect: connect failed to (<agent_hostname>:3872): 
      No connection could be made because the target machine actively refused it.
 (error = 10061)
2008-01-17 14:13:32 Thread-752 ERROR main: nmectla_agentctl: Error connecting to 
      https://<agent_hostname>:3872/emd/main/. Returning status code 1
2008-01-17 14:14:23 Thread-172 WARN  http: snmehl_connect: connect failed to 
      (<agent_hostname>:3872): 
      No connection could be made because the target machine actively refused it. (error = 10061)
2008-01-17 14:14:23 Thread-172 ERROR main: nmectla_agentctl: Error connecting to 
      https://<agent_hostname>:3872/emd/main/. Returning status code 1
 

# $ORACLE_HOME/sysman/log/emagent.trc contains
 
2008-01-17 14:14:18 Thread-192 WARN  TargetManager: Regenerating all Metadata
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: In getTypeAndNameForHostTarget
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: getTypeAndNameForHostTarget found host, <agent_hostname>
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: In getTypeAndNameForHostTarget
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: getTypeAndNameForHostTarget found host, <agent_hostname>
2008-01-17 14:14:18 Thread-192 WARN  collector: Regenerating all DefaultColls
2008-01-17 14:14:18 Thread-192 ERROR upload: Failed to upload file A0000001.xml, ret = -6
2008-01-17 14:14:19 Thread-192 WARN  upload: FxferSend: received fatal error in header from repository: 
 
https://<oms_hostname>:1159/em/upload
 
FATAL_ERROR::500|ORA-20206: Target does not exist:  Agent does not exist for https://<agent_hostname>:3872/emd/main/
ORA-06512: at "SYSMAN.EMD_LOADER", line 1795
ORA-06512: at line 1
 
.
.
.
 
2008-01-17 14:14:21 Thread-2720 ERROR upload: number of fatal error exceeds the limit 3
2008-01-17 14:14:21 Thread-2720 ERROR upload: agent will shutdown now
2008-01-17 14:14:21 Thread-2720 ERROR : Signalled to Exit with status 55
2008-01-17 14:14:21 Thread-4032 ERROR main: EMAgent abnormal terminating

‘Solution’

The status of the Agent is unclear, first thing to do is check the SYSMAN.MGMT$TARGET table, if your agent shows up here try to remove it using th OMS frontend or use methods below.

[IAWE239@PGRIDC03.AH.NL:AHC536]
SQL> exec sysman.mgmt_admin.cleanup_agent('<agent_hostname>:3872')
BEGIN sysman.mgmt_admin.cleanup_agent('<agent_hostname>:3872'); END;
 
*
ERROR at line 1:
ORA-20206: Target does not exist: <agent_hostname>:3872
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 788
ORA-06512: at line 1
 
[IAWE239@PGRIDC03.AH.NL:AHC536]
SQL> exec sysman.mgmt_admin.delete_target_internal('<agent_hostname>:3872', 'oracle_emd');
BEGIN sysman.mgmt_admin.delete_target_internal('<agent_hostname>:3872', 'oracle_emd'); END;
 
*
ERROR at line 1:
ORA-20206:  The target does not exist or may have already been deleted:  Target name = <agent_hostname>:3872 
      and Target type = oracle_emd
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 645
ORA-06512: at line 1

Now check the records in SYSMAN.MGMT_TARGETS_DELETE

[IAWE239-SYSMAN@PGRIDC03.AH.NL:AHC536]
SQL> SELECT target_name, delete_complete_time, target_guid
  2  FROM sysman.mgmt_targets_delete
  3  WHERE target_type = 'oracle_emd';
 
TARGET_NAME                      DELETE_COM TARGET_GUID
-------------------------------- ---------- --------------------------------
<agent_hostname>:3872            17-01-2008 CACA6898BA9051954366482B01EBE9CB
 

Oooops, my agent should have been correcly removed…

Really, really, REALLY remove agent from OMS repository

Sick and tired of this I decided ‘solution’ below: select/remove all recors with GUID of my Agent.

DEFINE TARGET_GUID = "CACA6898BA9051954366482B01EBE9CB"
 
SELECT   'SELECT COUNT(*) FROM ' || table_name || ' WHERE target_guid = HEXTORAW(''&TARGET_GUID'');'
-- SELECT   'DELETE FROM ' || table_name || ' WHERE target_guid = HEXTORAW(''&TARGET_GUID'');'
FROM   ( SELECT table_name FROM dba_tab_columns NATURAL JOIN dba_tables WHERE column_name = 'TARGET_GUID')
/
 
SQL> REM Or check ALL raw columns

SELECT 'SELECT COUNT(*) FROM ' || table_name || ' WHERE ' || column_name || ' = HEXTORAW(''&TARGET_GUID'');'
  FROM ( 
       SELECT table_name, column_name 
         FROM dba_tab_columns NATURAL JOIN dba_tables 
        WHERE owner = 'SYSMAN' 
          AND data_type = 'RAW' 
       )
/
 

In my environment there was a mismatch between the MGMT_TARGETS and the MGMT_AGENT_SEC_INFO:

SQL> SELECT COUNT(*) FROM mgmt_targets WHERE target_type = 'oracle_emd' ;
  COUNT(*)
----------
        30
 
SQL> SELECT COUNT(*) FROM mgmt_agent_sec_info ;
  COUNT(*)
----------
        32
 
SQL> SELECT * FROM mgmt_agent_sec_info WHERE target_guid = HEXTORAW( '&TARGET_GUID');
TARGET_GUID                      AGENT_KEY
-------------------------------- ----------------------------------------
CACA6898BA9051954366482B01EBE9CB ¿¿¿¿.¿¿¿9¿2¿¿KM(¿<+¿¿H¿¿¿¿!¿E>¿¿

(Windows Only?) ‘emctl clearstate agent’ does NOT remove XML files from $ORACLE_HOME/sysman/emd/upload, remove manually from the OS to prevent ‘Failed to upload file A0000001.xml’ and ‘FxferSend: received fatal error in header from repository’ errors.

Needless to say, use at your own risk; also see oracle forum thread delete old agent