Thanks to this blog I found script below wich displays the various sysaux components and their sizes.

SQL> col "Space (M)" for 999,999.99  
SQL> 
SQL> SELECT 
  2    occupant_name,  
  3    round( space_usage_kbytes/1024) "Space (M)",  
  4    schema_name, 
  5    move_procedure
  6  FROM 
  7    v$sysaux_occupants  
  8  ORDER BY 
  9    1  
 10  /
OCCUPANT_NAME          Space (M) SCHEMA_NAME          MOVE_PROCEDURE
-------------------- ----------- -------------------- ----------------------------------------
AO                          1.00 SYS                  DBMS_AW.MOVE_AWMETA
EM                           .00 SYSMAN               emd_maintenance.move_em_tblspc
JOB_SCHEDULER               1.00 SYS
LOGMNR                      6.00 SYSTEM               SYS.DBMS_LOGMNR_D.SET_TABLESPACE
LOGSTDBY                    1.00 SYSTEM               SYS.DBMS_LOGSTDBY.SET_TABLESPACE
ODM                          .00 DMSYS                MOVE_ODM
ORDIM                        .00 ORDSYS
ORDIM/PLUGINS                .00 ORDPLUGINS
ORDIM/SQLMM                  .00 SI_INFORMTN_SCHEMA
SDO                          .00 MDSYS                MDSYS.MOVE_SDO
SM/ADVISOR                 17.00 SYS
SM/AWR                  3,748.00 SYS
SM/OPTSTAT                294.00 SYS
SM/OTHER                   52.00 SYS
STATSPACK                    .00 PERFSTAT
STREAMS                      .00 SYS
TEXT                         .00 CTXSYS               DRI_MOVE_CTXSYS
ULTRASEARCH                  .00 WKSYS                MOVE_WK
WM                           .00 WMSYS                DBMS_WM.move_proc
XDB                          .00 XDB                  XDB.DBMS_XDB.MOVEXDB_TABLESPACE
XSAMD                        .00 OLAPSYS              DBMS_AMD.Move_OLAP_Catalog
XSOQHIST                    1.00 SYS                  DBMS_XSOQ.OlapiMoveProc
 
22 rows selected.
 
SQL>   
SQL> 

Next thing I retrieve the oldest and latest AWR snapshot, turns out to be 2007…

SQL> SELECT 
  2    snap_id, begin_interval_time, end_interval_time 
  3  FROM 
  4    SYS.WRM$_SNAPSHOT 
  5  WHERE 
  6    snap_id = ( SELECT MIN (snap_id) FROM SYS.WRM$_SNAPSHOT)
  7  UNION 
  8  SELECT 
  9    snap_id, begin_interval_time, end_interval_time 
 10  FROM 
 11    SYS.WRM$_SNAPSHOT 
 12  WHERE 
 13    snap_id = ( SELECT MAX (snap_id) FROM SYS.WRM$_SNAPSHOT)
 14  /
   SNAP_ID BEGIN_INTERVAL_TIME   END_INTERVAL_TIME
---------- --------------------- ---------------------
      7556 15-09-07 08:44:42,810 15-09-07 11:01:11,950
     23698 18-08-09 11:00:41,612 18-08-09 12:00:53,074
 

I prefer SYS.WRM$_SNAPSHOT instead of DBA_HIST_SNAPSHOT because I saw strange results in (Release 10.1.0.5).

Now use the dbms_workload_repository package to remove the AWR snapshots.

BEGIN                                                               
  dbms_workload_repository.drop_snapshot_range(low_snap_id => 7556, high_snap_id=>15000);                                         
END;
/

Speed up ‘removal’ of old AWR reports

@#$%^&*()_ removing the entries takes ages and fails on undo errors … Metalink note Doc ID: 852028.1 states that I can safely remove the AWR metadata tables and recreate them.

If none of the above suits as everything is set proper then consider clean up and rebuild AWR repository to clear all the space.

SQL> connect / as sysdba 
SQL> @?/rdbms/admin/catnoawr.sql 
SQL> @?/rdbms/admin/catawrtb.sql 

I decided the try to restore the database with T-SQL, with T-SQL the restore completed successfully! Another advantage of using T-SQL is that T-SQL gave more logging. This logging shows me that the BAK file (delivered from a third party) is not a SQL Server 2005 backup but presumably a SQL Server 2000 database backup …

DECLARE
  @BackupFile varchar(8000),
  @sql varchar(8000)
  SET @BackupFile = 'E:\DB2000.BAK'
  SET @sql = 'RESTORE DATABASE ADB01 FROM DISK = ''' + @backupfile + ''' WITH FILE = 1,  
    MOVE N''DB2000_dat'' TO N''E:\mssql\data\ADB01\ADB01_01.mdf'',  
    MOVE N''DB2000_log'' TO N''F:\mssql\tlog\ADB01\ADB01_01.ldf'',  
    NOUNLOAD,  REPLACE,  STATS = 10, RECOVERY'
  exec (@sql)
go
 
10 percent processed.
20 percent processed.
30 percent processed.
40 percent processed.
50 percent processed.
60 percent processed.
70 percent processed.
80 percent processed.
90 percent processed.
100 percent processed.
Processed 2792 pages for database 'ADB01', file 'DB2000_dat' on file 1.
Processed 1 pages for database 'ADB01', file 'DB2000_log' on file 1.
Converting database 'ADB01' from version 539 to the current version 611.
Database 'ADB01' running the upgrade step from version 539 to version 551.
Database 'ADB01' running the upgrade step from version 551 to version 552.
Database 'ADB01' running the upgrade step from version 552 to version 553.
Database 'ADB01' running the upgrade step from version 553 to version 554.
Database 'ADB01' running the upgrade step from version 554 to version 589.
Database 'ADB01' running the upgrade step from version 589 to version 590.
Database 'ADB01' running the upgrade step from version 590 to version 593.
Database 'ADB01' running the upgrade step from version 593 to version 597.
Database 'ADB01' running the upgrade step from version 597 to version 604.
Database 'ADB01' running the upgrade step from version 604 to version 605.
Database 'ADB01' running the upgrade step from version 605 to version 606.
Database 'ADB01' running the upgrade step from version 606 to version 607.
Database 'ADB01' running the upgrade step from version 607 to version 608.
Database 'ADB01' running the upgrade step from version 608 to version 609.
Database 'ADB01' running the upgrade step from version 609 to version 610.
Database 'ADB01' running the upgrade step from version 610 to version 611.
RESTORE DATABASE successfully processed 2793 pages in 1.513 seconds (15.118 MB/sec).
 

Software/hardware :****
Problem & TS :
A job hangs in ****,
Job name: ****

User would like to be called back on the phone number provided if more information is needed.
User said he cannot explain it to me because its too difficult to understand and i wouldnt understand anything anyways.
**** production DB issue

copy from call ticket 600-01-12184003 with incident ticket 600-02-6637739

hey team,
talked with **** **** about this problem and she assigned it to your team.
Can you please take a look at it?
Thank you

Here’s the scenario:

1. You have an Alert that your tablespace is ‘full’,
2. You enlarge the tablespace,
3. The ‘Tablespace Full Alert’ is still in your dashbord for the next 24 (x) hours since you’ve configured this metric to be collected once every 24 (x) hours,
4. You want however to ‘remove’ (clear) this Alert. In order to achive this you will have to to re-evaluate this metric collection …

Here is how you do it (for anything storage related on your database server):

$ emctl control agent runCollection <hostname>:host host_storage

See the Oracle docs, 2.7.6 Reevaluating Metric Collections. It shows you how to retrieve the correct collection name for any metric.

You will find yourself grep-ing through ‘$ORACLE_HOME/sysman/admin/metadata/*xml’ and ‘$ORACLE_HOME/sysman/admin/default_collection/*xml’ …

Specify non default tcp ports

C:\>osql -Stcp:<hostname>,<portname>[\<instance_name>] -E
1>

See http://support.microsoft.com/kb/313295

To increase the idle time for the OMS frontend sessions edit the ‘oracle.sysman.eml.maxInactiveTime’ configuration parameter your Oracle Management Server’s emoms.properties file. The parameter value unit is in minutes and the default is 15 minutes which is a bit short I suppose. I have set it to 4 hours as you can see in the example below.

$ cat $ORACLE_HOME/sysman/config/emoms.properties | grep maxInactiveTime
oracle.sysman.eml.maxInactiveTime=240
$
 

For more info see the ‘Enterprise Manager Advanced Configuration‘ book

execute sp_dboption

1> exec sp_dboption 'testdb01', 'single user', 'FALSE';
2> go
Msg 5064, Level 16, State 1, Server REMIDIAN01, Line 1
Changes to the state or options of database 'testdb01' cannot be made at this
time. The database is in single-user mode, and a user is currently connected to
it.
Msg 5069, Level 16, State 1, Server REMIDIAN01, Line 1
ALTER DATABASE statement failed.
sp_dboption command failed.

Now I can use TSQL below to find all the database sesions connected to this database.

TSQL script to retrieve client sessions per database.

select d.name, d.dbid, spid, login_time, nt_domain, nt_username, loginame
  from sysprocesses p inner join sysdatabases d on p.dbid = d.dbid
 where d.name = 'testdb01'
go
 

Kill the particular session(s) with the ;kill’ command.

1> kill 51
2> go
1>

Now I can ‘remove’ the database from Sinlge user mode.

1> exec sp_dboption 'testdb01', 'single user', 'FALSE'
2> go
1>

I hate babysitting multiple sessions having to create multiple SQL Scripts to spread the workload (it’s on a 20+ CPUS’s database server) So I wrote this PL/SQL ‘application’ in a hurry to enable starting multiple databases sessions in nohup, each session picks up the next SQL statement ‘available’ (WHERE endtime IS NULL). It will save me some sleep.

Setup

The table holding my SQL statements

CREATE TABLE batchsql
 ( batchid              VARCHAR2(64)    NOT NULL
 , sqlid                INTEGER         NOT NULL UNIQUE
 , sqltext              varchar2(4000)  NOT NULL
 , starttime            DATE
 , endtime              DATE
 , sqlcode              INTEGER
 , sqlerrm              VARCHAR2(256)
 , error_stack          VARCHAR2(4000)
 , error_backtrace      VARCHAR2(4000)
 );

would be nicer to make the ‘sqlid’ column a compisone unique key with the batchid (harmless off course).

The anonymous PL/SQL block that will loop and execute these statements

DECLARE
  v_batchid  batchsql.batchid%TYPE;
  i_sqlid    batchsql.sqlid%TYPE;
  v_sqltext  batchsql.sqltext%TYPE;
 
  i_sqlcode  PLS_INTEGER;
  v_sqlerrm  VARCHAR2(256);
 
  i          PLS_INTEGER;
 
BEGIN
 
  v_batchid := 'DB01_ANALYZE_20080118';
  i         := 0;
 
  LOOP
 
    -- get statement that nobody is working on
    SELECT sqlid,
           sqltext
      INTO i_sqlid,
           v_sqltext
      FROM batchsql
     WHERE batchid = v_batchid
       AND starttime IS NULL
       AND rownum = 1;
 
    i := i +1;
 
    -- open new block for 'sqltext' statement
    BEGIN
 
      -- first set the starttime, (lock statement) to make sure 
      -- no other session will pick up the same statement
      UPDATE batchsql
         SET starttime = SYSDATE
       WHERE sqlid = i_sqlid;
 
      COMMIT;
 
      -- execute the statement
      EXECUTE IMMEDIATE v_sqltext;
 
      -- everything went well, register endtime and commit
      UPDATE batchsql
         SET endtime = SYSDATE
       WHERE sqlid = i_sqlid;
 
      COMMIT;
 
    EXCEPTION
      WHEN OTHERS THEN
 
        -- error occured, register endtime and errors
        i_sqlcode := SQLCODE;
        v_sqlerrm := SQLERRM;
 
        UPDATE batchsql
           SET endtime      = SYSDATE
             , sqlcode      = i_sqlcode
             , sqlerrm      = v_sqlerrm
             , error_stack  = DBMS_UTILITY.FORMAT_ERROR_STACK()
-- Uncomment 'FORMAT_ERROR_BACKTRACE()' for pre 10G
-- , error_backtrace = DBMS_UTILITY.FORMAT_ERROR_BACKTRACE()
         WHERE sqlid = i_sqlid;
 
    COMMIT;
    END;
 
  END LOOP;
 
EXCEPTION
  WHEN NO_DATA_FOUND THEN
 
    IF i = 0 THEN
      DBMS_OUTPUT.PUT_LINE ( 'No SQL Statements found for batch ''' || v_batchid || '''');
    ELSE
      DBMS_OUTPUT.PUT_LINE ( 'Carried out ' || TO_CHAR(i) || ' SQL statements');
    END IF;
 
    DBMS_OUTPUT.PUT_LINE ( 'Time is ''' || to_CHAR(SYSDATE, 'DD-MON-YY hh24:mi:ss') || '''');
 
END;
/
 

Example

Suppose I have to analyze a bunch of tables, I will first have to fill my ‘batchsql’ table with all the SQL Statements that belong to this batch, I identify the batch to ‘DB01_ANALYZE_20080118′.

Generate some SQL statements and insert them in the batchsql table

SQL> DEFINE BATCHID  = "''DB01_ANALYZE_20080118''"
SQL> DEFINE QUOTE    = "''"
SQL> SELECT 'INSERT INTO batchsql 
  2          ( batchid, sqlid, sqltext)
  3          VALUES
  4          ( &BATCHID,' || ROWNUM || ', &QUOTE.ANALYZE TABLE ' || table_name || ' COMPUTE STATISTICS&QUOTE);'
  5     FROM user_tables
  6    WHERE rownum < 5
  7  /
'INSERTINTOBATCHSQL(BATCHID,SQLID,SQLTEXT)VALUES(''DB01_ANALYZE_20080118'','||ROWNUM||',''ANALYZETABLE'||TABLE_
---------------------------------------------------------------------------------------------------------------
INSERT INTO batchsql
        ( batchid, sqlid, sqltext)
        VALUES
        ( 'DB01_ANALYZE_20080118',1, 'ANALYZE TABLE BATCHSQL COMPUTE STATISTICS');
 
INSERT INTO batchsql
        ( batchid, sqlid, sqltext)
        VALUES
        ( 'DB01_ANALYZE_20080118',2, 'ANALYZE TABLE DEMO COMPUTE STATISTICS');
 
SQL> 

Do test the error handling I insert one bogus statement that will generate an error.

SQL> INSERT INTO batchsql
  2   ( batchid, sqlid, sqltext)
  3   VALUES
  4   ( 'DB01_ANALYZE_20080118', 5, 'HELLO WORLD!');
 
1 row created.
 
SQL> 
SQL> 

Now look at the contents of the batchsql table, we see three statements for the ‘DB01_ANALYZE_20080118′ batch.

SQL> select batchid, sqlid, sqltext, starttime, endtime, sqlcode, sqlerrm from batchsql;
BATCHID                SQLID SQLTEXT                                    STARTTIME ENDTIME      SQLCODE SQLERRM
--------------------- ------ ------------------------------------------ --------- --------- ---------- ---------
DB01_ANALYZE_20080118      5 HELLO WORLD!
DB01_ANALYZE_20080118      1 ANALYZE TABLE BATCHSQL COMPUTE STATISTICS
DB01_ANALYZE_20080118      2 ANALYZE TABLE DEMO COMPUTE STATISTICS
 
SQL> 

Start a session that will start executing statements found in the batchsql table for batch ‘DB01_ANALYZE_20080118′:

SQL> ed
Wrote file afiedt.buf
 
  1  DECLARE
  2    v_batchid  batchsql.batchid%TYPE;
  3    i_sqlid    batchsql.sqlid%TYPE;
  4    v_sqltext  batchsql.sqltext%TYPE;
  5    i_sqlcode  PLS_INTEGER;
  6    v_sqlerrm  VARCHAR2(256);
  7    i          PLS_INTEGER;
  8  BEGIN
  9    v_batchid := 'DB01_ANALYZE_20080118';
 10    i     := 0;
 11    LOOP
 12      -- get statement that nobody is working on
 13      SELECT sqlid,
 14             sqltext
 15        INTO i_sqlid,
 16             v_sqltext
 17        FROM batchsql
 18       WHERE batchid = v_batchid
 19         AND starttime IS NULL
 20         AND rownum = 1;
 21      i := i +1;
 22      -- open new block for 'sqltext' statement
 23      BEGIN
 24        -- first set the starttime, (lock statement) to make sure
 25        -- no other session will pick up the same statement
 26        UPDATE batchsql
 27           SET starttime = SYSDATE
 28         WHERE sqlid = i_sqlid;
 29        COMMIT;
 30        -- execute the statement
 31        EXECUTE IMMEDIATE v_sqltext;
 32        -- everything went well, register endtime and commit
 33        UPDATE batchsql
 34           SET endtime = SYSDATE
 35         WHERE sqlid = i_sqlid;
 36        COMMIT;
 37      EXCEPTION
 38        WHEN OTHERS THEN
 39          -- error occured, register endtime and errors
 40          i_sqlcode := SQLCODE;
 41          v_sqlerrm := SQLERRM;
 42          UPDATE batchsql
 43             SET endtime      = SYSDATE
 44               , sqlcode      = i_sqlcode
 45               , sqlerrm      = v_sqlerrm
 46               , error_stack  = DBMS_UTILITY.FORMAT_ERROR_STACK()
 47  -- Uncomment 'FORMAT_ERROR_BACKTRACE()' for pre 10G
 48  -- , error_backtrace = DBMS_UTILITY.FORMAT_ERROR_BACKTRACE()
 49           WHERE sqlid = i_sqlid;
 50      COMMIT;
 51      END;
 52    END LOOP;
 53  EXCEPTION
 54    WHEN NO_DATA_FOUND THEN
 55      IF i = 0 THEN
 56        DBMS_OUTPUT.PUT_LINE ( 'No SQL Statements found for batch ''' || v_batchid || '''');
 57      ELSE
 58        DBMS_OUTPUT.PUT_LINE ( 'Carried out ' || TO_CHAR(i) || ' SQL statements');
 59      END IF;
 60      DBMS_OUTPUT.PUT_LINE ( 'Time is ''' || to_CHAR(SYSDATE, 'DD-MON-YY hh24:mi:ss') || '''');
 61* END;
 62  /
Carried out 3 SQL statements
Time is '18-JAN-08 18:37:01'
 
PL/SQL procedure successfully completed.
 
SQL> REM and again
SQL> /
No SQL Statements found for batch 'DB01_ANALYZE_20080118'
Time is '18-JAN-08 18:37:20'
 
PL/SQL procedure successfully completed.
 
SQL> 
SQL> 

Pretty straigh forward, not?

• The first run the starttime column is NULL (for every row),
• The statements are executed, both the starttime and the endtime are updated at every single execution
• The second run find no rows where starttime IS NULL and will not execute any statement at all.

See the log results.

SQL> select batchid, sqlid, sqltext, starttime, endtime, sqlcode, sqlerrm from batchsql;
BATCHID                SQLID SQLTEXT                                    STARTTIME ENDTIME   SQLCODE SQLERRM
--------------------- ------ ------------------------------------------ --------- --------- ------- --------------------
DB01_ANALYZE_20080118      5 HELLO WORLD!                               18-JAN-08 18-JAN-08    -900 ORA-00900: invalid S
                                                                                                    QL statement
 
DB01_ANALYZE_20080118      1 ANALYZE TABLE BATCHSQL COMPUTE STATISTICS  18-JAN-08 18-JAN-08
DB01_ANALYZE_20080118      2 ANALYZE TABLE DEMO COMPUTE STATISTICS      18-JAN-08 18-JAN-08
 
SQL> 

Of course, you can add extra functionalities as many as you would like, store the sessionid for every SQL statement, have multiple runs for a batch to reuse statements, etc etc. Oh and of course I should rewrite it to a SQL Procedure or Package.

The babysitting is now minimized to running this query:

 
col batch               for 9999
col completed           for 9999
col todo                for 9999
col activesessions      for 9999999     head "ACTIVE|SESSIONS"
col errors              for 99999
col elapsed_seconds     for 9999999     head "ELAPSED|SECONDS"
col avg_execution_time  for 9999999     head "AVG|EXECUTION|TIME"
col percentage_complete for 9999999     head "PERCENTAGE|COMPLETE"
 
-- as I only have one batch loaded I don't need to filter on the batchid
select batch.c                                                    batch,
       completed.c                                                completed,
       batch.c - completed.c                                      todo,
       to_char( times.started, 'DD-MM-HH hh24:mi')                started,
       to_char( times.lastcompleted, 'DD-MM-HH hh24:mi')          lastcompleted,
       round( completed.c / batch.c * 100, 1)                     percentage_complete,
       activesessions.c                                           activesessions,
       errors.c                                                   errors,
       ( sysdate - times.started) *24*60*60                       elapsed_seconds,
       trunc( ( sysdate - times.started)*24*60*60 / completed.c)  avg_execution_time,
       to_char( sysdate +
         ( batch.c - completed.c) *
         ( (sysdate - times.started) / completed.c)
         , 'DD-MM-HH hh24:mi')                                    est_end_time
 from ( select count(*) as c from batchsql )    batch
    , ( select count(*) as c from batchsql
         where endtime is not null)             completed
    , ( select count(*) as c
          from batchsql
         where starttime is not null
           and endtime is null)                 activesessions
    , ( select count(*) as c
          from batchsql
         where sqlcode is not null )            errors
   , ( select min(starttime) started,
              max(starttime) lastcompleted
          from batchsql )                       times
/
 

                                                                                              AVG
                                                    PERCENTAGE   ACTIVE         ELAPSED EXECUTION
BATCH COMPLETED  TODO STARTED        LASTCOMPLETED    COMPLETE SESSIONS ERRORS  SECONDS      TIME EST_END_TIME
----- --------- ----- -------------- -------------- ---------- -------- ------ -------- --------- --------------
 1524       492  1032 18-01-06 18:45 18-01-10 22:01         32        6      1    11756        23 19-01-04 04:52
 
SQL> 
SQL> 
SQL> select*from batchsql where sqlcode is not null;
BATCHID                                                               SQLID
---------------------------------------------------------------- ----------
SQLTEXT
-----------------------------------------------------------------------------------------------------------------------
STARTTIME  ENDTIME       SQLCODE SQLERRM
---------- ---------- ---------- --------------------------------------------------------------------------------------
ERROR_STACK
-----------------------------------------------------------------------------------------------------------------------
ERROR_BACKTRACE
-----------------------------------------------------------------------------------------------------------------------
DB01_ANALYZE_20080118                                                84
ANALYZE TABLE ********
18-01-2008 18-01-2008      -1555 ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2$" too small
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2$" too small
 
SQL> 
 

UPDATE 2010-05-10T11:04:21+00:00

Make sqlid non unique and add trigger to generate sqlid’s.

CREATE TABLE batchsql
 ( batchid              VARCHAR2(64)    NOT NULL
 , sqlid                INTEGER         NOT NULL
 , sqltext              varchar2(4000)  NOT NULL
 , starttime            DATE
 , endtime              DATE
 , sqlcode              INTEGER
 , sqlerrm              VARCHAR2(256)
 , error_stack          VARCHAR2(4000)
 , error_backtrace      VARCHAR2(4000)
 );
 
CREATE SEQUENCE s_batchsql;
 
CREATE OR REPLACE TRIGGER t_batchsql
BEFORE INSERT ON batchsql
FOR EACH ROW
WHEN (new.sqlid IS NULL)
BEGIN
  SELECT s_batchsql.NEXTVAL
  INTO   :new.sqlid
  FROM   dual;
END;
/

Create script to execute statements from batch table

export ORAENV_ASK=NO;
export ORACLE_SID=sfsse001;
/usr/local/bin/oraenv
 
sqlplus -silent '/ as sysdba' <<EOF
 
column d new_value d
select to_char( sysdate, 'YYYYMMDD-HH24MISS') d from dual;
 
set serveroutput on
 
spool batch_&d.log
 
@r.sql
 
spool off
 
EOF
 

$ nohup r.sh &
 

Scenario

• Removed (Windows) Agent from Grid. (With SQL Server Management plugin)
• Tried to restart this SAME agent, errors below occur.

# $ORACLE_HOME/sysman/log/emagent.log contains
 
2008-01-17 14:13:52 Thread-4032 EMAgent started successfully (00702)
2008-01-17 14:14:21 Thread-2720 <Upload Manager> Unsuccessful Upload attempts for XML file exceeds specified limit=3, 
      Agent will shutdown (00851)
2008-01-17 14:14:21 Thread-4032 EMAgent abnormal terminating (00704)

# $ORACLE_HOME/sysman/log/emdctl.trc contains
 
2008-01-17 14:13:32 Thread-752 WARN  http: snmehl_connect: connect failed to (<agent_hostname>:3872): 
      No connection could be made because the target machine actively refused it.
 (error = 10061)
2008-01-17 14:13:32 Thread-752 ERROR main: nmectla_agentctl: Error connecting to 
      https://<agent_hostname>:3872/emd/main/. Returning status code 1
2008-01-17 14:14:23 Thread-172 WARN  http: snmehl_connect: connect failed to 
      (<agent_hostname>:3872): 
      No connection could be made because the target machine actively refused it. (error = 10061)
2008-01-17 14:14:23 Thread-172 ERROR main: nmectla_agentctl: Error connecting to 
      https://<agent_hostname>:3872/emd/main/. Returning status code 1
 

# $ORACLE_HOME/sysman/log/emagent.trc contains
 
2008-01-17 14:14:18 Thread-192 WARN  TargetManager: Regenerating all Metadata
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: In getTypeAndNameForHostTarget
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: getTypeAndNameForHostTarget found host, <agent_hostname>
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: In getTypeAndNameForHostTarget
2008-01-17 14:14:18 Thread-192 DEBUG TargetManager: getTypeAndNameForHostTarget found host, <agent_hostname>
2008-01-17 14:14:18 Thread-192 WARN  collector: Regenerating all DefaultColls
2008-01-17 14:14:18 Thread-192 ERROR upload: Failed to upload file A0000001.xml, ret = -6
2008-01-17 14:14:19 Thread-192 WARN  upload: FxferSend: received fatal error in header from repository: 
 
https://<oms_hostname>:1159/em/upload
 
FATAL_ERROR::500|ORA-20206: Target does not exist:  Agent does not exist for https://<agent_hostname>:3872/emd/main/
ORA-06512: at "SYSMAN.EMD_LOADER", line 1795
ORA-06512: at line 1
 
.
.
.
 
2008-01-17 14:14:21 Thread-2720 ERROR upload: number of fatal error exceeds the limit 3
2008-01-17 14:14:21 Thread-2720 ERROR upload: agent will shutdown now
2008-01-17 14:14:21 Thread-2720 ERROR : Signalled to Exit with status 55
2008-01-17 14:14:21 Thread-4032 ERROR main: EMAgent abnormal terminating

‘Solution’

The status of the Agent is unclear, first thing to do is check the SYSMAN.MGMT$TARGET table, if your agent shows up here try to remove it using th OMS frontend or use methods below.

[IAWE239@PGRIDC03.AH.NL:AHC536]
SQL> exec sysman.mgmt_admin.cleanup_agent('<agent_hostname>:3872')
BEGIN sysman.mgmt_admin.cleanup_agent('<agent_hostname>:3872'); END;
 
*
ERROR at line 1:
ORA-20206: Target does not exist: <agent_hostname>:3872
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 788
ORA-06512: at line 1
 
[IAWE239@PGRIDC03.AH.NL:AHC536]
SQL> exec sysman.mgmt_admin.delete_target_internal('<agent_hostname>:3872', 'oracle_emd');
BEGIN sysman.mgmt_admin.delete_target_internal('<agent_hostname>:3872', 'oracle_emd'); END;
 
*
ERROR at line 1:
ORA-20206:  The target does not exist or may have already been deleted:  Target name = <agent_hostname>:3872 
      and Target type = oracle_emd
ORA-06512: at "SYSMAN.MGMT_ADMIN", line 645
ORA-06512: at line 1

Now check the records in SYSMAN.MGMT_TARGETS_DELETE

[IAWE239-SYSMAN@PGRIDC03.AH.NL:AHC536]
SQL> SELECT target_name, delete_complete_time, target_guid
  2  FROM sysman.mgmt_targets_delete
  3  WHERE target_type = 'oracle_emd';
 
TARGET_NAME                      DELETE_COM TARGET_GUID
-------------------------------- ---------- --------------------------------
<agent_hostname>:3872            17-01-2008 CACA6898BA9051954366482B01EBE9CB
 

Oooops, my agent should have been correcly removed…

Really, really, REALLY remove agent from OMS repository

Sick and tired of this I decided ‘solution’ below: select/remove all recors with GUID of my Agent.

DEFINE TARGET_GUID = "CACA6898BA9051954366482B01EBE9CB"
 
SELECT   'SELECT COUNT(*) FROM ' || table_name || ' WHERE target_guid = HEXTORAW(''&TARGET_GUID'');'
-- SELECT   'DELETE FROM ' || table_name || ' WHERE target_guid = HEXTORAW(''&TARGET_GUID'');'
FROM   ( SELECT table_name FROM dba_tab_columns NATURAL JOIN dba_tables WHERE column_name = 'TARGET_GUID')
/
 
SQL> REM Or check ALL raw columns

SELECT 'SELECT COUNT(*) FROM ' || table_name || ' WHERE ' || column_name || ' = HEXTORAW(''&TARGET_GUID'');'
  FROM ( 
       SELECT table_name, column_name 
         FROM dba_tab_columns NATURAL JOIN dba_tables 
        WHERE owner = 'SYSMAN' 
          AND data_type = 'RAW' 
       )
/
 

In my environment there was a mismatch between the MGMT_TARGETS and the MGMT_AGENT_SEC_INFO:

SQL> SELECT COUNT(*) FROM mgmt_targets WHERE target_type = 'oracle_emd' ;
  COUNT(*)
----------
        30
 
SQL> SELECT COUNT(*) FROM mgmt_agent_sec_info ;
  COUNT(*)
----------
        32
 
SQL> SELECT * FROM mgmt_agent_sec_info WHERE target_guid = HEXTORAW( '&TARGET_GUID');
TARGET_GUID                      AGENT_KEY
-------------------------------- ----------------------------------------
CACA6898BA9051954366482B01EBE9CB ¿¿¿¿.¿¿¿9¿2¿¿KM(¿<+¿¿H¿¿¿¿!¿E>¿¿

(Windows Only?) ‘emctl clearstate agent’ does NOT remove XML files from $ORACLE_HOME/sysman/emd/upload, remove manually from the OS to prevent ‘Failed to upload file A0000001.xml’ and ‘FxferSend: received fatal error in header from repository’ errors.

Needless to say, use at your own risk; also see oracle forum thread delete old agent

For some obscure reason Oracle is very reluctant in providing the information that is described here below. Search Metalink and you will find no entries… The information below is from dbasupport.com and helps me to clear alerts that the Agent does NOT clear for me.

sysman.em_severity

[REMI@DB01.REMIDIAN.COM]
SQL>  @desc sysman.em_severity
.
.
PROCEDURE DELETE_CURRENT_SEVERITY
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 P_TARGET_GUID                  RAW                     IN
 P_METRIC_GUID                  RAW                     IN
 P_KEY_VALUE                    VARCHAR2                IN
.
.
 

Generate SQL to remove alerts from specific targets

select t.target_name
,      t.target_type
,      collection_timestamp
,      message
,      'exec em_severity.delete_current_severity(''' ||
           t.target_guid || ''',''' ||
           metric_guid || ''',''' ||
           key_value || ''')' em_severity
from   mgmt_targets t
inner join
       mgmt_current_severity s
on
       t.target_guid = s.target_guid
where
       target_name like '&TARGET'

Remove the Alert

[REMI@DB01.REMIDIAN.COM]
SQL> begin
  2  em_severity.delete_current_severity('3DFABFF28D9C283435DBD2491D35B92B','3E6F70DB22758B7B9756EF342180E7BB','UNDO');
  3  end;
  4  /
 
PL/SQL procedure successfully completed.
 
[REMI@DB01.REMIDIAN.COM]
SQL>