Kevin Kempf's Blog

July 8, 2009

When EM Flips Out

Filed under: Bugs, Enterprise Manager, Oracle — Tags: , , , — kkempf @ 7:35 am

For no apparent reason, yesterday afternoon about 2pm Enterprise Manager went haywire. It was using 100% of 1 CPU, and causing considerable I/O (log writer). The truth is, it doesn’t take much to kill a 1-CPU virtual machine, but this was without precedent.
EM1EM2

After some digging, the offending SQL was:
SELECT execution_id
FROM MGMT_JOB_EXEC_SUMMARY e, MGMT_JOB j
WHERE e.job_id = j.job_id AND j.is_corrective_action = 0
AND status IN (5,4,3,18,8)
AND (CAST(SYS_EXTRACT_UTC(SYSTIMESTAMP) AS DATE) – e.start_time) > (:tf)
AND ROWNUM < 500 ORDER BY start_time desc

A few searches on Metalink, and I find DOC 839080.1, which tells me to apply patch 8517252. This patch applies against the EM Application Home, not the database. So I shut down the application, opatch in 8517252, but the post script @post_install_script.sql is hanging. Had to hard kill (kill -9) the 100% cpu process at the OS level and this cleared the lock contention. Finished installing the patch, fired up the EM application again, and all was good (this is in the graphs above, about 5pm).

Create a free website or blog at WordPress.com.