Kevin Kempf's Blog

March 29, 2011

Stuck in status pending

Filed under: 11g, Enterprise Manager — kkempf @ 4:00 pm

How annoying is this

After a brief network outage, 3 of my EM database targets were showing as status pending.  The databases were fine.  The agents were fine.

The quickfix

Go to the OS where the agent resides corresponding to the stuck database

  • emctl stop agent
  • cd $AGENT_HOME/sysman/emd
  • rm -rf state/*
  • rm -rf upload/*
  • rm -rf collection/*
  • rm lastupld.xml
  • rm agntstmp.txt
  • rm blackouts.xml
  • emctl clearstate agent
  • emctl start agent
  • emctl upload agent

From EM go to Setup->Agents

  • click on the now jacked up agent you just messed with
  • click on “Start Resynchronization”
  • after that, you may need to unblock the agent, as well, from the Setup->Agents screen

End result

After a minute or two, the agent unscrewed itself, and realized that the database was up as well.

March 28, 2011

Makes me wonder if I want the upgrade..

Filed under: 11g, Oracle, RMAN — kkempf @ 8:09 am

RMAN Repository Upgrade 101

I know, I’m the last DBA on the planet who still uses and RMAN repository.  Perhaps it’s somewhat of a throwback, but there are situations where having one will save time recovering a database, and since I have to keep a database running for Enterprise Manager anyways, I just stick it there with that database.  No extra maintenance, just extra insurance.  Regardless, I upgraded my EM database to and began receiving complaints when connecting to the catalog from an database:
connected to recovery catalog database
PL/SQL package RCAT_USER.DBMS_RCVCAT version in RCVCAT database is not current
PL/SQL package RCAT_USER.DBMS_RCVMAN version in RCVCAT database is not current

Been down this road before

I connect to the recovery catalog, and type upgrade catalog (twice).  Error.

error creating set_site_key_for_single_site_dbs
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-06004: ORACLE error from recovery catalog database: ORA-00001:
unique constraint (RMAN.SITE_TFATT_P) violated

Great.  I think every time I’ve tried to upgrade the catalog in 11g there’s been some kind of error.  This one was easy enough to find on My Oracle Support: Note 1221983.1 covers it well.  The fix?  Basically, manually edit $ORACLE_HOME/rdbms/admin/recover.bsq and correct their sql:

update site_tfatt set site_key = onesite_row.site_key where
tf_key in (select tf_key from df, dbinc
where dbinc.dbinc_key = df.dbinc_key
and dbinc.db_key = onesite_row.db_key);

update site_tfatt set site_key = onesite_row.site_key where
tf_key in (select tf_key from tf, dbinc
where dbinc.dbinc_key = tf.dbinc_key
and dbinc.db_key = onesite_row.db_key);

We all make typos right?

This error is well documented, and easy to find.  There’s nothing too alarming about it, except that this code could never have worked. Meaning, Oracle shipped code with their latest, flagship product, which somehow failed quality control in the grandest fashion.   It’s not some obscure combination of events which cause a blip in database performance or throw an error.  It’s code which could not ever work, could not ever be right.   It’s not some recent acquisition, some 3rd party conversion product which 7 people are interested in.  It’s RMAN!  It’s been around forever.   Makes me wonder, what else is wrong with this version of RCVCAT/RCVMAN?

March 24, 2011

Oracle VirtualBox

Filed under: Uncategorized — kkempf @ 9:23 am

Credit where credit is due

I thought I’d share a little gem I found about a year ago, which has served me so well that it merits a mention here.  I run 64-bit Ubuntu 10.04 LTS (Lucid Lynx) as my desktop (technically, notebook) OS at work.  There are times, however, when Windows is required:

  • Anytime IE is needed.  Whether because of some active-X (or whatever proprietary thing M$ is building these days),  IE-only check on a website, or IE “enhanced” features on Exchange webmail
  • Applications which are built for Windows only (Discoverer 11g administrator comes to mind)
  • GoTo Meeting web conferences

Wow this list got short!  It seemed just a few years ago, I was in Windows all the time for stuff.  Now I go weeks without firing up my VirtualBox.  I’m sure there’s more examples, I just can’t think of them now.

Regardless of why you might need a secondary OS, my point is that VirtualBox flawlessly delivers this capability to your PC for free.  Admittedly, that’s not really a word in Oracle’s vocabulary, so it’s no surprise that this product arose from their Sun acquisition, and it’s future might be in some doubt.  You can read about it and download it here.

One of the best things about this product is that, unlike most Oracle products, you can probably get it installed and mostly have it working without having to read a 50 page install document.  It’s really intuitive, and about the hardest part of the install was determining your disk/memory footprint and installing the client tools so it didn’t keep stealing my mouse.

What it does

Effectively, this program carves out a bit of disk (at your instruction) for the guest OS, and you allocate how much memory the new OS gets.  It then launches this Virtual machine in a window and you put in your install media and it behaves exactly like a new system.  When you are done, you have a windowed OS a click away.

Click start and the Wow is Now!

How is looks

Here’s a simple shot of my Windows 7 Desktop, running as a guest OS on my Ubuntu Linux Dekstop.  Full screen, you’d be hard pressed to know you’re even in a virtual machine.

Windows 7 running on Ubuntu

Don’t Forget

If you do set up this tool, enable virtualization technology (VT) on your CPU in the BIOS if it isn’t already by default.  Also, install the client tools into the guest OS so that the mouse doesn’t drive you crazy.  There are a few limitations you should be aware of, the biggest of which is that it cannot handle 3D graphics.

March 17, 2011

EBS Technology Stack Blog in transition

Filed under: 11i, Oracle — kkempf @ 10:30 am

I was rather saddened to find this in my inbox this morning.  I hope it’s restored/replaced/enabled soon.  My worst fear is that the corporate beast that is Oracle is beginning to close the doors on these kinds of “loose ends”.  I don’t know where else the consumers of their product can get honest, timely information information with no spin.   Thanks for your efforts, Steven!

EBS Technology Stack Blog in transition.

March 4, 2011

That can’t be a solution!

Filed under: 11i, R12 — kkempf @ 11:22 am

Ghetto IT Volume I

Dabbling in R12

I began an R12 upgrade in a sandbox environment a few weeks back, to see what had changed since the last time I’d done the technical upgrade.  I know R12 is inevitable, and with the release of 12.1.3 I wanted to see how painful the path was.  Turns out, it’s about the same as it always has been applying a major patch in Oracle E-Business Suite:

  1. Wait for a worker to fail.
  2. Search on the worker name in Metalink.  Open SR if you can’t find it to get a fix
  3. adctrl to restart the worker or use the not-so-secret option #8 to skip the worker
  4. pray the fix works.

Does Oracle operate in the real world?

The problem I hit this morning is described in Note 1072538.1: Basically, 3 PER workers failed because of a code problem:

When trying to upgrade from to 12.1.1, “Patch 6678700 – 12.1.1 : Upgrade Patch” has failed again with INVALID_SEE_COMMENT_IN_SOURCE against the following modules:

FAILED: file pereirsd.sql on worker 4.
FAILED: file peaeirsd.sql on worker 6.
FAILED: file peaeiasd.sql on worker 7.

The fix works, but the real-world application is beyond stupid.  They want a patch applied, to fix the error, so you can continue your R12 upgrade patch.  Right.  Because as soon as I start another patch, it asks to truncate all the existing patch data tables (in process) which exist because my current patch isn’t complete and is waiting for me to fix failed workers.  In other words, I’m 20 hours into an upgrade, and now I have to apply a fix-it patch and start over?  No thanks.

Ghetto IT

It may be un-PC, but we use this term in our department to refer to a cheap, easy method for getting things done which nobody wants to talk about or admit works.  In my case, I pulled the patch, dug into it to find only one piece of sql was in it.  I ran the sql separately in another sqlplus session as apps:

  • @/scratch/stage/r12/r12.1.1/8845395/per/patch/115/sql/peasgasddrop.sql

Then I restarted the 3 failed workers with adctrl and, magically, they completed normally and the patch went on (6678700, or the R12 upgrade patch).

Create a free website or blog at