Kevin Kempf's Blog

November 12, 2010

Babysitting the Java Workflow Mailer in 11i

Filed under: 11i — kkempf @ 3:20 pm

Love, Hate

When it works, it’s great.  Users love being able to respond to workflows via the inbound java notification mailer in 11i.  They don’t have to log into the apps to do their work, and in our case (iProcurement) they can even approve a purchase from their Blackberry while out of the office.   Not exactly cutting edge technology, but it’s a visible, worthwhile enhancement which costs about nothing to implement.

The issue

The java workflow mailer gets hung up all the time.  On average, about once a week, it would hang or shut itself down.  As far as I can tell from the logs, many times this was because someone would reply directly to the inbound mailer via email, but not all the time.  Despite uploading log after log in SRs, and confirming over and over I had the latest patch set it, the only surefire way to fix this was to stop the workflow mailer service (container)  and restart the container.   This has two major problems.  First, it can be hung up without my knowledge, and people get annoyed when they approve a PO and it doesn’t show up.  Second, it’s a manual effort to check all the time, and if I’m out of the office I have to plan for someone to “check and see if the workflow notification mailer is running”.

My Fix

Unbeknown to me, the solution was right in front of me, and relatively painless.  Based on the assumption that 1) restarting the container always fixed it and 2) restarting it didn’t break anything, I came up with this fix.  Using the Service Components’ own internal settings, I told it to shut down every day at 8am and restart at 8:10am.  While I will freely admit that this is a Microsoft solution (reboot to fix), I just don’t care.  Since implementing it, I have not had a single problem with the mailer in nearly a month.

  1. From OAM, Navigate to Applications System:SID> Workflow
  2. Select the radio button for Workflow Notification Mailer and hit the Edit button
  3. Hit the Advanced button.
  4. Hit the Next button until you get to step 5 of 8 (Edit Workflow Mailer: Schedule Events: SID)
  5. Hit the button Add another Row
  6. Hit the flashlight and select stop, then accept
  7. Fill in the start date, and time.  For Interval, use 1440 (minutes) for every day
  8. Hit the flashlight and select start, then accept
  9. Fill in the same start date, and time from step 7 +10 minutes.  For Interval, use 1440 (minutes) for every day
  10. Hit Finish

Setting up a recurring stop & start

November 9, 2010

New Certify on Metalink, for anyone but 11i users

Filed under: Uncategorized — kkempf @ 8:43 am

New, Improved and Incomplete

I recently received an unsolicited email from Oracle regarding the new certify setup on the world’s slowest support site (My Oracle Support):

We have heard a lot of complaints concerning Certifications in My Oracle Support. Maybe they were from you? And sure enough it wasn’t going to be a simple fix. If you are someone who relies on the certify search engine, we want to hear from you! Check out the video previews of the new Certification system and send us feedback!

My first thought was that if you need a video to use it, that’s a bad sign.  But it’s actually rather intuitive from the get-go.  I give Oracle kudos for fixing this.  I’ve blogged before that since last November (when the worlds slowest support site went online), Certify has been wholly unusable.  Well, now it’s usable.  With a caveat: The one thing I really use it for, 11i certifications, is deliberately excluded from this.  Turns out R12 is in there, so the 4 or 5 customers on that version of the apps should be happy.  Those on 11i are stuck with the “classic” certify, which means you should just check Steven Chan’s blog.  You can check out the new certify team updates here.



Apps 11i? Never heard of it.


November 5, 2010

The 11i Patch Wizard

Filed under: 11i — kkempf @ 7:41 pm

Getting to the bottom of the “to do” list for 2010


Wizard.. or dunce?

I’d been meaning to take a look at the patch wizard for 11i for a long time.  It’s been one of those things which Steven Chan had repeatedly mentioned on his blog, yet sounded too good to be true.  This week, we completed a major year long project and I found some free time to take a closer look.
My curiosity had been especially peaked after seeing that there was a special patch for patch wizard which would tell you if your current environment was in line with th extended support patching baseline.   I’ll get on my soapbox for one moment and ask, again in vain, why Oracle elected to dish out a rag tag list of oddball patches required for extended support instead of just issuing as a neat bundle containing these.  I’m going to put my neck out and say the vast majority of Oracle Ebusiness Suite installs are going to fall into this 11i extended support model, and it would have been really  convenient for that vast majority of their Ebusiness customers.  Incidentally, Oracle annoyingly keeps the R12 adoption numbers pretty secret, with good reason.  Based on numerous conversations with consulting outfits  and even some “off the record” Oracle people, the uptake on R12 is slow (usually this is attributed to the global economic glut, partnered with no compelling business reason to move to R12).

To be honest, I didn’t even know how to run this wizard, or where it resided, so I began by watching an excellent overview video from Oracle.  It’s really straightforward; the only way my experience differed from the video was that (despite the fact that I have proxy server internet access) my OAM and EM can never seem to get out of our firewall.  This meant I had to go manually pull the 250mb  Infobundle file and land it on my 11i application server.

A quick overview on Setup

There’s really not a lot to be said here, the video explains pretty much everything you need to know.  I went into patch manager via OAM:



Then started at the patch wizard preferences screen:






Manually Install the Patch Information Bundle

I have no idea why, but OAM and EM 11g can’t figure out how to get to Metalink/MOS from my servers.  I give it the proxy name, validate a test successfully, but when it comes down to crunch time, they always say something to the effect of  “cannot find host”.  So I had to manually land the patch information bundle on my server (in my case, in /scratch/patches11i/prod/patchwizard).  If you find yourself in this situation, you can just download the file from the world’s slowest support site here.

Now, from the main Patch Wizard screen, click on the Tasks Icon next to Recommend/Analyze Patches, and from this screen, click on the “Upload patch information bundle (no analysis)” radio button, then hit OK.  

Upload Patch Information Bundle

 This should push the data from the bundle via concurrent request:

Concurrent request running in forms

Create a Recommendation

Once the patch information bundle is loaded, it’s a trivial feat to actually run the “recommendation” job.  From the main screen, once again select the Tasks icon next to Recommend/Analyze Patches and this time click the flashlight next to “Using Patch Filter” and select “New Code Levels” and hit accept.  This also runs a long time, it felt like hours, at least the first time.

Results, Take I

Here’s where it gets interesting.  The first time I received results, they were totally $&!*.  It told me I needed to apply 348000 ( and a whole host of other patches which I’m certain have already been applied:

You are an AWFUL DBA!

I immediately wrote off the wizard, as everything it reported was suspect.  But being a good DBA, I decided to launch a ticket with Oracle (on the world’s slowest support site) to ask about it.  The analyst came to the awesome conclusion that my problem was that because I may have missed some pre-requisites to, that would show up as missing, even though applied.  I gave up on the SR, and researched it on my own, with the help of some of Steven’s folks at Oracle (thanks again, Steven, you’re invaluable).   It felt like the issue was that the (cloned) environment didn’t have the right patch information, but I wasn’t sure where that was kept.

The Bug

It turns out that when I tried to run “maintain current snapshot” from adadmin, it would fail, due to a known bug with 11g database (patch 7000281).  This behavior (which I should have noticed, but honestly, how closely do you check that your cloned environments maintain their snapshots successfully?) kept my TEST instance from knowing that many of my patches had been applied, and therefore caused patch wizard to throw out pages of erroneous patch recommendations.

Results, Take II

After patching, and re-running maintain current snapshot successfully through adadmin, I re-ran the “New Code Levels” to see what recommendations it would make now.

Much better!


Actually, after it finally ran right, the patch wizard did find one patch which I missed for the new baselines, so I have to give it credit there.  Actually, it found a bunch of them, but only one was from a relevant product we actually used.  Will I ever use this product again?  Seems like I might, but not on 11i.  It wouldn’t be a bad thing to fire up before I patch up a regression testing environment, just to ensure I didn’t miss anything relevant.  Would I ever trust it to pull patches, or update itself?  No.  Oracle just seems to be really bad at that.  Products like OEM, OAM, and OCM seem to prove over and over that they can’t consistently push or pull data to or from Oracle.  Then I’d have to ask myself, is that really something I want my systems to do anyway?

Blog at