Kevin Kempf's Blog

May 25, 2016

When ADOP breaks because of the magic of checkfile

Filed under: Online Patching, Oracle, R12.2 — kkempf @ 3:57 pm

As usual, it’s been some time since I’ve thrown anything out here.  What’s been keeping me busy has been largely to do with ADOP.  I have a real love/hate relationship with this new addition to the EBS family, and lately it’s been more hate.

Oracle defines nocheckfile as an unsupported option which isn’t to be used unless explicitly stated in the readme for a patch.  Intriguing, wonder what that’s all about, why would it be there at all?  Let’s start from the beginning, defining default adop behavior and what checkfile does.  In theory, during the adop apply phase, the default option “checkfile” says “go see if I already know I have a more advanced or equal version of this code before I apply it”.  If it thinks it already has ingested this particular bit of code, don’t process it in an effort to save time during the patch.  Couple of key things here to note.  First, it’s trying to save me time.  Second, is it ever wrong about whether it’s already ingested an advanced version of the code?

First, let’s talk about saving time.  That’s a good thing.  If I only have to ingest 30% of a large patch, even using adop with the EBS available, that’s a good thing.  Now the second part.  It’s only good if it’s accurate.  100% accurate.  Because if it skips code I actually need, it turns out you wind up with a big, stinking mess to sort through.  There’s a little foreshadowing there.

Take this scenario, which you’ve probably guessed by now happened to me.  Some bug surfaced as a result of some aborted patch cycles which made my EBS adop cycle think it could safely skip pieces of a patch which it had thought were applied.  Only they weren’t, that patch cycle was aborted.

Here’s your first hint that you may be experiencing this: you get a warning during the apply phase that the patch already exists in your system, but you know that not to be the case.  At this point, Oracle thinks the patch is in, but you know it’s not.  So you innocently go on and use forceapply, per the adop “HINT”.  Now you also think the patch is in your system.  But it’s not.  Because of the magic of checkfile, and the fact that the patch was previously aborted and adop thinks it’s in there, you’re in a world of hurt now.

It’s a really insidious state to find yourself in, and the fix is to reapply the patches, using nocheckfile and forceapply.  This basically says “apply the patch despite the fact that you already think it’s installed, and force every bit of code into the system regardless of whether you think you already have it”.

I can tell you this: never run an adop abort without a full cleanup, as well as an fsclone to reset your patch environment.  That should keep you in good shape.  Officially Oracle now advertises this in Oracle E-Business Suite Maintenance Guide, Release 12.2 (Part Number E22954-20).  I strongly recommend reading that from cover to cover.  Because early on in 12.2, this wasn’t how adop abort was explained even by Oracle University.

Blog at WordPress.com.