Kevin Kempf's Blog

June 5, 2017

The trouble with editions

Filed under: Online Patching, R12.2 — kkempf @ 10:27 am
socrates
(fake quote to grab your attention)

EBS 12.2 and Edition Based Redefinition

Unless you’ve been living under a rock for the past 5 years, you’re probably aware that EBS 12.2 online patching leverages Edition Based Redefinition (EBR).  In a nutshell, when you begin an online patch cycle, the adop phase=prepare command creates a new edition of the database which “contains” all current objects, but will also absorb database changes as you apply patches.  Simplified, it becomes your “on deck” database, ready to take effect (or step up to the plate, if you prefer the analogy) after you complete the adop phase=cutover.

Recognizing you have a problem

Left unattended, this system will continue to work for a long time without issue.  In my case, over 2.5 years, before I realized I had become an edition hoarder and needed help.  I realized that I had old editions hanging around, back to the day of our 11i to 12.2 go live.  Oracle very conveniently names the editions after the date/time they were created.  So you can see our first 12.2 edition was V_20141127_2225 (11/27/2014 at 10:25pm).

select edition_name from dba_editions order by edition_name;
ORA$BASE
V_20141127_2225
V_20141128_1314
V_20141128_1504
V_20141128_2136
V_20141129_0027
V_20141129_0218
V_20141129_0709
V_20141129_0803
V_20141129_0852
V_20141129_1244
V_20141130_0623
V_20141130_0719
V_20150108_0953
V_20150127_1002
V_20150208_1037
V_20150208_1222
V_20150209_1058
V_20151006_1147
V_20151105_1218
V_20160107_1508
V_20160129_1412
V_20160318_0805
V_20160408_0959
V_20160420_1007
V_20160614_1519
V_20160616_1401
V_20160928_1047
V_20161111_0914
V_20170316_1417
V_20170320_1356
V_20170322_1318
V_20170329_1355
V_20170330_1819
V_20170512_1050

35 rows selected.

Get Help

The fix is pretty simple, but I’ll add a real-world “gotcha” to Oracle’s documentation.  You can look  at EBS 12.2 Maintenance guide(https://docs.oracle.com/cd/E26401_01/doc.122/e22954.pdf) Pg 3-27 to see the official verbiage, but here’s the gist of how to drop the old editions:

  • adop phase=prepare
  • adop phase=apply patches=(whatever patches you want to do, or none)
  • adop phase=actualize_all
  • adop phase=finalize finalize_mode=full
  • adop phase=cutover
  • adop phase=cleanup cleanup_mode=full

I’ll point out that Oracle’s document does not include the apply phase, however you can indeed apply patches during an edition cleanup.  Not only have we done it, but Oracle said it was fine to do in an SR.

The other thing the document doesn’t tell you is that this is not necessarily an impact-free operation.  We had a long maintenance window on a Sunday afternoon and decided to defer the phase=cleanup cleanup_mode=full until Monday morning when users were in the system.  Within 10 minutes, we had calls to the help desk because of massive object contention and locks, to the point where nobody could log into EBS.  I look at Enterprise Manager and it’s bright red, with hundreds of blocking locks in the system.  I don’t know if this is a result of the fact that we were dropping 30+ editions, or if it will happen just doing 1 edition.  Regardless, we ended up stopping the front end during the next maintenance cycle, then running the cleanup, and it was done in 45 mins or less.

SQL> select edition_name from dba_editions order by edition_name;

EDITION_NAME
------------
V_20170512_1050

October 11, 2016

Determining your EBS Code Level and Family Pack

Filed under: Online Patching, Oracle, R12.2 — kkempf @ 10:29 am

Quick and Dirty

There’s lots of posts out there telling you how to determine your EBS patch levels, code levels, etc.  But whenever I google them, I can’t find the one that actually tells me what I want in a simplified way.

Here’s what matters most often to me as the Apps DBA:

select
  abbreviation
,codelevel
from
  ad_trackable_entities
where
  abbreviation in( 'ad','txk','fnd','fwk','atg_pf','icx' )
order by
  abbreviation;

 

ABBREVIATION                   CODELEVEL                                                                                                                                            
------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------------------
ad                             C.7                                                                                                                                                   
atg_pf                         C.4                                                                                                                                                   
fnd                            C.4                                                                                                                                                   
fwk                            C.4                                                                                                                                                   
icx                            D.3                                                                                                                                                   
txk                            C.7                                                                                                                                                   

 6 rows selected

May 25, 2016

When ADOP breaks because of the magic of checkfile

Filed under: Online Patching, Oracle, R12.2 — kkempf @ 3:57 pm

As usual, it’s been some time since I’ve thrown anything out here.  What’s been keeping me busy has been largely to do with ADOP.  I have a real love/hate relationship with this new addition to the EBS family, and lately it’s been more hate.

Oracle defines nocheckfile as an unsupported option which isn’t to be used unless explicitly stated in the readme for a patch.  Intriguing, wonder what that’s all about, why would it be there at all?  Let’s start from the beginning, defining default adop behavior and what checkfile does.  In theory, during the adop apply phase, the default option “checkfile” says “go see if I already know I have a more advanced or equal version of this code before I apply it”.  If it thinks it already has ingested this particular bit of code, don’t process it in an effort to save time during the patch.  Couple of key things here to note.  First, it’s trying to save me time.  Second, is it ever wrong about whether it’s already ingested an advanced version of the code?

First, let’s talk about saving time.  That’s a good thing.  If I only have to ingest 30% of a large patch, even using adop with the EBS available, that’s a good thing.  Now the second part.  It’s only good if it’s accurate.  100% accurate.  Because if it skips code I actually need, it turns out you wind up with a big, stinking mess to sort through.  There’s a little foreshadowing there.

Take this scenario, which you’ve probably guessed by now happened to me.  Some bug surfaced as a result of some aborted patch cycles which made my EBS adop cycle think it could safely skip pieces of a patch which it had thought were applied.  Only they weren’t, that patch cycle was aborted.

Here’s your first hint that you may be experiencing this: you get a warning during the apply phase that the patch already exists in your system, but you know that not to be the case.  At this point, Oracle thinks the patch is in, but you know it’s not.  So you innocently go on and use forceapply, per the adop “HINT”.  Now you also think the patch is in your system.  But it’s not.  Because of the magic of checkfile, and the fact that the patch was previously aborted and adop thinks it’s in there, you’re in a world of hurt now.

It’s a really insidious state to find yourself in, and the fix is to reapply the patches, using nocheckfile and forceapply.  This basically says “apply the patch despite the fact that you already think it’s installed, and force every bit of code into the system regardless of whether you think you already have it”.

I can tell you this: never run an adop abort without a full cleanup, as well as an fsclone to reset your patch environment.  That should keep you in good shape.  Officially Oracle now advertises this in Oracle E-Business Suite Maintenance Guide, Release 12.2 (Part Number E22954-20).  I strongly recommend reading that from cover to cover.  Because early on in 12.2, this wasn’t how adop abort was explained even by Oracle University.

January 9, 2015

Online Patching

Filed under: Online Patching, R12.2 — kkempf @ 11:11 am

ADOP my friend (today)

So I took the plunge and did the main thing we chose R12.2 for, as opposed to R12.1.  I applied a few patches prior to a downtime/maintenance cycle this weekend in my Production system.  Kind of scary, but everything went OK.  Here’s what’s interesting:

online patching load

Guess when I ran my first adop phase=apply (…) ?   <spoiler: 11:25am>

It was a curiously intense bit of work for the database to apply this patch (19523116 for the record) in online mode.  Oracle never advertised that online patching did its work with 0 impact, but it was a bit of a surprise to see the database system CPU maxed out as a result of it.

I should add that I did this yesterday, and didn’t get any calls or complaints so this in no way affected Production that I ‘m aware of.  My point is only that you should be aware that online patching has the potential to impact your server in significant ways.  Maybe don’t run it during month end close.

Create a free website or blog at WordPress.com.