Kevin Kempf's Blog

August 26, 2015

Fixing Enterprise Manager 12c Timeouts

Filed under: Enterprise Manager — kkempf @ 11:08 am

EM Timeout

Old Habits Die Hard

I have a monitor devoted almost exclusively to the performance page on my old EM 11g, still monitoring my main E-Business Suite production instance.  I can see at a glance if the system is busy, has blocking locks, massive I/O, etc.  It just feels comfortable and gives me a quick status at a glance.

Enter Enterprise Manager 12c.  I installed and have it monitoring everything but my most important environments at this point.  It took a bit of getting used to, but eventually I got the hang of it and worked out the handful of issues I had with it.  The biggest problem I had from a usability standpoint was that it keeps timing out on the performance home page I’m trying to keep running on my 3rd monitor.  There’s even a checkbox which says “Disable Timeout on Performance Home Page”, but apparently it doesn’t do anything.  And yes, I’m aware of the security issues and why it’s in place; in my case this is not a concern (physical security > application security).

If you scour My Oracle Support, you may come across Doc ID 1644004.1, which tells you how to increase the timeout.  What it doesn’t tell you is that if you set the value of to -1, you disable the timeout altogether.

Not Quite Done

Nothing is ever simple.  Apparently after making this change, EM defaults to a 45 minute value for oracle.sysman.eml.maxInactiveTime which results in this screen


So you need to do this also:

emctl set property -name oracle.sysman.eml.maxInactiveTime -value -1 -sysman_pwd [your sysman password]

emctl stop oms

emctl start oms

June 24, 2015

Database 12c: Support, Installation Types, Costs and Clarifications

Filed under: 12c, Licensing — kkempf @ 10:45 am


What the heck is this all about?

This is a bit off my usual rambling path, but because I think it’s important to understand I thought it a useful diversion.  Database 12c is out there, and hopefully you’re moving towards it.  That 11.2 database that you love because it works reliably?  It’s got one foot in the support grave.  So I started to research all of the implications and thought I might save you some time by summarizing what I found here.

Support Expiry

What’s the difference between premier, extended and sustaining?  You can find the official information here.


In short, I interpret premier as the “normal” state, where you can get support at your expected cost, security updates come out regularly, you can log bugs and support will create fixes as required.


When you wait too long to upgrade, you fall into the extended support bucket.  You lose some things.  For example, Oracle stops certifying products against this older release.


You’re way behind the curve.  Critical patches, and other rather critical updates stop rolling out unless they existed before sustaining support began.  Not where you want to be in production unless you have no other choice.

How much time do I have to migrate to 12c database?

Not much, unless Oracle throws out another premier support extension.  You can see the official GA information here(see page 4 for database); Oracle bases the support windows on what the General Availability (GA) date was for the release.  The bottom line is that as it stands right now, 11.2 is already out of premier support as of January, 2015.  Good news: they waived that for 1 year.  “NOTE: The fee for the first year of Extended Support for 11.2 has been waived for all customers. The Premier Support timeline has been extended one year to show that customers do not need to pay the Extended Support fee to receive Error Correction Support for that first year.

Now you have until January, 2016 to get to 12c, or you fall on extended support.  Who cares?  What’s the difference?  You better contact your sales rep, but I know in the past I’ve paid a support surcharge to stay on an older release (I didn’t have a choice, my 3rd party application wasn’t certified on the latest release yet) during extended support.  Here’s the official dates from Oracle:

GA Date September 2009 June 2013
Premier Support Ends January 2015 July 2018
Extended Support Ends January 2018 July 2021
Sustaining Support Indefinite Indefinite

Database 12c: Standard Edition One Versus Enterprise Edition, Container vs. Single Instance

This caused me a lot of confusion so I thought I’d clarify some things.  For anyone with their head in the sand, database 12c introduces multi-tenant.  In a nutshell, this means you can run multiple databases in one instance, and share the data dictionary overhead in the container (system, sysaux, and more) while the data lives in the pluggable.  It’s way more complicated than that, I know, but that’s the gist of it.

Standard Edition One Standard Edition Enterprise Edition Container Install In-Memory Feature Yes Yes Yes Optional No No No Yes Optional Yes

So what does this mean?  As of right now, if you’re running Standard Edition One 11g and trying to get to 12c, you cannot install, you must use  It doesn’t ask you what version you want to put down, it only does Enterprise Edition!  During the process, the GUI asks you if you want to create the database as a container database.  If you’re uninterested in multi-tenant (or can’t afford it.. more on that in a moment) you can simply not check that box and you get what Oracle now refers to as a “Single Instance” database.

It gets more interesting with Standard Edition One.  I have a handful of production databases running on SE1, because I want them isolated from my big production instance, they don’t do much, etc.  Well why not consolidate these under one multi-tenant SE1 environment and make my life easier?  Because you’re not allowed to.  SE1 allows 1 container and 1 pluggable per license.  Why anyone would do that is a mystery to me, but that was the answer I received from Oracle sales.  I don’t know for certain if that also applies to Standard Edition but for Standard Edition One, you can’t run multiple pluggables.  Did I mention SE1 was designed to compete with M$ SQL Server and they’ve been doing their version of multi-tenant for years?

Break out your wallet

If you want to use any of the cool new features of 12c, including multi-tenant, bring your wallet to the negotiating table.  You can find list pricing here.  If you click on US Technology Commercial Price List, you will see that two of the most core, critical features of 12c come with an added cost:

Add On Cost Items

Multitenant In Memory
Named User Plus $350 $460
Software Update License & Support $77 $101.20
Processor License $17,500 $23,000
Software Update License & Support $3850 $5060

These two aren’t cheap.  Multitenant costs about an extra 33% of the base database price, and In-Memory almost 50%.  If you don’t know what In-Memory is, I’m not talking about the buffer cache.  You need to check it out, it’s amazing.

How does this pertain to EBS?

It’s complicated.  If you look at the good work being done by Steven Chan, you can see was officially certified with EBS 12.1 last fall, and if you look at certify at support you can see for EBS 12.2 there are some “lagging” certifications (most notably dataguard, active dataguard, transportable tablespaces).


Bottom line: do your homework, your mileage may vary.  I’d like to think that because these certifications came well after the RDBMS GA date, Oracle might be generous and buy us EBS customers an exception with more time on 11gR2 or 12c, but you’d be foolish to count on it.

Parting Shot

This page includes tabs to downloadable .pdf’s of the 12c database architecture diagrams, performance/data dictionary view, background process descriptions, multitenant architecture, etc.

April 28, 2015

Security Patch Releases with EBS: Mission Impossible

Filed under: R12.2, Security — kkempf @ 12:33 pm

Regression Test Time

We’re about to enter our first R12.2 regression test.  High on my list is to get to Delta 6 on the AD/TXK side, and since the security patches just came out I figured I’d get up to date there.  For pretty good reason, we’re still running base, and it took me some time to get the ETCC (patch 17537119) happy that I’ve got all the required patches.

Per 1967243.1 I figured I’d start with the core database.  Now we have combo PSU’s for database, OJVM, and GI, in addition to SPU’s and stand-alone PSU’s for each component.  It’s gotten complicated since I looked last!  The patches are all opatch installed; ideally I’d get the OJVM, SPU and PSU applied to the database home.

OJVM (20406239)

This is a documented problem on MOS.  Apparently, it’s not really cumulative, because I have to be on Oct 2014 or better security set to apply this.  Next.

The following make actions have failed :

Re-link fails on target “jox_refresh_knlopt ioracle”.

Do you want to proceed? [y|n]
User Responded with: N


PSU (20299013)

This one was even worse!

There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
20488666, 20299013, 19791273, 19730032, 18260550, 17420796

Following patches are not required, as they are subset of the patches in Oracle Home or subset of the patches in the given list :
17811789, 19393542, 18828868, 18614015, 17892268, 17600719, 17468141, 16992075, 16929165

Following patches will be rolled back from Oracle Home on application of the patches in the given list :
20488666, 17811789, 19791273, 19730032, 19393542, 18260550, 18828868, 17420796, 18614015, 17892268, 17600719, 17468141, 16992075, 16929165

Conflicts/Supersets for each patch are:

Patch : 20299013

Bug Conflict with 20488666
Conflicting bugs are:
17912217 ETCC R12.2 requirement per 1594274.1

Bug Superset of 17811789
Super set bugs are:

Conflict with 19791273 ETCC R12.2 requirement per 1594274.1
Conflict details:

Bug Conflict with 19730032 ETCC R12.2 requirement per 1594274.1
Conflicting bugs are:
17174582,  18282562,  18244962,  17614134,  18674024,  17050888,  17478145,  18331850,  18964939,  17883081,  18436307

Bug Superset of 19393542
Super set bugs are:

Conflict with 18260550 ETCC R12.2 requirement per 1594274.1
Conflict details:

Bug Superset of 18828868
Super set bugs are:

Conflict with 17420796 ETCC R12.2 requirement per 1594274.1
Conflict details:

Bug Superset of 18614015
Super set bugs are:

Bug Superset of 17892268
Super set bugs are:

Bug Superset of 17600719
Super set bugs are:

Bug Superset of 17468141
Super set bugs are:

Bug Superset of 16992075
Super set bugs are:

Bug Superset of 16929165
Super set bugs are:

Security Patch Update (20299015)


Conflicts/Supersets for each patch are:

Patch : 18203837

Conflict with 18260550 ETCC R12.2 requirement per 1594274.1
Conflict details:

Patch : 19972566

Conflict with 18614015 ETCC R12.2 requirement per 1594274.1
Conflict details:

Patch : 20506715

Conflict with 19730032 ETCC R12.2 requirement per 1594274.1
Conflict details:

Patch : 20631274

Bug Superset of 17600719 ETCC R12.2 requirement per 1594274.1
Super set bugs are:

Following patches have conflicts: [   18260550   18203837   18614015   19972566   19730032   20506715 ]
Refer to My Oracle Support Note 1299688.1 for instructions on resolving patch conflicts.

Security or Compatibility: Why are these two at odds?

I’m trying to do the right thing, but Oracle is making it so hard to do that I’ve lost interest.  I don’t have a month to figure out what merged patches I need, what patch can go and what can stay. Why can’t there be a single note for EBS by version, with all patches for all components?  A database patch set which already deconflicted all the oddball ETCC requirements and delivered something which you could actually install out of the gate would make life much easier.


April 7, 2015

ADOP intelligent warnings, the ever-fluid recommended tech patches, and multiple versions of the truth

Filed under: R12.2 — kkempf @ 1:34 pm

So there I was..

Running adop phase=prepare in my production R12.2 environment, and I notice a warning on the screen and in the logs:

[WARNING]   The following required database bug fixes <19393542> are missing from this node, <database hostname>. Refer to My Oracle Support Knowledge Document 1594274.1 to identify the patch that delivers this bug fix.

This raises some questions:  How did adop know I needed a patch?  How serious is this?

Consolidated ≠ Updated

I went and pulled Consolidated List of Patches and Technology Bug Fixes (aka Doc ID 1594274.1) and it’s changed a bit since we went live.  Mind you, I expected to find patches in there we hadn’t applied as part of our go live; we locked our code last October before the upgrade.  What I didn’t expect was to find 14 patches on the database alone.  After some tedious cross checking of my inventory versus the document, I remembered (patch 17537119).

Patch 17537119 is really cool. This little gem compares an .xml file in the patch with your inventory, and tells you what you’re missing.  Sweet!

$ unzip
inflating: README.txt
inflating: checkDBpatch.cmd
inflating: txk_R1220_DB_base_bugs.xml
$ ./
|    Copyright (c) 2005, 2014 Oracle and/or its affiliates.     |
|                     All rights reserved.                      |
|               EBS Technology Codelevel Checker                |
Executing Technology Codelevel Checker version: 120.23
Enter ORACLE_HOME value : /u01/appprod/oracle/proddb/
Is this a RAC environment [y/n] : n
Enter ORACLE_SID value : PROD
Bugfix XML file version: 120.0.12020000.20
Proceeding with the checks…
Getting the database release …
Setting database release to
DB connectivity successful.
Table to store Technology Codelevel Checker results exists in the database.
STARTED Pre-req Patch Testing : Tue Apr  7 14:29:20 EDT 2015
Log file for this session : ./checkDBpatch_6400.log
Got the list of bug fixes to be applied and the ones to be rolled back.
Checking against the given ORACLE_HOME
Opatch is at the required version.
Found patch records in the inventory.
All the required one-offs are not present in the given ORACLE_HOME
List of missing bug fixes:
** Please refer to MOS Doc ID “1594274.1:Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes” to find the corresponding patch that delivers the bug fix. If an overlay patch is needed for any particular patch listed, the footnote for that patch also applies to the overlay patch.**
Stored Technology Codelevel Checker results in the database successfully.
Apply the missing bug fixes and run the script again to confirm.
FINISHED Pre-req Patch Testing : Tue Apr  7 14:29:24 EDT 2015

One lies, one tells the truth

I realized that the output of did not agree with 1594274.1.  In fact, it contained more patches than 1594274.1.  I would have assumed they were aligned, and this is really confusing as a customer: Which one is the definitive source?

In the end, I went with the, since it was a superset of the information contained in the Consolidated List of Patches and Technology Bug Fixes.  Perhaps a name change is in order?

March 20, 2015

Installing Oracle Linux on your PC

Filed under: Oracle Linux — kkempf @ 9:16 am


A break from the usual

I’m not a big fan of Windows as a PC operating system in the workplace, at least for what I do every day.  For the past 4-5 years I’ve been running Ubuntu and I recently upgraded to their version 14.  It was fine, but being a Debian release it was still clumsy and some things were very awkward/required hacks to make work (for example, the Oracle client).  So I decided to try something new.  The only requirement was it had to pass some arbitrary usability threshold in my head, and be installable from a USB key because I don’t have a DVD drive.

Fedora 21

The screenshots looked promising, but in the end this was a debacle.  I followed the instructions on their website, which included how to create a bootable USB drive.  The download took forever for the smallish install, though when complete I had a bootable USB drive (after using the creation utility they recommended).  I booted to it, but once in the GUI it was really acting up.  I have two monitors, and the GUI kept going on and off between the two.  It made installing it really hard, and by the time I got to the disk partitioning part and it said I didn’t have enough room, I was done with this release.  To be fair, the PC had existing partitions on it, and they were encrypted, this may have been the issue.  I can’t believe this is a show stopper for an installer, but something wasn’t right.

Oracle Linux 6.6

Actually figuring out how to get the installer going is a bit tricky, and the essence of this post, but once installed I’m feeling like it’s where I’ll be for awhile.  I’m assuming you’re doing this from Windows; if that’s not the case, you will have to read up on how to create the USB drive from your operating system.  Here’s the path to happiness:

  • Get an 8GB or larger USB drive
  • Download the install
    • Go to Oracle E-Delivery
    • Sign away your life, and select Oracle Linux 6 Update 6 Media Pack for x86_64 (64 bit) or 32-bit I suppose if you’re into that
    • Here you’ll see 5 downloads, but we’re only concerned with 2
      • Oracle Linux Release 6 Update 6 for x86_64 (64 Bit)
      • Oracle Linux Release 6 Update 6 Boot iso image for x86_64 (64 bit)
    • OLdownloads
    • Now head over to Linux Live USB creator, and install it as a Windows application
      • Point this application at the smaller Boot ISO image (226MB one), select your USB drive as the target, hit go, and now your USB key is bootable
    • This is the part which tripped me up and caused some grief: after creating your boot key, copy the big raw ISO file (3.7GB) to the USB key also
      • After you configure the basics in the installer GUI, it needs this ISO to actually do the install
      • If you don’t do this, you get an error like: Missing ISO image The installer has tried to mount image #1, but cannot find it on the hard drive.
        • It seems possible that you could have this ISO on another USB drive and point it to it, but I had no success with it
        • It appears that the installer will not recognize a dynamic mount of a USB key in the middle of the process, so you can’t switch out keys or add a 2nd one
          • I even tried to put two USB keys in, booting from the small image and then pointing to the big ISO (on /dev/sdc) when the installer got confused but it didn’t work either
    • At this point, you just put the USB key into your PC, ensure USB is in the boot sequence at the bios level, and start the PC
    • Early in the installer, the TUI (text user interface: a big red box) asks where something is located.  I’m sorry I can’t be more specific, nor do I remember what it was looking for.  But I have the answer in general:
      • My PC is simple: one hard drive, and the USB drive
        • Linux sees these as sda1, sda2, sda3 (my hard drive, with 3 partitions) and sdb1 (my usb drive)
      • What it needs to be told is sdb1 (your USB drive) and then it chugs along happily


I feel obligated to say that you have a strong chance of wiping out whatever is on your PC if you don’t know what you’re doing.  In my case, that was the whole idea, so the risk was low.  If your goal is to create a live USB operating system, or dual boot OS, that’s beyond the scope of what I’m talking about, and requires more/different steps.


Let’s assume you’ve installed the OS, and you’re booted to the GUI now.  You need to add the public yum repos to get updates

  • $ su –
  • # export http_proxy=  (skip if you don’t have a proxy, obviously requires some tweaking)
  • # wget -P /etc/yum.repos.d/
  • # yum update

Customizing Your Environment

It takes time to get all the pieces working; your requirements will be different from mine.  I’ll give you a few obvious ones

  • Java (JDK)
    • yum remove java-1.6.0-openjdk
    • yum remove java-1.7.0-openjdk
    • go download Oracle java
    • rpm -ivh *.rpm (it puts it in /usr/java/jdk…)
  • Java plugin for Firefox so you can run Forms in the Ebusiness Suite
    • $ cd ~/.mozilla/plugins
    • $ ln -s /usr/java/jdk1.7.0_67/jre/lib/amd64/   (substitute your version of java where I’ve listed jdk1.7.0_67)
  • SQL Developer
    • Go download the most recent version
    • rpm -Uvh sqldeveloper*rpm (it installs in /usr/local/bin)
    • $ cd /usr/local/bin/
    • ./sqldeveloper
    • when prompted, point it to your JDK installation
      • Type the full pathname of a JDK installation (or Ctrl-C to quit), the path will be stored in /home/kkempf/.sqldeveloper/4.0.0/product.conf
      • enter: /usr/java/jdk1.7.0_67 (obviously change your version to match)

You’re on your feet

The rest is up to you


March 9, 2015

R12.2 Apache won’t start up

Filed under: Linux, Oracle Linux, R12.2, Weblogic — kkempf @ 9:45 am



I had to bounce my R12.2 non-production front end this weekend to switch kernels around.  I restarted services, but admit I didn’t pay much attention to them as it wasn’t PROD.  We run multiple instances of R12.2 on different port pools but on one host (well 2, one for the databases, one for the application servers). We do this not because I like it, but because we don’t have infinite budgets.  Regardless, I get a complaint that our audit environment isn’t working.  Sure enough, the login is unavailable, and Weblogic says the web tier process EBS_web_AUDIT is down.

Identify the problem: To the command line!

I try to simply start it in Weblogic but it errors out, and there’s far too many log links to click for this admin to stay in a GUI.   I run start and get:

You are running version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance … exiting with status 204

Follow the bouncing ball tells me to check my log file: $INST_TOP/logs/appl/admin/log/adapcctl.txt, and here’s what it tells me:

ias-instance id=EBS_web_AUDIT_OHS1

--> Process (index=1,uid=1810789619,pid=11169)
  failed to start a managed process after the maximum retry limit

03/09/15-09:52:40 :: exiting with status 204
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address [::]:10059
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] no listening sockets available, shutting down
[2015-03-09T09:38:34.0591-04:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] Unable to open logs

Closing in

So it appears that my user (oraaudit) can’t get a port it wants (10059), and after a little hunting on Google I find the right syntax. I wanted to use netstat, but I couldn’t figure out the PID and lsof made it really easy:

# lsof -i:10059
java    13815 oratrain  305u  IPv6  82113      0t0  TCP (myhostname).com:10059->(myhostname).com:rds2 (ESTABLISHED)

Take down the suspect

It’s some random java process from another instance (Train). See ya.

# kill -9 13815
# lsof -i:10059

Restart Apache start

You are running version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ... exiting with status 0

Nothing to see here

This is something most apps DBAs have seen at some time or in some form. The path to follow changes, but in the end the fix is usually similar to this. Knowing with certainty that it’s OK to kill the blocking PID is always a delicate task. There’s absolutely no way you could have fixed this from within the Weblogic admin console that I can think of, short of going into the other environment and shutting it down completely.

February 25, 2015

R12 Take Aways and Warnings, Part 3 (Java Clients, Dataguard with editioning, logs, crons and the Workflow Notification Mailer)

Filed under: 11g, Dataguard, R12.2 — kkempf @ 1:58 pm


More R12.2 War Stories

I thought I’d go back to my R12.2 war stories today.  To be honest, I still have plenty to talk about even after this post, but a journey of a thousand miles begins with one step, right?

Java Clients

What a pain in the 4th point of contact.  We went into this upgrade knowing we wanted to deploy the most advanced Java client to the desktop (for forms) we could, so we didn’t have to deal with it.  Wow it’s a terrible hot mess.  We settled on 1.7.67.  Oracle, could we just call it 7.67?  Why do we need a 1 in front?  Sorry back to the topic at hand.  Java 7 is smart, and won’t run unsigned Jar files without much ado and many, many user clicks.  This is all good in the name of security, I suppose, but it sucks to administer.  At one point here I talked about the problems that arise.  Oracle tries to help mitigate this with a new product called Java Advanced Managment Console but it’s really new and really doesn’t seem to fully meet our needs.  So bottom line here: figure out how to get a java code signing certificate from your favorite CA (Thawte worked great for us) and what the new rules are for signing jar files.  You can search Enhanced Jar Signing on MOS, really you just need to understand adjkey to build the key, pick your CA, and use adjkey again to import the certificate.  Then you run adadmin (1,4) to regenerate jar files.


We utilize active dataguard with a physical standby against our Linux x86_64 database.  You may think Dataguard is a curious topic for an R12 upgrade.  The truth is that the upgrade itself has little to do with dataguard; in fact I disabled it before starting the upgrade because a) I didn’t want to ship hundreds of gigabytes of data down the pipe and b) I had no faith it would actually work.  After I was done with the upgrade I rebuilt the dataguard environment using RMAN, and it crashed every week for 6 weeks.  There’s a few known issues out there, but the error manifested itself like this:

Sat Dec 13 14:17:30 2014
Errors in file /u01/appprod/oracle/proddb/diag/rdbms/proddg/PROD/trace/PROD_lgwr_24265.trc:
ORA-04021: timeout occurred while waiting to lock object
LGWR (ospid: 24265): terminating the instance due to error 4021
Sat Dec 13 14:17:30 2014
System state dump requested by (instance=1, osid=24265 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /u01/appprod/oracle/proddb/diag/rdbms/proddg/PROD/trace/PROD_diag_24236_20141213141730.trc
Dumping diagnostic data in directory=[cdmp_20141213141730], requested by (instance=1, osid=24265 (LGWR)), summary=[abnormal instance termination].
Instance terminated by LGWR, pid = 24265

And now you have no dataguard!  After trying various patches (17588480, 19631234, and 16299727), it appears that the real problem was that dataguard didn’t handle editioning well and the answer was patch 16299727 and adding

_adg_parselock_timeout=500  # dataguard editioning fix
event=”16717701 trace name context forever, level 104887600″ # dataguard fix

to my pfile/spfile fixed the issue.


There’s lots of logs in R12.2.  Pretty safe, obvious statement, right?  Well you need to go figure out where they are, because they all moved in 12.2.  If you don’t, well you’re gonna have a mess or possibly run out of disk.  I cron’d up a bunch of find commands to get rid of them; obviously don’t paste blindly, you need to tweak the retentions to your needs.  Some of these are supposed to be covered by cleanup jobs, but I find this is a nice insurance policy:

# Remove old outputs $APPLCSF/out or /u02/appprod/fs_ne/inst/PROD_hostname/logs/appl/conc/out
0 8 * * 1 /usr/bin/find /u02/appprod/fs_ne/inst/PROD_hostname/logs/appl/conc/out -mtime +30 -type f -exec rm -rf {} \;
# Remove old reqs $APPLCSF/log or /u02/appprod/fs_ne/inst/PROD_hostname1/logs/appl/conc/log
0 8 * * 1 /usr/bin/find /u02/appprod/fs_ne/inst/PROD_hostname/logs/appl/conc/log -mtime +30 -type f -exec rm -rf {} \;
# Remove Weblogic oacore Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server4/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server4/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic Admin Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic Forms-c4ws Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms-c4ws_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms-c4ws_server1/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic oafm Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oafm_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oafm_server1/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic forms Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server4/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server4/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic wlst Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic OHS Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/webtier/instances/EBS_web_PROD_OHS1/diagnostics/logs/OHS/EBS_web_PROD -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/webtier/instances/EBS_web_PROD_OHS1/diagnostics/logs/OHS/EBS_web_PROD -mtime +7 -type f -exec rm -rf {} \;


While we’re on the subject of cleaning up logs, let’s talk about crons.  If it weren’t for cron, my R12.2 environment would not run.  Period.  Besides cleaning up logs, I have crons to do things like

  • rotate MSCA ports (telnet server)
  • bounce MSCA (daily!  it’s horribly unstable)
  • monitor various R12 related things from the OS (for example, use curl in a shell script to ensure the front end login page is up)
  • bounce JVMs every week (really, you need to do this)
  • cleanup apps sessions which otherwise won’t timeout or die (more on this later, worst surprise of the upgrade)
  • preclone the apps tier

Workflow Notification Mailer

If you’re using this, especially if your mailer accepts IMAP/inbound email responses as a part of the workflow, make sure you understand how it looks in 12.2.  It does not survive the upgrade from 11i, and the screens are totally different.  As far as I can tell, they don’t require any new or additional information to work, they just changed around the configuration screens so as to confuse you.  Bottom line is this: it mostly works once you get it configured right, but you should be aware that there’s a patch 18842914 for the mailer which greatly improved its reliability.  I put that in because it kept crashing when some 3rd party server was spamming it with email bounces (yeah, I’m not going into that any further).

Coming Soon

I still haven’t covered the most annoying things… like adop, the techstack, custom tops, MSCA in general, report manger replacing the ADI desktop client, and having to manually kill sessions to keep the database from dying…

January 30, 2015

Determine Weblogic Server Version in EBS R12.2

Filed under: R12.2, Weblogic — kkempf @ 10:08 am

A diversion from my topic in progress

Stumbled across this via an analyst on an SR I was working, thought it might be useful for others.

$ cd $FMW_HOME/wlserver_10.3/server/lib

$ java -cp weblogic.jar weblogic.version

WebLogic Server Tue Nov 15 08:52:36 PST 2011 1441050

Use ‘weblogic.version -verbose’ to get subsystem information

Use ‘weblogic.utils.Versions’ to get version information for all modules

January 13, 2015

R12 Take Aways and Warnings, Part 2 (AD Changes and your new Techstack)

Filed under: R12.2, Techstack — kkempf @ 9:53 am


ADOP and your new Techstack

As promised, I’m going to continue the discussion about about my R12.2 upgrade and the lessons learned as a part of it by digging into the techstack.  There’s a lot of ground to cover here!

General Upgrade Flow and Layout

For a general understanding, I thought I’d start with a basic explanation of the upgrade process.  Let’s assume a simple installation, where you have a database tier and an applications tier.  As usual, I’m tossing out my disclaimer that this list is NOT complete.  It’s a high level look at the basic tasks.  Follow the official doc!

  • 11i Pre-Patching/Prep Work

You will need to first prep your old environment (in my case 11i, though I suspect this will hold true if you’re on 12.0/12.1) for the upgrade.  This means under 11i, you need to apply database patches, run checkers, clean up your customizations, etc.  Since R12.2 allows for a 64-bit front end, we chose to create a new front end host and leave the database where it was (since it ran fine).  Compared to 11i, these new certifications allow you all kinds of nice things, such as running Linux 6, and allocating huge sums of memory (with a 64-bit OS) to the host without running a PAE (physical address extension for memory) kernel.  This in turn allows you to easily accommodate many, large JVMs on the front end.  Regardless, you need to be at the appropriate baseline before you begin the upgrade.

  • 11i Database Patching

Pretty self explanatory; there’s a bunch of mandatory patches to the RDBMS tier.

  • 12.2 Rapid Install

You first lay down the file system(s) on whatever will be your new R12 application server.  Note: You can’t do this ahead of time in production.  At this point you’re touching the database and affecting 11i

  • Application DBA Tasks

In the middle of your upgrade, you need to perform a bunch of AD tasks.  Such as getting on the right version of the techstack, extending tablespaces, fixing customizations, etc.

  • Adpatch

This is the last time you’ll run adpatch, and it’s a monster: the largest, longest bit of work involved.  Lots of worker fails with errors which you have to go research.

  • Another 12.2 Rapid Install pass

Short pass here, against your new R12 filesystem.  Congratulations, after this you’re on 12.2 (not a production release, but you can login after this).

  • Enabling 12.2 Online Patching

This is why you decided to install 12.2 right?  This is a long, database intensive task.

  • Applying Techstack Patches

You can mitigate this by using the latest install CD for 12.2.X, but you’ll be installing patches against Weblogic, forms and reports, http server, FMW etc

  • Applying 12.2.3 (or 12.2.4, presumably!)

This is it’s own patch, applied with adop

  • Post-install Tasks

The devil is in these details, which are specific to your installation.  An example is configuring Vertex (tax) for your financials

JVM Sizing

If you’re serious about this upgrade, please Google up Oclay Sariouglu and Dimas Chbane’s (both from Oracle) presentation from East Coast Oracle Users Group conference titled “E-Business Suite 12.2: Architecture and New Features”.  Specifically, a PowerPoint called Chbane 12.2 Architecture.pdf is worth reviewing; I’m sorry I don’t know with certainty that you can get it easily if you didn’t attend the conference, you can probably get it through official channels.  Here’s the bottom line: Oracle has a scaling recommendation (and a revised scaling recommendation!) for adding oacore and forms JVM’s based on the number of users.  The default installation for R12.2 gives you one (1) JVM for forms (forms_server1) and one (1) JVM for oacore (oacore_server1), and each take maybe 256MB or 512MB (I forget which).  This is ludicrously small, for any real deployment, and I’m glad I went to this conference and attended this session to at least get the references on how to size them and how to add them.  These JVMs live inside of Weblogic, and you need to size them correctly.  Because if you go live with the defaults and more than a few dozen users, your JVMs will crash.  Check out 1905593.1.  We went live with 4 of each (forms, oacore) at 2GB per for an installation footprint of about 1000 users (about a third of whom are in professional forms) and it’s served us well.  The truth is our forms footprint is probably too big, but we haven’t cut it back yet.

Adding JVMs is a somewhat manual, command line process, or at least was when we did it, and for being rather essential to a go-live, it stands out as an after-thought to the whole process.  This task also includes changes to the context files, which also has hooks to your start/stop scripts (  Consequently, this has the side effect of making an application server startup (or a maintenance adop cutover) even slower (if it will go down cleanly at all, but I digress).  Another bit of fallout is that when you clone, you may need to cut back these JVMs, depending on the memory resources you have available on your non-production host(s).  If you find yourself in this situation, I’ll offer this tidbit: this can be accomplished by changing the s_oacorename and s_formsname parms in the context file to eliminate, for example, oacore_server2, oacore_server3, etc. until you get it sized appropriately.

AD/TXK Patching

As you probably know, officially, AD is the product abbreviation for Applications DBA, and TXK corresponds to Teckstack.  The two are tightly interwoven, to the point that in R12.2 (perhaps in earlier editions, unsure) they must always be on the exact same version, and patched together.  It raises an interesting aside, in that to me, as the DBA, as they’re now indistinguishable why bother to make any distinction and not just roll them up under one product… but I digress.

In the middle of your upgrade, you apply AD.C.  In my original plan, I think I had the choice of applying AD/TXK delta 3 or delta 4.  Since I had nothing to lose, I went with delta 4, and during the course of our upgrade testing delta 5 was released.  Delta 5 fixed so many critical bugs in the AD stack that I had to beg the project manager to let me put it in late in the testing phase.  For example, we use mobile supply chain (MSCA) and under delta 4 the telnet ports would change during an adop cutover phase (imagine if you will telling hundreds of manufacturing operators to change their telnet ports on PC’s and hand-held devices after every maintenance weekend).  All of this gets rather confusing, because officially I’m on AD/TXK delta 5, but that’s not all.  Now there’s essential updates on top of AD/TXK which get released, seemingly at random.  For example, patches 19445058/19581770 (now superseded by 20034256: additional fixes that were not in the first bundle/November bundle patches for AD Delta 5!) are called “BUNDLE FIXES FOR R12.TXK.C.DELTA.5” and “BUNDLE FIXES FOR R12.AD.C.DELTA.5”.  This is a mess.  To me, it would be much simpler to consolidate the AD/TXK patches and put some meaning to the versions.  Why can’t AD/TXK delta 5 with bundle fix 1 be called, simply, AD.C.5.1?  No TXK, I don’t care about the distinction since I always have to apply them together, why are they distinct products to the customer?  It’s like selling a cup and the water separately at a restaurant.

Regardless, my point about the AD/TXK C, Delta 3, 4 and 5, additional fixes, essential bundles, mandatory mess ups, fantastic fixes, or whatever else they call it: they change a lot.  Research what the latest, greatest version is and install it.  Now welcome to another new job task: you need to keep up with this in 12.2 (subscribe to Steven Chan’s blog, at least you’ll get an email when these are released, among other things).

I’ve rambled enough for today, I have plenty more to cover.  Up next time: more AD changes and your new techstack!

January 9, 2015

Online Patching

Filed under: Online Patching, R12.2 — kkempf @ 11:11 am

ADOP my friend (today)

So I took the plunge and did the main thing we chose R12.2 for, as opposed to R12.1.  I applied a few patches prior to a downtime/maintenance cycle this weekend in my Production system.  Kind of scary, but everything went OK.  Here’s what’s interesting:

online patching load

Guess when I ran my first adop phase=apply (…) ?   <spoiler: 11:25am>

It was a curiously intense bit of work for the database to apply this patch (19523116 for the record) in online mode.  Oracle never advertised that online patching did its work with 0 impact, but it was a bit of a surprise to see the database system CPU maxed out as a result of it.

I should add that I did this yesterday, and didn’t get any calls or complaints so this in no way affected Production that I ‘m aware of.  My point is only that you should be aware that online patching has the potential to impact your server in significant ways.  Maybe don’t run it during month end close.

Older Posts »

The Silver is the New Black Theme. Create a free website or blog at


Get every new post delivered to your Inbox.

Join 40 other followers