Kevin Kempf's Blog

September 5, 2014

11i Apache v and Symbolic Links

Filed under: 11i — kkempf @ 10:48 am


Minding my own business

While cloning 11i environments to try to get to R12.2, I ran into this error when logging in after completing the clone:


You don’t have permission to access /OA_HTML/AppsLocalLogin.jsp on this server.

Wow that’s pretty serious looking.  What did I forget?  Bad values in the context file for ssl certs?  No.  Talking ssl over a port not allowed by our proxy server?  No.  It turns out, I got a little creative in trying to re-arrange environments (on an old 32-bit front end Oracle Linux 5 box) and I built this particular 11i front end off of a symbolic link in Linux pointing to an unused 11i disk path (which has already been converted to R12, on a 64-bit front end Oracle Linux 6 box).  Sometimes you have to do strange things in the name of expediency, and also because the guy who allocates disk from the SAN is out today.


The fix

Edit your context file in $APPL_TOP/admin, and change s_options_symlinks from Options -FollowSymLinks to Options +FollowSymLinks.  This writes to httpd.conf and httpd_pls.conf under your IAS Home.

Shut down your services, run autoconfig and restart services.

Use at your own caution, as it’s probably violating 10 different security rules, but I don’t care I am throwing this environment away within 3 days.

September 3, 2014

Fun with R12

Filed under: R12.2 — kkempf @ 2:38 pm


Some genius at Oracle decided to land a user-owned lock file in /tmp for the Weblogic admin server in R12.2.  This would be OK, except it doesn’t have any unique identifier in it such as PID, instance name, etc.  It’s just called .ovdlock.tmp.  It’s even funnier because they make it a hidden file so it’s challenging to find.  

Regardless, as a result, if you have multiple instances running on one (Linux) host with separate OS users for each, one will start just fine, while the other will say

Starting server AdminServer …
Error Starting server AdminServer: weblogic.nodemanager.NMException: Exception while starting server ‘AdminServer’

Without much other explanation…. the fix is to su and chown 777 /tmp/.ovdlock.tmp and retry on the host which wouldn’t start.  There was a Weblogic bug for this, and maybe even a fix, but I haven’t applied it because it didn’t appear to be for my version (10.3.6) of WLS.  Read about it at 1906029.1

August 7, 2014

Changing the JRE version pushed by R12.2

Filed under: Java, R12.2 — kkempf @ 12:02 pm

Giving credit where credit is due

In light of all the recent fervor about Java, it’s appropriate that I give credit to Oracle when they get something right.

Changing JRE push version (minimum) in 12.2

It’s unbelievably easy.  Follow 393931.1

  • Download whatever version of Java you want to run this week for your enterprise </sarcasm> to /tmp
    • The way they’re pushing updates, I think I could script this whole thing to pull the latest version every week and install it….
  • cp /tmp/jre-7u67-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se17067.exe
    • land it in the “push” directory
  • $FND_TOP/bin/ 17067
    • run this utility to link the AD utilities and your context file to the new version
  • adop phase=fs_clone
    • make sure it gets sync’d to both fs1 and fs2

Done.  That’s it.  10 minutes tops.  Your EBS now pushes 1.7 update 67.

R12.2, Oracle Java, Microsoft, Google and Security

Filed under: Java, Oracle, Windows 7 — kkempf @ 11:51 am


The Playing Field

It’s no secret Google hates Microsoft (M$) hates Oracle hates Google hates Oracle hates Microsoft.  But EBS users are caught in the crossfire this time, and it’s not pretty.

M$ recently announced here they were going to start blocking Java launches out of IE.  You know, because they’re looking out for us.

Java started squalking about unsigned Jar files awhile back, but if your version was old enough, you wouldn’t see this message because it was too dumb to know it.  It also pesters you incessantly if you’re not running the latest version.  Always.

Google started stopping Java from running in Chrome which I noted here.  It was technically never certified with Oracle EBS, so I hold them least culpable at the moment.


I appreciate that all of these players are protecting their own interests and implementing these changes in the name of security.  People complain about Java security?  Great, now Oracle will issue an update every week, and if you’re more than 2 updates behind it will nag you forever.  Launching EBS which doesn’t have signed JAR files?  Big scary window pops up warning the user.  Want to run old Java on Windows 7?  Microsoft says nope.

I’m in the middle of an R12.2 upgrade.  I have the luxury of taking the time to go get a certificate from Thawte and sign my jar files, and test it.  The truth is that it’s pretty easy.  But so is the process of getting a certificate.  Meaning, nothing against Thawte, but by signing my jar files all I’ve done is satisfy the JRE plug in.  I haven’t really enhanced my security, in my opinion.  Same thing with upgrading to the latest version of Java.

Meanwhile, here in the real world

I work for a manufacturer.  The floor workers who interact with Oracle EBS don’t know/care/understand what Java is, what a signed jar file is, what version of Java they’re running, etc.  They just want to get into Oracle EBS.  We’re not internet-exposed.  Sure there’s risk of someone exploiting a security hole, but it would probably have to come from the inside.  This is a known risk we’re probably more willing to take than the risk of having to deal with hours and hours of confusion every time M$ makes a unilateral decision about Java, or Oracle delivers yet another Java update.   We’ve been running EBS 11i on Java 1.6_12 for over 5 years now.  Never caused a problem.  It’s ancient, but it works.  Much like, I suspect, many companies, we have a lot of legacy Java running.  The effort required to push a newer version of Java to these machines is not insignificant, since they’re all locked down on the manufacturing floor.

Battle of the sexes

I’m sick of it.  Java consumers (and I’m not talking about coffee here) are stuck in the crossfire between M$ and Oracle.  Here’s what I see happening:

1. We regression test and deliver R12.2 with Java 1.7.0 update 67 in 6 months

2. Oracle issues 25 more versions of Java 7 thus every user gets the “Update (Recommended), Block or Later” dialogue box

3. Some users block, some update, some say later.  66.66% of these scenarios cause IT problems

4. Microsoft, not to be outdone, declares Java 1.7 update 67 obsoleted, and thus adds it to a block list

Managing the Mess

Sure, we can manage Win7 updates through policy, and the same for Java updates.  But are we actually improving security, in any way?  Or just creating busy work for ourselves, where the version which runs on our isolated network is largely immaterial?



June 10, 2014

Let the games begin!

Filed under: 11i, R12.2, Ubuntu — kkempf @ 8:34 am


R12.2.3 Testing has begun

R12.2 changes a LOT of things, especially for an applications DBA.  We’re underway with our upgrade plans to 12.2, and I was happy to see if you have the patience to make all the pieces work (Chrome, Ubuntu, Java, PopUp Blocker, Chrome Plug-In for Oracle R12) you can actually launch forms from Ubuntu.

Initial Upgrade Impressions

There’s a lot of work to upgrade from 11i to 12.2.3.  The upgrade documents are extensive, and not light reading.  Customizations are a bugger to bring over, the whole landscape has changed with edition based redefinition.  Much of this revolves around the changes to the techstack (Weblogic), and online patching.  The new patching utility, adop, works in conjunction with other old favorite ad utilities, but it’s not terribly intuitive.  There was something reassuring watching my workers progress and run; maybe there’s a way to do it and I just haven’t figured out the appropriate CLI switch.  You can still look at them via adctrl, but it’s not the same, I don’t get the reassurance of knowing I only have 123,000 jobs left to finish.  Mostly, though, things look the same.  More to follow.

June 9, 2014

Google’s message to Ubuntu users of Chrome 35 and JRE

Filed under: Chrome, Ubuntu — kkempf @ 10:59 am


Take that Chrome Users!

OK, they really didn’t do that.  But they might as well have.

Ubuntu at Work

I like using Ubuntu as my primary desktop at work.  I have to keep Windoze around for compatibility issues, but Linux is much faster and more reliable.  Recently, I found that I kept getting the error “Java Plugin Not Installed”.  I finally had time to dig into it, and it turns out it was due to an upgrade from Chrome 34 to Chrome 35.  The sym link in my /opt/google/chrome/plugins directory was still good, but Chrome 35 just ignores it.  I found all this out (ironically) with a lot of Googling, I found this

I won’t say I understand all the jargon, but the gist of it is this: for whatever reason, Chrome on Linux no longer supports Oracle JDK/JRE as of a few weeks ago.  Here’s a funny thread showing me I’m not the only one unamused by this:

The Fix

If you find yourself in this spot, just use Ubuntu software center to uninstall Chrome, go pull version 34 here and reinstall.  Then it magically works again, and I can get into the EBS forms.  Google, I know you don’t care what I think, but seriously?  It’s Oracle Java.  Pretty universal, and this open source Iced Tea stuff doesn’t cut it for big boy applications.  Please restore this functionality, or I’ll have to move to the F-browser and nobody wants that!

April 24, 2014

Simplifying an R12 Install

Filed under: R12 — kkempf @ 7:18 am

Pre-validation RPMs

I’d discussed installing an RPM to check your kernel settings and do other trivial OS/Linux tasks before installing an Oracle database here.  Now Oracle has officially released the same mechanism to validate an R12 (12.1, 12.2) environment; I caught wind of it on Steven Chan’s blog here.  This is a nice touch, since the requirements (especially on the application server side) include many x86 packages which are a pain to get.  You can read about it here in Doc ID 1330701.1.  Essentially, you ensure your yum settings allow you to be subscribed to Oracle add-on’s (I’m assuming Oracle Linux here, if you’re not on OL I guess you could point to the public repositories?).  

yum install oracle-ebs-server-R12-preinstall 



April 16, 2014

Password Reset Error

Filed under: 11i, Cloning, R12 — kkempf @ 12:31 pm

Just minding my business

I received an email saying the following error occurs in an 11i cloned instance.  Mind you, this is because we added users to the environment, and by rule Oracle requires them to change their password the first time they login so the password is different than what the system administrator assigned.

Internet Explorer error message:



I’ll spell out a few keywords to the search engines can index this one: AppsChangePassword.jsp java.lang.NoClassDefFoundError JSP Error

If you’re running Chrome, here’s your error message





That Was Supposed to be a Joke

From Chrome, I just got a white page, no error, no nothing.  I had to debug this from IE, which just pains me.  Oh yeah, I’m not supposed to use Chrome for Ebusiness suite, even though it works fine and I’ve been using it from a Linux desktop for 8 years now.  Incidentally, there’s a plug in for Chrome to make R12 work, it turns out it does work fine though I have some privacy concerns… you can find it here.

The Fix

First I tried to bounce apache, but that did nothing, so I opened an SR.  I think I found this fix on MOS before the analyst gave it to me, but it didn’t seem like a great fit based on the description in the doc so I didn’t do it until he told me to.  Hat tip to MOS and the ATG team, they identified a fix for an issue quickly and without asking me 10 irrelevant questions first.  Anyways, run this script

$JTF_TOP/admin/scripts/ –compile -s ‘AppsChangePassword.jsp’ –flush

March 11, 2014

A Fun Diversion?

Filed under: Uncategorized — kkempf @ 11:01 am

What do others see on your resume?

I ran across a suggestion today to see what a recruiter sees on your resume. It’s kind of a fun exercise; just copy your resume to the clipboard and paste it in. It looks a LOT prettier on their website…

Here’s where mine ended up:

February 26, 2014

Controlling Data Guard Replication Lag

Filed under: Uncategorized — kkempf @ 12:35 pm


Tuning Standby Lag with Oracle Active Data Guard

We bought active data guard because of increasing reporting demands on our primary database in our ERP environment. While active data guard doesn’t play nicely with Apps 11i out of the box (it’s read-only, and just to establish a forms session you need to be able to write), it can fill a nice role offloading CPU load by serving up near-real time reports for specific (in our case bolt on) applications which only need to read tables.

What is Near-Real Time?

Once an SLA is established for the “oldest” data the report can return, you can tweak Oracle to honor it. In my case, the example below shows me going from no lag target to 300 seconds and finally to 600 seconds. Note that once you settle on a number, you’d be wise to add a “scope=both” to the end of the alter system.

$ sqlplus / as sysdba SQL*Plus: Release Production on Wed Feb 26 10:26:49 2014

Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
 Oracle Database 11g Enterprise Edition Release - 64bit Production
 With the Partitioning, OLAP, Data Mining and Real Application Testing options
 SQL> show parameter archive_lag_target;

 ------------------------------------ ----------- ------------------------------
 archive_lag_target integer 0
 SQL> alter system set archive_lag_target = 300;
 System altered.
 SQL> alter system set archive_lag_target = 600;
 System altered.

Lag Behavior

I found it interesting to note that left to “its own devices” the pattern of archivelog ships (and therefore application on the other end)becomes an inverse function of how active your database is. In other words, the recency of your data at your standby is related to how fast you fill your online redo logs (which is also a function of how big they are), plus the odd twist of system-driven logfile switches. Lets say you had 200MB redo logs with a nearly idle system. Your lag can get huge if not tuned!

The graphic below captures the result of SQL:
select to_number(substr(value,instr(value,':',1,2)+1,length(value))) + 60 * to_number(substr(value,instr(value,':',1,1)+1,2)) seconds from v$dataguard_stats@apps_to_dataguard where name = 'apply lag';


The left part of the graphic (before 10:21) shows data transport left untuned. From 10:21 to 11:21 you can see where I had it set to 300 seconds.
From 11:21 onward it’s set to 600 seconds.

Near-Real Time on Steroids: Real Time Apply

Check your licensing, your mileage may vary. The easiest way to keep the standby up to date is to use real-time apply and standby logs. To create standby logs, you go to your standby and cancel recovery:

alter database recover managed standby database cancel;

Next, add your standby logs. They need to be the same size as the online redo logs on the primary. Make N+1 of them on the standby. The syntax looks like this:

alter database add standby logfile group 41 ('/usr/local/oracle/redo/log41b.dbf','/u04/appprod/proddata/log41a.dbf') size 100M;
alter database add standby logfile group 42 ('/usr/local/oracle/redo/log42b.dbf','/u04/appprod/proddata/log42a.dbf') size 100M;
alter database add standby logfile group 43 ('/usr/local/oracle/redo/log43b.dbf','/u04/appprod/proddata/log43a.dbf') size 100M;
alter database add standby logfile group 44 ('/usr/local/oracle/redo/log44b.dbf','/u04/appprod/proddata/log44a.dbf') size 100M;

Finally, restart your recovery with the real-time apply:

alter database recover managed standby database disconnect using current logfile;
Now your apply lag and transport lag drop to zero:


In the graph above, you can see the final version of data apply rates to the standby.

8:58a-10:21a: unmanaged apply rate (mostly happening when a log on the primary got full)

10:21a-11:21a: honoring the 300 second alter system set archive_lag_target=300;

11:21a-4:30p: honoring the 600 second alter system set archive_lag_target=600;

4:30p-end: real time apply started with standby logfile groups (effectively 0).

Thanks Roth!

Thanks Roth!

Older Posts »

Theme: Silver is the New Black. Get a free blog at


Get every new post delivered to your Inbox.

Join 30 other followers