Kevin Kempf's Blog

August 7, 2014

Changing the JRE version pushed by R12.2

Filed under: Java, R12.2 — kkempf @ 12:02 pm

Giving credit where credit is due

In light of all the recent fervor about Java, it’s appropriate that I give credit to Oracle when they get something right.

Changing JRE push version (minimum) in 12.2

It’s unbelievably easy.  Follow 393931.1

  • Download whatever version of Java you want to run this week for your enterprise </sarcasm> to /tmp
    • The way they’re pushing updates, I think I could script this whole thing to pull the latest version every week and install it….
  • cp /tmp/jre-7u67-windows-i586.exe $COMMON_TOP/webapps/oacore/util/javaplugin/j2se17067.exe
    • land it in the “push” directory
  • $FND_TOP/bin/ 17067
    • run this utility to link the AD utilities and your context file to the new version
  • adop phase=fs_clone
    • make sure it gets sync’d to both fs1 and fs2

Done.  That’s it.  10 minutes tops.  Your EBS now pushes 1.7 update 67.

R12.2, Oracle Java, Microsoft, Google and Security

Filed under: Java, Oracle, Windows 7 — kkempf @ 11:51 am


The Playing Field

It’s no secret Google hates Microsoft (M$) hates Oracle hates Google hates Oracle hates Microsoft.  But EBS users are caught in the crossfire this time, and it’s not pretty.

M$ recently announced here they were going to start blocking Java launches out of IE.  You know, because they’re looking out for us.

Java started squalking about unsigned Jar files awhile back, but if your version was old enough, you wouldn’t see this message because it was too dumb to know it.  It also pesters you incessantly if you’re not running the latest version.  Always.

Google started stopping Java from running in Chrome which I noted here.  It was technically never certified with Oracle EBS, so I hold them least culpable at the moment.


I appreciate that all of these players are protecting their own interests and implementing these changes in the name of security.  People complain about Java security?  Great, now Oracle will issue an update every week, and if you’re more than 2 updates behind it will nag you forever.  Launching EBS which doesn’t have signed JAR files?  Big scary window pops up warning the user.  Want to run old Java on Windows 7?  Microsoft says nope.

I’m in the middle of an R12.2 upgrade.  I have the luxury of taking the time to go get a certificate from Thawte and sign my jar files, and test it.  The truth is that it’s pretty easy.  But so is the process of getting a certificate.  Meaning, nothing against Thawte, but by signing my jar files all I’ve done is satisfy the JRE plug in.  I haven’t really enhanced my security, in my opinion.  Same thing with upgrading to the latest version of Java.

Meanwhile, here in the real world

I work for a manufacturer.  The floor workers who interact with Oracle EBS don’t know/care/understand what Java is, what a signed jar file is, what version of Java they’re running, etc.  They just want to get into Oracle EBS.  We’re not internet-exposed.  Sure there’s risk of someone exploiting a security hole, but it would probably have to come from the inside.  This is a known risk we’re probably more willing to take than the risk of having to deal with hours and hours of confusion every time M$ makes a unilateral decision about Java, or Oracle delivers yet another Java update.   We’ve been running EBS 11i on Java 1.6_12 for over 5 years now.  Never caused a problem.  It’s ancient, but it works.  Much like, I suspect, many companies, we have a lot of legacy Java running.  The effort required to push a newer version of Java to these machines is not insignificant, since they’re all locked down on the manufacturing floor.

Battle of the sexes

I’m sick of it.  Java consumers (and I’m not talking about coffee here) are stuck in the crossfire between M$ and Oracle.  Here’s what I see happening:

1. We regression test and deliver R12.2 with Java 1.7.0 update 67 in 6 months

2. Oracle issues 25 more versions of Java 7 thus every user gets the “Update (Recommended), Block or Later” dialogue box

3. Some users block, some update, some say later.  66.66% of these scenarios cause IT problems

4. Microsoft, not to be outdone, declares Java 1.7 update 67 obsoleted, and thus adds it to a block list

Managing the Mess

Sure, we can manage Win7 updates through policy, and the same for Java updates.  But are we actually improving security, in any way?  Or just creating busy work for ourselves, where the version which runs on our isolated network is largely immaterial?



June 10, 2014

Let the games begin!

Filed under: 11i, R12.2, Ubuntu — kkempf @ 8:34 am


R12.2.3 Testing has begun

R12.2 changes a LOT of things, especially for an applications DBA.  We’re underway with our upgrade plans to 12.2, and I was happy to see if you have the patience to make all the pieces work (Chrome, Ubuntu, Java, PopUp Blocker, Chrome Plug-In for Oracle R12) you can actually launch forms from Ubuntu.

Initial Upgrade Impressions

There’s a lot of work to upgrade from 11i to 12.2.3.  The upgrade documents are extensive, and not light reading.  Customizations are a bugger to bring over, the whole landscape has changed with edition based redefinition.  Much of this revolves around the changes to the techstack (Weblogic), and online patching.  The new patching utility, adop, works in conjunction with other old favorite ad utilities, but it’s not terribly intuitive.  There was something reassuring watching my workers progress and run; maybe there’s a way to do it and I just haven’t figured out the appropriate CLI switch.  You can still look at them via adctrl, but it’s not the same, I don’t get the reassurance of knowing I only have 123,000 jobs left to finish.  Mostly, though, things look the same.  More to follow.

June 9, 2014

Google’s message to Ubuntu users of Chrome 35 and JRE

Filed under: Chrome, Ubuntu — kkempf @ 10:59 am


Take that Chrome Users!

OK, they really didn’t do that.  But they might as well have.

Ubuntu at Work

I like using Ubuntu as my primary desktop at work.  I have to keep Windoze around for compatibility issues, but Linux is much faster and more reliable.  Recently, I found that I kept getting the error “Java Plugin Not Installed”.  I finally had time to dig into it, and it turns out it was due to an upgrade from Chrome 34 to Chrome 35.  The sym link in my /opt/google/chrome/plugins directory was still good, but Chrome 35 just ignores it.  I found all this out (ironically) with a lot of Googling, I found this

I won’t say I understand all the jargon, but the gist of it is this: for whatever reason, Chrome on Linux no longer supports Oracle JDK/JRE as of a few weeks ago.  Here’s a funny thread showing me I’m not the only one unamused by this:

The Fix

If you find yourself in this spot, just use Ubuntu software center to uninstall Chrome, go pull version 34 here and reinstall.  Then it magically works again, and I can get into the EBS forms.  Google, I know you don’t care what I think, but seriously?  It’s Oracle Java.  Pretty universal, and this open source Iced Tea stuff doesn’t cut it for big boy applications.  Please restore this functionality, or I’ll have to move to the F-browser and nobody wants that!

April 24, 2014

Simplifying an R12 Install

Filed under: R12 — kkempf @ 7:18 am

Pre-validation RPMs

I’d discussed installing an RPM to check your kernel settings and do other trivial OS/Linux tasks before installing an Oracle database here.  Now Oracle has officially released the same mechanism to validate an R12 (12.1, 12.2) environment; I caught wind of it on Steven Chan’s blog here.  This is a nice touch, since the requirements (especially on the application server side) include many x86 packages which are a pain to get.  You can read about it here in Doc ID 1330701.1.  Essentially, you ensure your yum settings allow you to be subscribed to Oracle add-on’s (I’m assuming Oracle Linux here, if you’re not on OL I guess you could point to the public repositories?).  

yum install oracle-ebs-server-R12-preinstall 



April 16, 2014

Password Reset Error

Filed under: 11i, Cloning, R12 — kkempf @ 12:31 pm

Just minding my business

I received an email saying the following error occurs in an 11i cloned instance.  Mind you, this is because we added users to the environment, and by rule Oracle requires them to change their password the first time they login so the password is different than what the system administrator assigned.

Internet Explorer error message:



I’ll spell out a few keywords to the search engines can index this one: AppsChangePassword.jsp java.lang.NoClassDefFoundError JSP Error

If you’re running Chrome, here’s your error message





That Was Supposed to be a Joke

From Chrome, I just got a white page, no error, no nothing.  I had to debug this from IE, which just pains me.  Oh yeah, I’m not supposed to use Chrome for Ebusiness suite, even though it works fine and I’ve been using it from a Linux desktop for 8 years now.  Incidentally, there’s a plug in for Chrome to make R12 work, it turns out it does work fine though I have some privacy concerns… you can find it here.

The Fix

First I tried to bounce apache, but that did nothing, so I opened an SR.  I think I found this fix on MOS before the analyst gave it to me, but it didn’t seem like a great fit based on the description in the doc so I didn’t do it until he told me to.  Hat tip to MOS and the ATG team, they identified a fix for an issue quickly and without asking me 10 irrelevant questions first.  Anyways, run this script

$JTF_TOP/admin/scripts/ –compile -s ‘AppsChangePassword.jsp’ –flush

March 11, 2014

A Fun Diversion?

Filed under: Uncategorized — kkempf @ 11:01 am

What do others see on your resume?

I ran across a suggestion today to see what a recruiter sees on your resume. It’s kind of a fun exercise; just copy your resume to the clipboard and paste it in. It looks a LOT prettier on their website…

Here’s where mine ended up:

February 26, 2014

Controlling Data Guard Replication Lag

Filed under: Uncategorized — kkempf @ 12:35 pm


Tuning Standby Lag with Oracle Active Data Guard

We bought active data guard because of increasing reporting demands on our primary database in our ERP environment. While active data guard doesn’t play nicely with Apps 11i out of the box (it’s read-only, and just to establish a forms session you need to be able to write), it can fill a nice role offloading CPU load by serving up near-real time reports for specific (in our case bolt on) applications which only need to read tables.

What is Near-Real Time?

Once an SLA is established for the “oldest” data the report can return, you can tweak Oracle to honor it. In my case, the example below shows me going from no lag target to 300 seconds and finally to 600 seconds. Note that once you settle on a number, you’d be wise to add a “scope=both” to the end of the alter system.

$ sqlplus / as sysdba SQL*Plus: Release Production on Wed Feb 26 10:26:49 2014

Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
 Oracle Database 11g Enterprise Edition Release - 64bit Production
 With the Partitioning, OLAP, Data Mining and Real Application Testing options
 SQL> show parameter archive_lag_target;

 ------------------------------------ ----------- ------------------------------
 archive_lag_target integer 0
 SQL> alter system set archive_lag_target = 300;
 System altered.
 SQL> alter system set archive_lag_target = 600;
 System altered.

Lag Behavior

I found it interesting to note that left to “its own devices” the pattern of archivelog ships (and therefore application on the other end)becomes an inverse function of how active your database is. In other words, the recency of your data at your standby is related to how fast you fill your online redo logs (which is also a function of how big they are), plus the odd twist of system-driven logfile switches. Lets say you had 200MB redo logs with a nearly idle system. Your lag can get huge if not tuned!

The graphic below captures the result of SQL:
select to_number(substr(value,instr(value,':',1,2)+1,length(value))) + 60 * to_number(substr(value,instr(value,':',1,1)+1,2)) seconds from v$dataguard_stats@apps_to_dataguard where name = 'apply lag';


The left part of the graphic (before 10:21) shows data transport left untuned. From 10:21 to 11:21 you can see where I had it set to 300 seconds.
From 11:21 onward it’s set to 600 seconds.

Near-Real Time on Steroids: Real Time Apply

Check your licensing, your mileage may vary. The easiest way to keep the standby up to date is to use real-time apply and standby logs. To create standby logs, you go to your standby and cancel recovery:

alter database recover managed standby database cancel;

Next, add your standby logs. They need to be the same size as the online redo logs on the primary. Make N+1 of them on the standby. The syntax looks like this:

alter database add standby logfile group 41 ('/usr/local/oracle/redo/log41b.dbf','/u04/appprod/proddata/log41a.dbf') size 100M;
alter database add standby logfile group 42 ('/usr/local/oracle/redo/log42b.dbf','/u04/appprod/proddata/log42a.dbf') size 100M;
alter database add standby logfile group 43 ('/usr/local/oracle/redo/log43b.dbf','/u04/appprod/proddata/log43a.dbf') size 100M;
alter database add standby logfile group 44 ('/usr/local/oracle/redo/log44b.dbf','/u04/appprod/proddata/log44a.dbf') size 100M;

Finally, restart your recovery with the real-time apply:

alter database recover managed standby database disconnect using current logfile;
Now your apply lag and transport lag drop to zero:


In the graph above, you can see the final version of data apply rates to the standby.

8:58a-10:21a: unmanaged apply rate (mostly happening when a log on the primary got full)

10:21a-11:21a: honoring the 300 second alter system set archive_lag_target=300;

11:21a-4:30p: honoring the 600 second alter system set archive_lag_target=600;

4:30p-end: real time apply started with standby logfile groups (effectively 0).

Thanks Roth!

Thanks Roth!

February 24, 2014

Oracle RDBMS tier is Virtualized

Filed under: Uncategorized — kkempf @ 2:24 pm



So we’ve been running non-Production versions of our Oracle 11i E-Business suite environment on VMWare since about 2006, and my Linux x86 PAE 11i front end on VMWare since at least 2008.  I never opened a ticket or had any issue with VMWare affecting a guest OS in any way, whether the kernel was Red Hat, Oracle Linux “Red Hat compatible” or even now the UEK.  Oracle protects themselves with this (Doc ID 249212.1):

 Oracle has not certified any of its products on VMware virtualized environments. Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

The truth is, they try to scare you away from it, but it runs just fine, in my experience.  The big unknown was always the core database, running on 64-bit Linux.  It really does work hard, and has lots of moving parts.  As we began the project to convert it, we realized we had a lot of questions without firm answers, so we engaged a prominent 3rd party Oracle to VMWare integrator.  They did a great job managing the project, but to be honest, unbeknownst to us we’d already figured out most of the technical detail.

Project Flow

There was an extra wrinkle for our production go-live.  The server was physically moving from one data center to another one about 50 miles up the road.  We had good WAN links, but it still added some time.  While we considered leveraging dataguard to accomplish this, in the end we landed on an RMAN restore and Data Domain replication.

  • Create a VM with lots of CPUs and the same memory footprint as production.  Put in all the patches, updates, kernel parameters, and set up huge pages for the 64gb SGA.
  • Add disk from the SAN leveraging LVM and ext3.   We could have used ext4, but the Linux OS was Oracle Linux 5.10 (UEK) and ext3 felt more “tried and true”
  • Create a VMWare template at this point (this was actually done several times, after many, many OS tweaks)
  • Install the Oracle RDBMS software ( in my case) as well as deploying the EM agent
  • Use RMAN to bring the database to the new VM from the physical box.  In the case of test runs, this was an RMAN duplicate.  For the final, live production run it was a little trickier:
    • RMAN full backup of the RDBMS the night prior, so that full can get across the WAN and my incremental difference will be smaller
    • Shut down the 11i front end so users can’t get in
    • RMAN (hot) backup the RDBMS, shut it down
    • Wait for the Data Domain to replicate the RMAN backup and archivelogs to where the new server was
    • RMAN restore and recover the database
    • IP changes
      • Perform internal DNS changes to the new vlan
      • Bring up the 11i front end at the new location with a VEEAM restore
      • Re-IP the two machines, via /etc/hosts and /etc/sysconfig/network, as well as FND_NODES changes (see 751328.1)

Results & Advantages

Since the move to VMWare, we’ve had no issues whatsoever.  The database runs mostly in memory, and the users are none the wiser.  VMWare does bring some advantages to the table:

  • Unlike a physical machine, where I have to be in the data center at a console, or possibly over a DRAC card or the like, when I reboot my machine now, I can watch it in VCenter.
  • I don’t have to worry about drivers.  I had a serious issue when our oddball 10gb NICs decided to stop working after a yum update on the physical box during a reboot.
  • If the physical server (which in the case of the RDBMS is effectively the same thing as the ESX host, as the host is entirely devoted to running Oracle) running the RDBMS breaks, overheats, has a bad memory chip, burns up a CPU, or gets struck by lightening, we can shut it down cold and storage v-motion it to another host in minutes.
  • It’s a mainstream, mature product.  I did a lot of homework on this, and the general consensus was that Oracle VM wasn’t ready yet, but there’s LOTS of people running Oracle on VMWare.

December 31, 2013

2013 in review

Filed under: Uncategorized — kkempf @ 7:55 am

The stats helper monkeys prepared a 2013 annual report for this blog.   I guess I need to blog more.  Haven’t had as much time this year, nor relevant content.  Next year I have a few heavy hitters on the docket: leveraging active data guard, tearing down Discoverer 11g and replacing it with Oracle Apex (nearly done, by the way), virtualizing the main ERP Production database on VMWare, and (finally) an upgrade to R12.2.  While continuing to work on my MBA.  Should be fun, see you there!

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 49,000 times in 2013. If it were a concert at Sydney Opera House, it would take about 18 sold-out performances for that many people to see it.

Click here to see the complete report.

Older Posts »

The Silver is the New Black Theme. Blog at


Get every new post delivered to your Inbox.

Join 29 other followers