Kevin Kempf's Blog

February 25, 2015

R12 Take Aways and Warnings, Part 3 (Java Clients, Dataguard with editioning, logs, crons and the Workflow Notification Mailer)

Filed under: 11g, Dataguard, R12.2 — kkempf @ 1:58 pm


More R12.2 War Stories

I thought I’d go back to my R12.2 war stories today.  To be honest, I still have plenty to talk about even after this post, but a journey of a thousand miles begins with one step, right?

Java Clients

What a pain in the 4th point of contact.  We went into this upgrade knowing we wanted to deploy the most advanced Java client to the desktop (for forms) we could, so we didn’t have to deal with it.  Wow it’s a terrible hot mess.  We settled on 1.7.67.  Oracle, could we just call it 7.67?  Why do we need a 1 in front?  Sorry back to the topic at hand.  Java 7 is smart, and won’t run unsigned Jar files without much ado and many, many user clicks.  This is all good in the name of security, I suppose, but it sucks to administer.  At one point here I talked about the problems that arise.  Oracle tries to help mitigate this with a new product called Java Advanced Managment Console but it’s really new and really doesn’t seem to fully meet our needs.  So bottom line here: figure out how to get a java code signing certificate from your favorite CA (Thawte worked great for us) and what the new rules are for signing jar files.  You can search Enhanced Jar Signing on MOS, really you just need to understand adjkey to build the key, pick your CA, and use adjkey again to import the certificate.  Then you run adadmin (1,4) to regenerate jar files.


We utilize active dataguard with a physical standby against our Linux x86_64 database.  You may think Dataguard is a curious topic for an R12 upgrade.  The truth is that the upgrade itself has little to do with dataguard; in fact I disabled it before starting the upgrade because a) I didn’t want to ship hundreds of gigabytes of data down the pipe and b) I had no faith it would actually work.  After I was done with the upgrade I rebuilt the dataguard environment using RMAN, and it crashed every week for 6 weeks.  There’s a few known issues out there, but the error manifested itself like this:

Sat Dec 13 14:17:30 2014
Errors in file /u01/appprod/oracle/proddb/diag/rdbms/proddg/PROD/trace/PROD_lgwr_24265.trc:
ORA-04021: timeout occurred while waiting to lock object
LGWR (ospid: 24265): terminating the instance due to error 4021
Sat Dec 13 14:17:30 2014
System state dump requested by (instance=1, osid=24265 (LGWR)), summary=[abnormal instance termination].
System State dumped to trace file /u01/appprod/oracle/proddb/diag/rdbms/proddg/PROD/trace/PROD_diag_24236_20141213141730.trc
Dumping diagnostic data in directory=[cdmp_20141213141730], requested by (instance=1, osid=24265 (LGWR)), summary=[abnormal instance termination].
Instance terminated by LGWR, pid = 24265

And now you have no dataguard!  After trying various patches (17588480, 19631234, and 16299727), it appears that the real problem was that dataguard didn’t handle editioning well and the answer was patch 16299727 and adding

_adg_parselock_timeout=500  # dataguard editioning fix
event=”16717701 trace name context forever, level 104887600″ # dataguard fix

to my pfile/spfile fixed the issue.


There’s lots of logs in R12.2.  Pretty safe, obvious statement, right?  Well you need to go figure out where they are, because they all moved in 12.2.  If you don’t, well you’re gonna have a mess or possibly run out of disk.  I cron’d up a bunch of find commands to get rid of them; obviously don’t paste blindly, you need to tweak the retentions to your needs.  Some of these are supposed to be covered by cleanup jobs, but I find this is a nice insurance policy:

# Remove old outputs $APPLCSF/out or /u02/appprod/fs_ne/inst/PROD_hostname/logs/appl/conc/out
0 8 * * 1 /usr/bin/find /u02/appprod/fs_ne/inst/PROD_hostname/logs/appl/conc/out -mtime +30 -type f -exec rm -rf {} \;
# Remove old reqs $APPLCSF/log or /u02/appprod/fs_ne/inst/PROD_hostname1/logs/appl/conc/log
0 8 * * 1 /usr/bin/find /u02/appprod/fs_ne/inst/PROD_hostname/logs/appl/conc/log -mtime +30 -type f -exec rm -rf {} \;
# Remove Weblogic oacore Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server4/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oacore_server4/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic Admin Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic Forms-c4ws Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms-c4ws_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms-c4ws_server1/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic oafm Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oafm_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/oafm_server1/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic forms Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server1/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server2/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server3/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server4/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/forms_server4/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic wlst Server Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/logs -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/logs -mtime +7 -type f -exec rm -rf {} \;
# Remove Weblogic OHS Logs
0 8 * * 1 /usr/bin/find /u02/appprod/fs1/FMW_Home/webtier/instances/EBS_web_PROD_OHS1/diagnostics/logs/OHS/EBS_web_PROD -mtime +7 -type f -exec rm -rf {} \;
0 8 * * 1 /usr/bin/find /u02/appprod/fs2/FMW_Home/webtier/instances/EBS_web_PROD_OHS1/diagnostics/logs/OHS/EBS_web_PROD -mtime +7 -type f -exec rm -rf {} \;


While we’re on the subject of cleaning up logs, let’s talk about crons.  If it weren’t for cron, my R12.2 environment would not run.  Period.  Besides cleaning up logs, I have crons to do things like

  • rotate MSCA ports (telnet server)
  • bounce MSCA (daily!  it’s horribly unstable)
  • monitor various R12 related things from the OS (for example, use curl in a shell script to ensure the front end login page is up)
  • bounce JVMs every week (really, you need to do this)
  • cleanup apps sessions which otherwise won’t timeout or die (more on this later, worst surprise of the upgrade)
  • preclone the apps tier

Workflow Notification Mailer

If you’re using this, especially if your mailer accepts IMAP/inbound email responses as a part of the workflow, make sure you understand how it looks in 12.2.  It does not survive the upgrade from 11i, and the screens are totally different.  As far as I can tell, they don’t require any new or additional information to work, they just changed around the configuration screens so as to confuse you.  Bottom line is this: it mostly works once you get it configured right, but you should be aware that there’s a patch 18842914 for the mailer which greatly improved its reliability.  I put that in because it kept crashing when some 3rd party server was spamming it with email bounces (yeah, I’m not going into that any further).

Coming Soon

I still haven’t covered the most annoying things… like adop, the techstack, custom tops, MSCA in general, report manger replacing the ADI desktop client, and having to manually kill sessions to keep the database from dying…

Create a free website or blog at