Kevin Kempf's Blog

June 18, 2013

Going live on MSCA

Filed under: 11i, Linux, Mobile Supply Chain, Utilities — kkempf @ 3:20 pm

index

Oracle Mobile Supply Chain Application is live!

So we’re live on MSCA now. All things considered it went fine; from a technical perspective the issue which surprised me the most was the instability of the mwa server on the apps tier. I set up a dispatcher with 3 ports (plenty far apart) and at 5am I got a call from Ireland saying that there was only one device in the whole plant which could still connect.

AUTHENTICATION FAILED

The users weren’t getting their password wrong every time, though the message is the same.  Everyone got this “AUTHENTICATION FAILED” message. I’d seen this before; basically it meant that the port was hosed.

The behavior was such that even trying to do a stop_force against a “dead port” returned the message AUTHENTICATION FAILED.  Here’s a snip of my command line sequence to fix it in the middle of the night; basically I tried to gracefully shut down ports 10200, 10300 and 10400. 10300 and 10400 wouldn’t play nicely, so I had to find their PID and kill -9 on them, then restart all 3 ports:

$ cd $MWA_TOP/bin 
$ ps -ef|grep 10200 
oraprod  17027     1  0 Jun16 ?        Sl     0:48 /opt/jdk/bin/java -DPID=17021 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10200 
oraprod  21554 19963  0 05:00 pts/3    S+     0:00 grep 10200 
$ ps -ef|grep 10300 
oraprod  17059     1 30 Jun16 ?        Sl   742:11 /opt/jdk/bin/java -DPID=17053 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10300 
oraprod  21580 19963  0 05:00 pts/3    S+     0:00 grep 10300 
$ ./mwactl.sh -login apps/password stop_force 10200 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Telnet server shut down successfully. 
$ ./mwactl.sh -login apps/password stop_force 10300 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Error: ServerManagerListener returned 'AUTHENTICATION_FAILED' 
mwactl: Error shutting down Telnet server 
$ ./mwactl.sh -login apps/password stop_force 10400 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Error: ServerManagerListener returned 'AUTHENTICATION_FAILED' 
mwactl: Error shutting down Telnet server 
$ ps -ef|grep 10300 
oraprod  17059     1 30 Jun16 ?        Sl   742:57 /opt/jdk/bin/java -DPID=17053 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10300 
oraprod  22006 19963  0 05:01 pts/3    S+     0:00 grep 10300 
$ kill -9 17059 
$ ps -ef|grep 10400 
oraprod   7845     1  0 Jun17 ?        Sl     0:59 /opt/jdk/bin/java -DPID=7843 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10400 
oraprod  22063 19963  0 05:01 pts/3    S+     0:00 grep 10400 
$ kill -9 7845 
$ ps -ef|grep 10200 
oraprod  22169 19963  0 05:01 pts/3    S+     0:00 grep 10200 
$  ./mwactl.sh start 10200
Created server socket : listening on port 10200 
Server startup is successful.
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
$  ./mwactl.sh start 10300
Created server socket : listening on port 10300 
Server startup is successful.
$  ./mwactl.sh start 10400 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
$ Created server socket : listening on port 10400 
Server startup is successful.

mwactl.sh to the rescue

There’s no mwactl.sh status.  If there were, I could log in to the port, get the AUTHENTICATION FAILED error, kill -9 that PID and restart the telnet server on that same port.  So I have to do a “soft shutdown”.  Basically, I wrote a shell script to run every X hours via cron (I’ll start with 12).  It will bring up 3 new ports, and the old 3 ports will gracefully die (meaning, they’ll die after all users have disconnected).

Script to cycle ports with mwactl.sh gracefully

One note: the PASSWD line needs to be customized to your 8.0.6 Apache home.  The wdbsvr.app file contains a plain-text readable copy of your apps password.  At least in 11i.  You can also just hard code the value, but I’ll leverage Oracle’s security hole to my advantage and not propogate random hard coded passwords I have to change everywhere when I need to change the apps password.

#!/bin/bash
# mwa_port_roatate.sh   06/18/13 kmk

# Port Pool 1 : 10200, 10205, 10210, 10215, 10220, 10225

# Port Pool 2 : 10250, 10255, 10260, 10265, 10270, 10275

# mwa.TelnetServer=localhost.com:10200,10205,10210,10215,10220,10225,10250,10255,10260,10265,10270,10275

# Dispatcher: 10800

. /scratch/home/oraprod/env/PROD_APPS
  PASSWD=`cat /u02/appprod/prodias/iAS/Apache/modplsql/cfg/wdbsvr.app|grep -m1 password|sed -re 's/(^.+= )//'`
  USERNAME=`whoami`

if [ -f /tmp/mwa_port_pool1 ]
then
  # Port pool 1 is active, touch port pool 2 file and remove port pool 1 file
  rm -rf /tmp/mwa_port_pool1
  touch /tmp/mwa_port_pool2
 
  echo "====================STARTING=2=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Starting Port Pool 2" >> /tmp/mwa_port.log
  
  # start port pool 2
  echo "10250" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10250 >> /tmp/mwa_port.log 2>&1 &
  echo "10255" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10255 >> /tmp/mwa_port.log 2>&1 &
  echo "10260" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10260 >> /tmp/mwa_port.log 2>&1 &
  echo "10265" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10265 >> /tmp/mwa_port.log 2>&1 &
  echo "10270" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10270 >> /tmp/mwa_port.log 2>&1 &
  echo "10275" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10275 >> /tmp/mwa_port.log 2>&1 &

  sleep 10

  echo "====================STOPPING=1=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Stopping Port Pool 1" >> /tmp/mwa_port.log
  
  # stop port pool 1
  echo "10200" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10200 >> /tmp/mwa_port.log 2>&1 &
  echo "10205" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10205 >> /tmp/mwa_port.log 2>&1 &
  echo "10210" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10210 >> /tmp/mwa_port.log 2>&1 &
  echo "10215" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10215 >> /tmp/mwa_port.log 2>&1 &
  echo "10220" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10220 >> /tmp/mwa_port.log 2>&1 &
  echo "10225" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10225 >> /tmp/mwa_port.log 2>&1 &
 
  sleep 300

  for pids in `ps -fu ${USERNAME}|egrep '(10200)|(10205)|(10210)|(10215)|(10220)|(10225)'|grep -v grep|awk '{print ($2)}'`
  do
      echo "forcing kill on PID ${pids} (LIVE)" >> /tmp/mwa_port.log 2>&1 &
      kill -9 ${pids}
  done

else
# mwa.TelnetServer=localhost.com:10200,10205,10210,10215,10220,10225,10250,10255,10260,10265,10270,10275
  rm -rf /tmp/mwa_port_pool2
  rm -rf /tmp/mwa_port_pool1
  touch /tmp/mwa_port_pool1

  echo "====================STARTING=1=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Starting Port Pool 1" >> /tmp/mwa_port.log
  
  # start port pool 1
  echo "10200" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10200 >> /tmp/mwa_port.log 2>&1 &
  echo "10205" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10205 >> /tmp/mwa_port.log 2>&1 &
  echo "10210" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10210 >> /tmp/mwa_port.log 2>&1 &
  echo "10215" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10215 >> /tmp/mwa_port.log 2>&1 &
  echo "10220" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10220 >> /tmp/mwa_port.log 2>&1 &
  echo "10225" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10225 >> /tmp/mwa_port.log 2>&1 &

  sleep 10

  echo "====================STOPPING=2=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Stopping Port Pool 2" >> /tmp/mwa_port.log
  
  # stop port pool 2
  echo "10250" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10250 >> /tmp/mwa_port.log 2>&1 &
  echo "10255" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10255 >> /tmp/mwa_port.log 2>&1 &
  echo "10260" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10260 >> /tmp/mwa_port.log 2>&1 &
  echo "10265" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10265 >> /tmp/mwa_port.log 2>&1 &
  echo "10270" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10270 >> /tmp/mwa_port.log 2>&1 &
  echo "10275" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10275 >> /tmp/mwa_port.log 2>&1 &

  sleep 300

  for pids in `ps -fu ${USERNAME}|egrep '(10250)|(10255)|(10260)|(10265)|(10270)|(10275)'|grep -v grep|awk '{print ($2)}'`
  do
      echo "forcing kill on PID ${pids} (LIVE)" >> /tmp/mwa_port.log 2>&1 &
      kill -9 ${pids}
  done

fi

# cool command to return PIDS for all telnet servers
#
# ps -ef|egrep '(10200)|(10300)|(10400)|(10500)|(10600)|(10700)'|grep -v grep|awk '{print ($2)}'
Advertisements

May 6, 2010

Fun with adpatch

Filed under: 11i, Utilities — kkempf @ 10:21 am

Maintenance Time

This weekend I have a maintenance window (of course, on Mother’s Day) and hadn’t actually applied the approved patches in about 6 months.  Since I had the luxury of time to perform a “dry run” in an 11i sandbox environment, I went ahead and started with the patching.  The environment was a recent clone of production, so I didn’t anticipate any problems, but yet they found me.  I thought the nature of the error and the fix might be of interest.

Could not create the Java virtual machine

Yeah, what fun.  Shortly into the adpatch run, I hit the following error:

Running adjcopy.class:

 

adjava -mx512m -nojit oracle.apps.ad.jri.adjcopy @/u02/appmis/misappl/admin/MIS/out/apps.cmd

 

AutoPatch error:

ERROR  Running adjopy.class

 

AutoPatch error:

Error updating master archive

The Fix

What a mess.  The log pointed to the solution:

Unrecognized option: -nojit
Could not create the Java virtual machine.

A little (somewhat creative) digging on Metalink and I stumbled across ID 301542.1 which was a similar issue.
My $APPL_TOP/admin/adovars.env file had a bad environment variable:

#
# Set java program used by AD
#
ADJVAPRG=/opt/jdk/bin/jre (change jre to java)
export ADJVAPRG

To complete the fix, simply reinstantiate your environment (typically $APPL_TOP/APPSORA.env)

Minimum Baselines for 11i Extended Support

Incidentally, the patching I’m doing at this point is in response to 11.5.10.2 new minimum baselines (wouldn’t 11.5.10.3 have been a lot easier?) required for extended support for 11i beyond November 2010.

January 21, 2010

Fun with OPatch

Filed under: 10g, Oracle, Oracleisms, Utilities — kkempf @ 10:51 am

Since the January 2010 CPU patch came out recently, I began to apply the various pieces to my non-Production environments, as usual, just to ensure the process worked as normal.   OPatch is such a clumsy little tool, though I suppose it’s better than in the past when you had to manually pull about 20 readme’s and work for hours on the command line to apply security patches.  If you’re a glutton for bad memories, check out the application of Security Alert #68 or more specifically the database piece. It does make the current CPUs seem well polished by comparison.  But still, nothing ever goes as planned…

I go to apply PSU January 2010 (aka 10.2.0.4.3) on Linux x86-64 against a 10.2.0.4 database home  and I receive this error:

OPatch: ApplySession failed: Patch ID is null.

SEVERE:OPatch invoked as follows: 'apply '
INFO:
Oracle Home       : /u01/highjump/highjumpdb/10.2.0
Central Inventory : /opt/oracle/oraInventory
from           : /etc/oraInst.loc
OPatch version    : 10.2.0.4.2
OUI version       : 10.2.0.4.0
OUI location      : /u01/highjump/highjumpdb/10.2.0/oui
Log file location : /u01/highjump/highjumpdb/10.2.0/cfgtoollogs/opatch/opatch2010-01-20_09-48-35AM.log
INFO:Starting ApplySession at Wed Jan 20 09:48:36 EST 2010
INFO:Starting Apply Session at Wed Jan 20 09:48:36 EST 2010
SEVERE:OUI-67073:ApplySession failed: Patch ID is null.
INFO:System intact, OPatch will not attempt to restore the system
INFO:Finishing ApplySession at Wed Jan 20 09:48:36 EST 2010
INFO:Total time spent waiting for user-input is 0 seconds.  Finish at Wed Jan 20 09:48:36 EST 2010
INFO:Stack Description: java.lang.RuntimeException: Patch ID is null.
INFO:StackTrace: oracle.opatch.PatchObject.getPatchID(PatchObject.java:543)
INFO:StackTrace: oracle.opatch.ApplySession.loadAndInitPatchObject(ApplySession.java:1485)
INFO:StackTrace: oracle.opatch.ApplySession.process(ApplySession.java:5189)
INFO:StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:1588)
INFO:StackTrace: oracle.opatch.OPatch.main(OPatch.java:619)

Super.  What a nice, descriptive error.  A little hunting on MOS reveals that my OPatch version in the $ORACLE_HOME isn’t up to date.  Why couldn’t the error message be “Your version of OPatch is too old to support this patchset”??  Well this was remedied easily enough by downloading/upgrading to the latest version of 10.X OPatch.

Now the patch (9119284) progresses far enough to hit   a new issue:

Patch 7580744:
Archive Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/lib/libcore10.a/ldm.o" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot update file '/u01/highjump/highjumpdb/10.2.0/lib/libcore10.a' with '/ldm.o'
Archive Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/lib/libcore10.a/sldigpts.o" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot update file '/u01/highjump/highjumpdb/10.2.0/lib/libcore10.a' with '/sldigpts.o'
Archive Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/lib32/libcore10.a/ldm.o" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot update file '/u01/highjump/highjumpdb/10.2.0/lib32/libcore10.a' with '/ldm.o'
Archive Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/lib32/libcore10.a/sldigpts.o" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot update file '/u01/highjump/highjumpdb/10.2.0/lib32/libcore10.a' with '/sldigpts.o'
Copy Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/oracore/zoneinfo/timezone.dat" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot copy file from 'timezone.dat' to '/u01/highjump/highjumpdb/10.2.0/oracore/zoneinfo/timezone.dat'
Copy Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/oracore/zoneinfo/timezlrg.dat" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot copy file from 'timezlrg.dat' to '/u01/highjump/highjumpdb/10.2.0/oracore/zoneinfo/timezlrg.dat'
Copy Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/oracore/zoneinfo/readme.txt" does not exist.
'oracle.oracore.rsf, 10.2.0.4.0': Cannot copy file from 'readme.txt' to '/u01/highjump/highjumpdb/10.2.0/oracore/zoneinfo/readme.txt'
Archive Action: Source file "/u01/highjump/highjumpdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23/files/lib/libserver10.a/prm.o" does not exist.
'oracle.rdbms, 10.2.0.4.0': Cannot update file '/u01/highjump/highjumpdb/10.2.0/lib/libserver10.a' with '/prm.o'
DST10
Prerequisite check "CheckRollbackable" on auto-rollback patches failed.

Super.  Now my inventory of Rollbackable (wtf is that a word?) patches is messed up.  I chose option 3 from Doc 751107.1 and just copied the stupid files from another Oracle Home on the server:

$ cd /u01/highjump/highjumpdb/10.2.0/.patch_storage/
$cp -R /u01/labworks/labworksdb/10.2.0/.patch_storage/7580744_Dec_10_2008_23_06_23 .

Finally, OPatch grudgingly applied the PSU patch.  Cranked up the database, ran $OH/rdbms/admin/catbundle.sql psu apply I was done.

October 21, 2009

Licensing Products in 11i: adlicmgr.sh vs. OAM license manager

Filed under: 11i, Utilities — kkempf @ 2:28 pm

We’re looking at implementing an 11i product which we purchased but never implemented, and as I began to look at the setup steps from an administrator standpoint, I became re-acquainted with an license manager.  As the cobwebs shook free, it began something like this…

adlicmgr.sh is the command line tool used in the olden days, which requires a X session and command prompt access to the applications server.  In order to avoid problems, you first export the classpath, change to the right directory, and then launch it:

$ export CLASSPATH=$AD_TOP/java/adlicmgr.zip:$CLASSPATH
$ cd $AD_TOP/bin
$ ./adlicmgr.sh

Oracle Applications License Manager (Java)

Oracle Applications License Manager (Java)

From here, you wander through a few screens and try to find the product you wish to license, or, alternatively, you can review what is already licensed.

License or Report

License or Report

Good luck finding your product!

Good luck finding your product!

I happened to be looking for iProcurement, and it was nowhere to be found on the list.  To make things worse, the list isn’t even sortable!

I gave up on this and went into the OAM version of license manager… what an improvement!  The URL for OAM is, as a reminder:

https or http://appserverhostname:port/servlets/weboam/oam/oamLogin

OAM Site Map

OAM Site Map

From here, you get to this screen; I used the Products link under License at this point…

Screenshot-Oracle Applications Manager - Mozilla Firefox

Screenshot-Oracle Applications Manager - Mozilla Firefox-1

From here, I landed at this screen which had a few options.

Despite my screen shot, the correct selection to simply add 1 product is License Component Application.

List of Components Available to License

List of Components Available to License

From here, it’s just follow the bouncing ball; select the component by checking it, hit next, next and you are told:

Success

Success

That’s all there was to it… if I was using the command line adlicmgr.sh program, I believe I’d still be looking for the product… or it could be that it’s just not as updated with the new product names and abbreviations as OAM is.

I should make one final disclaimer: Just because you’re able to license a product doesn’t mean you’re legally able to use it.  In other words, if you bought the 11i ERP for one product, there’s nothing to stop you from illegally licensing the whole suite.  Except an army of Oracle lawyers if they ever catch you in an audit…

August 24, 2009

11i Stub Library: libc-2.1.3-stub.so (aka Concurrent Managers Won’t Start with GSM=Y) Finally Fixed!

Filed under: 11i, Cloning, Concurrent Managers, Linux, Oracle, Utilities — Tags: , — kkempf @ 10:06 am

I’ve been working this week on setting up my regression TEST environment for an upcoming cycle; among the patches I’m testing are ATG_PF_H RUP7, AD.I.7 and the latest autoconfig templates.  It never fails when I apply core framework technology: my 8.0.6 stub libraries get hosed.  By “getting hosed” I’m referring to various Signal 11 errors, or variants of these, which cause the concurrent managers to fail to start until you set GSM=N or fix it as I’ll detail further here.  But I got sick of applying 3830807 to fix it, and decided to dig deeper into the 8.0.6 homes…. here’s what I found….

As review, Developer 6i (Forms, Reports, CMs) in 11i is generally what people are talking about when they say $ORACLE_HOME or 8.0.6 Home on the application server.   The iAS_ORACLE_HOME is Application Server 9iR1 (technically 1.0.2.2.2 in my case) and runs Apache.

3830807 delivers 2 binaries and a shell script.  Basically the script goes out and fixes make files, if necessary, puts a few symbolic links in the directory, and lands the binaries in the 8.0.6 home under lib/stubs.

  • libc-2.1.3-stub.so ($8.0.6 Home/lib/stubs) needs to be 261328 bytes in size, and libc.so 71 bytes.  In my case, libc-2.1.3-stub.so was 131951 bytes and needed to be fixed.

First I had to understand what I was dealing with, and what caused it.  A quick scan of my environments showed all non-production environments had the wrong version, and they were all date stamped the date of the last clone.  It turns out that adcfgclone (adclone, autoclone, whichever you prefer) is the culprit.  But I’m on the latest everything as far as I know: ATG F RUP 6, ADI.7, TXK Autoconfig Templates T….

With help, I found that the setup_stubs.sh script in the $iAS Home (Apache Home, or $ORACLE_HOME/../iAS, generally, on the apps tier) was actually corrupting the libc-2.1.3-stubs.so file in the $8.0.6 Home/lib/stubs.  I could not figure out where the 131951 byte version of libc-2.1.3-stubs.so was even coming from, but there it was, sitting in the iAS Home/lib/stubs, dated June of 2001.  Wow!  That’s old! Another piece I was able to confirm: unless adadmin executable recompiles are forced, or a patch specifically calls for a recompile, this “bad” libc-2.1.3-stubs.so can sit there silently waiting to affect you later.  When I looked at all my cloned environments, they all had the “bad” version of the file, but because I hadn’t happened to manually force an adadmin executable relink (and why would I, under normal circumstances?) my concurrent managers and all other FND binaries still worked.

Now we’re getting somewhere… but I still don’t know one fundamental question: can I safely patch the iAS Home with a newer version of libc-2.1.3-stubs.so?

At this point, I turned to Oracle support..  It’s been there since… I’ll update this entry as I make progress.

Relevant Notes: 465629.1, 847775.1

Related Patch: 3830807

* edit* 9/2/09: still waiting on word from support…

* edit* 9/23/09: Support confirmed I can overwrite the $IAS_HOME/lib/stubs version (131951 byte) of the file libc-2.1.3-stubs.so with the (newer) version (261328 bytes) of the file from $8.0.6 Home/lib/stubs:

UPDATE
=======
Development confirm that you can safely copy the newer file over the older file.

QUESTION
=========
Can the older (131951 byte) version of libc-2.1.3.sub.so in the iAS home be overwritten by the one delivered with patch:3830807 (261328 bytes)?

ANSWER
========
Yes, based on reasearch this appears to be the intended situation. So please copy the newer file over the older file. This should then be a permanent fix for this problem.

August 17, 2009

11i Patch Merging

Filed under: Oracle, Utilities — Tags: , , — kkempf @ 12:59 pm

Twice a year before regression test cycles, I spend an inordinate amount of time researching the latest round of patches and their pre-requisites. I know there’s got to be a better way, but in my experience I just don’t trust Oracle’s automated methods completely.

As an example, just last week, I was re-applying an interoperability patch for native java (6863618) because I couldn’t recall if I’d applied it in a particular environment. I went into OAM under maintenance and punched in the patch number, it came back with no rows. Fair enough, I applied the patch. Queried OAM again – came back with no rows. Hence my lack of trust…

Regardless, life is simpler with patch merging, known on the command line as admrgpch.

Once I figure out logical groups for patching (for example, Inventory, WIP, etc) I put them into the same staging directory, and merge them. This way, when I have to run this in production, I don’t have to run 12 patches to complete one patch and its 11 pre-requisites. The basic syntax is as follows:

admrgpch -s /scratch/patches11i/test/Fall09_Regression/Functional/INV_RUP4 -d /scratch/patches11i/test/Fall09_Regression/Functional/INV_RUP4/merge -merge_name INV_RUP4_merge_Fall09

The -s is the source directory, where all the patches have been staged and unzipped. The -d directory is the destination, which I just make a subdirectory of the staging directory. The merge_name is just what it will ulitmately call the universal driver (in this case, u_INV_RUP4_merge_Fall09.drv). Regardless, this is a much more efficient way of bundling patches and I strongly recommend it if you’re not using it.

As I wrote about the universal driver, I had a weird recollection of the hell that was patching before universal drivers. Anybody remember the 3 types of patch drivers, for 10 geek points?

They were copy drivers, database drivers, and generate drivers. Copy did the binary copies & relinks, database ran the sql/plsql/java against the RDBMS, and generate rebuilt/compiled the plls, forms and reports. Those were the slow old days….

July 9, 2009

What’s your DBID?

Filed under: 11g, Oracle, RMAN, Utilities — Tags: , — kkempf @ 3:11 pm

I started reviewing recovery scenarios and realized that if I ever had to do a complete database loss recovery without my recovery catalog, determining my DBID would be problematic. So I did a little shell scripting and came up with a simple solution at the OS level. This is written for bash, on Linux, and assumes you can receive email from the server. A simple way to confirm this is to type echo “Test” | mail -s “Email Test” dba@yourcomany.com and see if it gets through. If it doesn’t, check to see if sendmail is running (#service sendmail status), or consult your system admins. Anyway, here’s the script:

cat incarnation.sh

# source your ORACLE_HOME that rman runs out of, however you do that
. /u01/appprod/oracle/proddb/11.1.0/PROD_myserver.env
RCATUSER=whatever your recovery catalog user is
RCATPASS=whatever your recovery catalog user password is
DBA=your email
# really only necessary if you’re not running this script out of the recovery catalog home
REPSID=SID of your recovery catalog

echo “list incarnation;”|rman catalog ${RCATUSER}/${RCATPASS}@${REPSID} | mail -s “RMAN DBIDs” $DBA

Now you can cron it up (crontab -e) and in the case below you’ll be updated every weekday @ 8am:
# email me the DBIDs every weekday
0 8 * * 1-5 /scratch/oracle/dba/scripts/incarnation.sh > /dev/null

Your email will look something like this:

Recovery Manager: Release 11.1.0.7.0 - Production on Thu Jul 9 15:39:21 2009
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
connected to recovery catalog database
RMAN>
List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
23832   23839   ADV_TRN  292427552        PARENT  1          05-FEB-06
23832   23833   ADV_TRN  292427552        CURRENT 440636     02-MAY-07
46144   46151   ADV_TEST 491394006        PARENT  1          05-FEB-06
46144   46145   ADV_TEST 491394006        CURRENT 440636     11-JUL-07
46214   46221   AAD_TEST 914518601        PARENT  1          05-FEB-06
46214   46215   AAD_TEST 914518601        CURRENT 440636     12-JUL-07
...

There may be a more elegant way to do this, but I couldn’t find one. I would have preferred this to be an EM job, except there is no option under RMAN scripts to connect to the recovery catalog only (you must select at least 1 target).

July 7, 2009

Java Color Scheme (POV) change after a clone

Filed under: Oracle, Utilities — Tags: , , , , — kkempf @ 12:50 pm

After a clone, I like to change the Java Color Scheme (Profile Option) of the new environment. This is mostly a visual queue for myself and anyone using the environment. First and foremost, it helps you confirm you’re not in PROD. Since I consistently use the same colors for each environment each time, it also helps confirm you’re in the right flavor of DEV/TRAIN/TEST etc.

After a clone, I got tired of always having to log in to the new environment and update this in the GUI, and thought this script might be helpful to others.  

declare
  cursor c2 is
    select
        fu2.user_id   
       ,fpo.profile_option_name pon
       ,fpot.user_profile_option_name upon
       ,fu2.user_name lov
       ,fpov.profile_option_value pov
    from
        fnd_profile_options_tl fpot
       ,fnd_profile_options fpo
       ,fnd_profile_option_values fpov
       ,fnd_user fu
       ,fnd_user fu2
    where
        fpot.user_profile_option_name = 'Java Color Scheme' 
    and
        fpot.profile_option_name = fpo.profile_option_name
    and
        fpo.profile_option_id = fpov.profile_option_id
    and
        fpo.created_by = fu.user_id
    and
        fpov.level_id = 10001  /* site (10004=user, 10001=site, 10002=Appl, 10003=Resp) */
    and
        fpov.level_value = fu2.user_id
    and
        fpot.language = Userenv('Lang')
    ;
begin
  dbms_output.disable;
  dbms_output.enable(100000)
  ;
  for c2_rec in c2 loop
    status := fnd_profile.save(c2_rec.pon,'&New_Color_Scheme_lower','SITE');
    if status then
      dbms_output.put_line('Java Color Scheme Updated');
    else
      dbms_output.put_line('Java Color Scheme FAILED');
    end if
    ;
    commit
    ;
  end loop
  ;
end
;
/

incidentally, valid values to enter are as follows:

blaf
blue
khaki
olive
purple
red
teal
titanium

As you can see, if you use your imagination, you can modify this script to update virtually any system profile option value.  If you have a user value which needs adjusting, it changes slightly in that a new parameter is required to pass to fnd_profile.save .  This is the user_id (in the c2_rec loop for your convenience).  For example, it might look like this:

status := fnd_profile.save(c1_rec.pon,null,’USER’,c1_rec.user_id);

Well anyways, enjoy, let me know what you think.

Create a free website or blog at WordPress.com.