About Oracle Applications doens’t display Forms Server Information

After a recent upgrade to Forms 6.0.8.28, I expected to go into Help->About Oracle Applications and be rewarded with the new version number; instead I got the screen on the left below.

Did a little digging and it took me quite some time to find the answer. As of ATG RUP5, it seems this has been purposely hidden for security purposes via a profile option value. This is detailed in Note 438274.1. Setting FND: Diagnostics to “Yes” at the site level and bouncing Apache yields results I’m more accustomed to seeing, on the right below.

TEST environment, with FND: Diagnostics set to No

TEST environment, with FND: Diagnostics set to Yes

My Oracle Support v. Metalink Searches

There’s an Oracle-acknowledged bug in the “new” Metalink (aka My Oracle Support) which prevents me from being able to upload files for SR’s via their flash interface.  It has been tied to Firefox 3.X, and hasn’t worked since at least this spring.  Because of this, and also because it’s generally faster, I frequently log into the “old” Metalink.

One thing I’ve noticed: searches yield completely different results in the two different flavors of Metalink.   In general, the new, My Oracle Support (could they have come up with a worse name for this?) yields better results, in my opinion.  Case in point, a recent search of mine: In both sites, put “libc-2.1.3” under the search (All Sources).  In the old Metalink, I get 4 hits, dated from 2001-2006.   In My Oracle Support, I get 8 hits, 7 of which are dated 2007 or 2008.

My oldest complaint with Metalink/My Oracle Support is pretty fundamental.  How is it that when I put a document ID in the search bar as the only criteria, it never comes back as the #1 result of the search?  I know I can do an advanced search and try to force the issue… but why should I have to?  If this was the only thing they fixed then they migrated to this new, flash based site, I would have readily embraced it.

11i Stub Library: libc-2.1.3-stub.so (aka Concurrent Managers Won’t Start with GSM=Y) Finally Fixed!

I’ve been working this week on setting up my regression TEST environment for an upcoming cycle; among the patches I’m testing are ATG_PF_H RUP7, AD.I.7 and the latest autoconfig templates.  It never fails when I apply core framework technology: my 8.0.6 stub libraries get hosed.  By “getting hosed” I’m referring to various Signal 11 errors, or variants of these, which cause the concurrent managers to fail to start until you set GSM=N or fix it as I’ll detail further here.  But I got sick of applying 3830807 to fix it, and decided to dig deeper into the 8.0.6 homes…. here’s what I found….

As review, Developer 6i (Forms, Reports, CMs) in 11i is generally what people are talking about when they say $ORACLE_HOME or 8.0.6 Home on the application server.   The iAS_ORACLE_HOME is Application Server 9iR1 (technically 1.0.2.2.2 in my case) and runs Apache.

3830807 delivers 2 binaries and a shell script.  Basically the script goes out and fixes make files, if necessary, puts a few symbolic links in the directory, and lands the binaries in the 8.0.6 home under lib/stubs.

• libc-2.1.3-stub.so ($8.0.6 Home/lib/stubs) needs to be 261328 bytes in size, and libc.so 71 bytes.  In my case, libc-2.1.3-stub.so was 131951 bytes and needed to be fixed.

First I had to understand what I was dealing with, and what caused it.  A quick scan of my environments showed all non-production environments had the wrong version, and they were all date stamped the date of the last clone.  It turns out that adcfgclone (adclone, autoclone, whichever you prefer) is the culprit.  But I’m on the latest everything as far as I know: ATG F RUP 6, ADI.7, TXK Autoconfig Templates T….

With help, I found that the setup_stubs.sh script in the $iAS Home (Apache Home, or $ORACLE_HOME/../iAS, generally, on the apps tier) was actually corrupting the libc-2.1.3-stubs.so file in the $8.0.6 Home/lib/stubs.  I could not figure out where the 131951 byte version of libc-2.1.3-stubs.so was even coming from, but there it was, sitting in the iAS Home/lib/stubs, dated June of 2001.  Wow!  That’s old! Another piece I was able to confirm: unless adadmin executable recompiles are forced, or a patch specifically calls for a recompile, this “bad” libc-2.1.3-stubs.so can sit there silently waiting to affect you later.  When I looked at all my cloned environments, they all had the “bad” version of the file, but because I hadn’t happened to manually force an adadmin executable relink (and why would I, under normal circumstances?) my concurrent managers and all other FND binaries still worked.

Now we’re getting somewhere… but I still don’t know one fundamental question: can I safely patch the iAS Home with a newer version of libc-2.1.3-stubs.so?

At this point, I turned to Oracle support..  It’s been there since… I’ll update this entry as I make progress.

Relevant Notes: 465629.1, 847775.1

Related Patch: 3830807

* edit* 9/2/09: still waiting on word from support…

* edit* 9/23/09: Support confirmed I can overwrite the $IAS_HOME/lib/stubs version (131951 byte) of the file libc-2.1.3-stubs.so with the (newer) version (261328 bytes) of the file from $8.0.6 Home/lib/stubs:

UPDATE
=======
Development confirm that you can safely copy the newer file over the older file.

QUESTION
=========
Can the older (131951 byte) version of libc-2.1.3.sub.so in the iAS home be overwritten by the one delivered with patch:3830807 (261328 bytes)?

ANSWER
========
Yes, based on reasearch this appears to be the intended situation. So please copy the newer file over the older file. This should then be a permanent fix for this problem.

ftp.updates.oracle.com I shall miss thee…

In their infinite wisdom, Oracle decided they’re going to do away with ftp.oracle.com as a site to go fetch patches.  It’s one thing I really like about Oracle support.  It’s a simple, rarely unavailable,  easy way to get my patches, and I don’t have to stage them on my desktop then sftp them to staging server.  It just works.  Not sure why they feel the need to get rid of it; I did my part and opened an SR under feedback and said they shouldn’t kill it.  They replied back that I can use wget with the URL of the patch; this is fairly self defeating as that requires me to go into the (clumsy) Metalink, click around a few different times and type in the patch number, so I can fetch the URL so I can past it into my terminal.

 If you’re a fan of this ftp site, I’d recommend letting Oracle know via the feedback button in Metalink (My Oracle Support).

My name is Kevin and I have an Oracle problem

I think support has become worse every time I open an SR. I’ve decided that going forward, I would be just as well off filling out whatever random information I can in the required fields, and under the Problem heading just typing “My name is Kevin and I have an Oracle problem.” As far as I can tell, the techs never look at what you fill in anyways, and sometimes I can’t even find an applicable category under the product selection. This way, I’ll only have to submit the information once.

Recently we upgraded our front end (forms, web, reports, concurrent managers, and Discoverer 10g) from RH4 to RH5. After this was done, the system admin noticed a few stopped processes on the host, which neither of us could recall seeing before. I opened an SR when I couldn’t find anything useful on Metalink (though I found a few references on Google which just said to kill them…)

Here’s how the SR is going… with my comments in bold… am I being too hard on the tech?

Problem Description:

Since RH5 upgrade, seeing dis51ws processes with status Tl

### Requested files:ACT(Note 183274.1),RDA (Note 175853.1). Files to be loaded. ###
None

### Detailed Problem Statement ###
[oraprod@zso-oraapp-01 env]$ ps -fu oraprod|grep dis51ws
oraprod 10061 15301 0 13:55 ? Tl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146629
oraprod 14183 15301 0 14:08 ? Sl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146630
oraprod 15123 15301 0 14:10 ? Sl 0:01 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146631
oraprod 18897 15301 0 14:20 ? Sl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146632
oraprod 24283 15301 0 14:37 ? Tl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146635
oraprod 26821 15301 0 14:44 ? Sl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146636
oraprod 27443 15301 0 14:45 ? Sl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146637
oraprod 27565 15301 0 14:46 ? Sl 0:01 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146638
oraprod 27814 15301 0 14:47 ? Sl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146639
oraprod 30701 15301 0 14:56 ? Sl 0:00 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146640
oraprod 6956 15301 0 15:21 ? Sl 0:01 /u02/appprod/proddisco/bin/dis51ws -port 0 -opmn_ping_time 20 -preferencePort 16001 -uid 946146641

### Steps to Reproduce Problem ###
run discoverer server 10.1.2.55.26

### Instance Name(s) and Type of System(s) Where Error Occurs ###
PROD production

### Recent Changes to this Environment ###
RH 5

### Workarounds Used ###
none

### How is this Issue Impacting Your Business ###

Can you easily recover from, bypass or work around the problem?
No

Does your system or application continue normally after the problem occurs?
Yes

Are the standard features of the system or application still available; is the loss of service minor?
Yes

And here’s how my stellar analyst replies:

Hi Kevin, 
Please provide the following results.

1. I would like you to confirm the existing version name of Discoverer Server.
 To get the existing version name of Discoverer Server on Unix, go to $ORACLE_HOME/bin and run the command:
 strings -a dis51ws | grep "Discoverer Version:"
 Otherwise you can start Discoverer Plus/Viewer and check the version.  Didn't I provide you this above?  10.1.2.55.26 ring a bell?

2. I would like you to confirm the Discoverer Server "cummulative patch" level.
 To list Discoverer interim patches applyed on your serevr I would like you to execute the script below iAS Discoverer side:
 > cd $ORACLE_HOME/OPatch
 > opatch lsinventory  I will admit, this is a legitimate request, but I could just tell him I'm on CP3

3. Executed the same command on my own RH5 machine  Congratulations... 
 [oracle@orldss-33 ~]$ ps -fu oracle|grep dis51ws
 oracle 3682 3645 0 11:04 pts/1 00:00:00 grep dis51ws
 [oracle@orldss-33 ~]$

What is the status Tl and Sl?  Really?  You don't know?  It's fairly central to my problem.  Why are you working this issue?  Can you Google it?

4. I would like you to execute and provide us the 'uname -a' output  Didn't I tell you I was on RH5?  

Thank you,
 Global Customer Services

* edit * as of today, this issue appears resolved by applying 8354043 which is Discoverer CP5 (Consolidated Patch 5).

11i Patch Merging

Twice a year before regression test cycles, I spend an inordinate amount of time researching the latest round of patches and their pre-requisites. I know there’s got to be a better way, but in my experience I just don’t trust Oracle’s automated methods completely.

As an example, just last week, I was re-applying an interoperability patch for native java (6863618) because I couldn’t recall if I’d applied it in a particular environment. I went into OAM under maintenance and punched in the patch number, it came back with no rows. Fair enough, I applied the patch. Queried OAM again – came back with no rows. Hence my lack of trust…

Regardless, life is simpler with patch merging, known on the command line as admrgpch.

Once I figure out logical groups for patching (for example, Inventory, WIP, etc) I put them into the same staging directory, and merge them. This way, when I have to run this in production, I don’t have to run 12 patches to complete one patch and its 11 pre-requisites. The basic syntax is as follows:

admrgpch -s /scratch/patches11i/test/Fall09_Regression/Functional/INV_RUP4 -d /scratch/patches11i/test/Fall09_Regression/Functional/INV_RUP4/merge -merge_name INV_RUP4_merge_Fall09

The -s is the source directory, where all the patches have been staged and unzipped. The -d directory is the destination, which I just make a subdirectory of the staging directory. The merge_name is just what it will ulitmately call the universal driver (in this case, u_INV_RUP4_merge_Fall09.drv). Regardless, this is a much more efficient way of bundling patches and I strongly recommend it if you’re not using it.

As I wrote about the universal driver, I had a weird recollection of the hell that was patching before universal drivers. Anybody remember the 3 types of patch drivers, for 10 geek points?

They were copy drivers, database drivers, and generate drivers. Copy did the binary copies & relinks, database ran the sql/plsql/java against the RDBMS, and generate rebuilt/compiled the plls, forms and reports. Those were the slow old days….

PC LOAD LETTER

Checker run found 55 new persistent data failures

This message shows up in EM, in a TEST ERP system, with no further information about the issue except when it occurred. It reminded me of PC LOAD LETTER from the movie Office Space. WTF does that mean? Who is checker? Metalink search is not particularly helpful. So I checked the alert log, and found that there were entries back in June when I had logical tablespace corruption, as well as last week, during the time when I was RMAN cloning this environment. That particular clone failed late in the process, because it wanted to create the duplicate (“B”) online redo logs in a directory that didn’t exist ($OH/dbs/blah/blah/blah some stupid default location). Then the clone sat all night and apparently checker came along and said “Hey! You have persistent data failures!”. Problem is, I’ve since redone the clone and run an RMAN backup validate check logical database without incident. So the best I can tell, this is a remnant of a failed RMAN duplicate. This is reinforced by the only relevant Metalink hit which stated this can occur at times during a create database command under some HPUX bug. Oh well, I just cleared it in EM so I didn’t have to look at it, but found it amusing nonetheless.

New from Oracle: Database Patch Set Updates (PSU)

Oracle recently spammed me about Database Patch Set Updates (PSU’s), a new, simplified way to update your Oracle Home to include the latest recommended patches as well as security patches. In their initial rollout, it appears they are only offering this for RDBMS 10.2 on the most “popular” platforms. This list includes:

• HP-UX Itanium
• HP-UX PA-RISC (64-bit)
• IBM AIX on POWER Systems (64-bit)
• Linux x86
• Linux x86-64
• Sun Solaris SPARC (64-bit)

Conspicuously absent: any flavor of Windows! I find this amusing, as I truly can’t stand Oracle RDBMS on Windows.

Well regardless, I think PSU’s are a good idea, long overdue.  Last night I changed hosts of  a small database from and migrated it to 64-bit, so I had to reinstall the RDBMS.  In doing so, I needed to confirm the DST patchset delivered with 10.2.0.4, and this 10.2.0.4.1 (PSU1).   Well 10.2.0.4 was easy, it delivers DST patchset 4, but the PSU took a little digging to confirm that it didn’t contain any DST updates. Since it was on DST 10 on the old host, I put that patch on, then applied 8576156 (10.2.0.4.1). This patch applies similar to a CPU, as opposed to using Oracle Universal Installer. It requires a combination of opatch followed by catbundle.sql psu apply at the end (a CPU uses catbundle.sql cpu apply). Well I’m happy to report that all proceeded uneventfully, and the database came back up fine and by the lack of complaints today, I’d call it a success.

edit: There is a new PSU patch available as described here

If memory serves me correctly

I accidentally left my shared_pool_size too large last database restart, and now I need to give up some shared pool to the buffer cache. It wasn’t like I typed the wrong number but didn’t notice it; it was tuned for using advanced compression (lots of objects in the buffer cache at half their normal size) and I didn’t really account for the additional needed space in the buffer cache when I uncompressed them. This is a dynamic parameter, so in theory I could shrink it anytime… but I don’t fully trust doing that. Last time I tried to shrink the SGA on the fly (sga_target, which is dynamic), the command hung for a few seconds, then my instance crashed. Admittedly, this was under RDBMS 10.2, but it left me unwilling to issue the command during normal business hours. So I’ll try it during my maintenance window this weekend. If it works, great, less work for me to do. If not, well, that’s pretty much what I expected.

Incidentally, we’re in the final throes of eliminating jinitator and moving to native (Sun? Oracle?) Java. Seems like the difficulty here is on the Windows side. We really restrict some of the plant floor machines to the point where they can’t install software, and don’t want to confuse our users with certificates and “trust this time” or “trust always” type dialog boxes. Looks like it will involve an internal certificate push, a silent java client install, and re-signing the jar files with the cert.