Kevin Kempf's Blog

August 26, 2011

When Discoverer stops starting

Filed under: 11i, Discoverer — kkempf @ 1:53 pm

A Bad Day

Corrupt memory on a blade in our blade center crashed about 4 virtual machines and 9 Oracle databases on me a few weeks back.  As luck would have it, one of the machines was running Oracle Enterprise Manager, so I received no alerts.   When I finally got around to starting up Weblogic Server/Discoverer, I received a rather cryptic error and opmn was kind of hung up.   I’d honestly hoped to post the exact failure log, but it seems that I can’t find it anywhere in the logs I know about in WLS and opmn.  So I’ll post the symptoms here, and perhaps you can tuck it away as a warning for some future day.

Starting Discoverer

As you may know if you’ve read my other entries on Discoverer 11g, I created a script which seemed to work great for me, because apparently Oracle thinks people like to run 4 disparate commands to get Discoverer started.

startdisco.sh

#!/bin/bash

export MIDDLEWARE_HOME=/u01/discoverer/Middleware
export DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain
export WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3
export ORACLE_HOME=/u01/discoverer/Middleware/as_1
export ORACLE_INSTANCE=/u01/discoverer/Middleware/asinst_1

rm -rf nohup.out

echo “Ensure NO processes related to disco 11g are running or this will fail”

nohup $DOMAIN_HOME/bin/startWebLogic.sh -Dweblogic.management.username=weblogic -Dweblogic.management.password=pw > /tmp/wls_start.log &

nohup $WL_HOME/server/bin/startNodeManager.sh > /tmp/start_nodemanager.log &

echo “sleeping”
sleep 60

nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh WLS_DISCO t3://myhost.mydomain:7002 > /tmp/start_mgdwls.log &

echo “sleeping”
sleep 60

$ORACLE_INSTANCE/bin/opmnctl startall
$ORACLE_INSTANCE/bin/opmnctl status
echo “If Discoverer doesn’t start properly, login to http://myhost-01.mydomain.com:7002/console”
echo “From the home page, click servers (Under Environment), then the control tab.  Check WLS_DISCO then click the start button below the checkbox”

You can take my script, or leave it, but the bottom line is that the following things need to be started to get Discoverer working:

  1. startWeblogic.sh
  2. startNodeManager.sh
  3. startManagedWeblogic.sh
  4. opmnctl startall

When I ran this script after the server crash, some OS processes would start, some would not, and I remember opmnctl status showing “pending” or something like that, instead of “starting” or “alive”.   Somewhere in some log, something pointed me to an error which said “cannot find /usr/lib/jvm/java-1.6.0-sun-1.6.0.22.x86_64” or the like.

The Nature of the Error

In short: the specific, hard coded JDK on the Linux host which WLS and Discoverer were looking for, was no longer there.  I don’t know why, but I suspect that through the course of normal RedHat updates, after a new version was installed either the sysadmin cleaned up the old version (i.e., deleted it) or it cleans up the old versions as part of the update manager process.  Either way, there’s no way Discoverer was going to start.   The reboot had, in fact, forced the issue; in theory it would have continued to run forever “in memory” despite the fact that the version it was using was no longer available on the disk.

The Quick Fix

As it turned out, I was at the airport when I got a call from our help desk, with users complaining that discoverer wasn’t available.   After finally figuring out what the issue really was (which is more confusing than it seems, since a bunch of processes start just fine without JDK), I did a ghetto-IT fix of creating a symbolic link as root:

/usr/lib/jvm/ln -s java-1.6.0-sun-1.6.0.22.x86_64 java-1.6.0-sun-1.6.0.26.x86_64

Basically, this puts a pointer on the disk saying, “if you’re looking for jdk 1.6.0.22, go look in 1.6.0.26 instead”.  It ain’t pretty, but I had a plane to catch.

Started Discoverer fine immediately afterwards.  Put this on my “to investigate” list.

The Better Fix

When I finally had time, I boiled down my investigation to two points:

  • Where was the JDK version hard coded in the WLS startup scripts
  • What versions of the JDK were certified with WLS/Disco

I got 1 out of 2.  The first one.  Turns out, I’m not the only one to see this problem.  Note 1058804.1 “How to Change Type of JDK (Sun/JRockit) for FMW 11g Domain” explains that the script $MIDDLEWARE_HOME/wlsserver_10.3/common/bin/commEnv.sh has a variable called JAVA_HOME in it which was hard coded to the path to /usr/lib/jvm/java-1.6.0-sun-1.6.0.22.x86_64.  I decided my best option here was to change it to /usr/lib/jvm/java-1.6.0-sun.x86_64 since it appears that the symbolic link in /usr/lib/jvm called java-1.6.0-sun.x86_64 would forever after point to whatever the latest version of the JDK was.  At least this way, I didn’t have to go monkey around in a WLS shell script every time the JDK changed versions.  Honestly, I don’t know if I pointed the WLS installer at a specific version of JDK, if it auto detected it, or installed it itself, but it was bad policy and needed correcting.

Addendum: After more confusion regarding this same issue, I found another file also contains a hard-coded reference to your JVM which apparently is created during installation.  In addition to commENV.sh, you must fix the SUN_JAVA_HOME reference in $MIDDLEWARE_HOME/user_projects/domains/ClassicDomain/bin/setDomainEnv.sh

The part I couldn’t discern was what version of Java was certified for my OS, WLS and Discoverer version.  The analyst was good and send me a bunch of spreadsheets which supposedly covered this, but it became apparent after a few minutes that it would take a higher power to discern what they meant.  I decided that if it’s running now, that’s what’s most important, and that if push ever came to shove, I’d downgrade to fix an issue at the insistence of an analyst.

Advertisements

February 22, 2011

Discoverer 11.1.1.4 certified with 11i

Filed under: 11i, Discoverer — kkempf @ 10:48 am

Of Course

Seems like Discoverer is the only thing I’m posting about these days.  No sooner had I completed the upgrade to 11.1.1.3 when Oracle certified 11.1.1.4 with 11i.

I’ve noticed it only runs on WLS 10.3.4, so that’s a new install too.  It appears that there is some sort of “Smart Update” program to update Weblogic Server to 10.3.4 and the patch set installer will upgrade me from 11.1.1.3 (or 11.1.1.2 if I were using that) to 11.1.1.4.   We’ll see.   At least Oracle has been consistent about incrementing their versioning:

  • Oracle Fusion Middleware 11.1.1.1.0 is certified with WebLogic Server 10.3.1
  • Oracle Fusion Middleware 11.1.1.2.0 is certified with WebLogic Server 10.3.2
  • Oracle Fusion Middleware 11.1.1.3.0 is certified with WebLogic Server 10.3.3
  • Oracle Fusion Middleware 11.1.1.4.0 is certified with WebLogic Server 10.3.4

Upgrade Complete

I’ve successfully upgraded my DEV environment to 11.1.1.4.  It wasn’t as painful as 11.1.1.2 to 11.1.1.3.  Some parts were a pleasant surprise, others were what I would consider to be almost expected failure points now that I’ve run this installer a few times.

Upgrading Weblogic Server

After shutting everything down, you first upgrade Weblogic to 10.3.4.  This is pretty easy.  To find this patch,  go to My Oracle Support, Patches & Updates, Product or Family (Advanced Search).  Type in Oracle Weblogic Server as the Product, and WLS 10.3.4 as the Release.   Hit search and you get appropriate choices (appropriate if you consider PLACEHOLDER BUG FOR WEBLOGIC SERVER 11GR1 (10.3.4) UPGRADE INSTALLER a good name for a patch).  Note how it says upgrade installer.  Well pull this, invoke your environment, run the binary on your Discoverer server, and uncheck “Launch Quickstart”.  Poof!  You’re done.

Upgrading Discoverer

For me, the worst part of this upgrade was finding and downloading the applicable patch.  When I got to My Oracle Support, click on Patches & Updates,  and do a “Product or Family (Advanced Search)” and punch in Oracle Discoverer and version iAS 11.1.1.4 I get 0 results under Linux x86, Linux x86-64, Windows 32-bit.  I know iAS 11.1.1.4 isn’t really what it is (I believe it should read FMW 11.1.1.4 or possibly WLS 11.1.1.4) but given that this is the first version of Discoverer to be called 11.1.1.4, it seems the most logical.  I hate the new My Oracle Support.  It consistently fails, in almost every conceivable way.  In the end, I found the patch numbers (not available on OTN, by the way) in an obscure location in one of the readme’s about how to upgrade FMW, WLS or Discoverer (I forget which).  To save you the time, I’ve listed them here:

  • WLS Upgrade 10.3.4: 11060985
  • Discoverer Upgrade 11.1.1.4: 11060983

In the end, just run the upgrader for Discoverer, and follow the bouncing ball answering the questions.  When I was completely done, two issues.  First, the webcache file permissions were hosed again (https://kkempf.wordpress.com/2011/02/15/discoverer-11g-soup-to-nuts-install-of-11-1-1-3/ Step 5 Number 2) and my passwords were no longer hard coded in the startup and shutdown scripts (https://kkempf.wordpress.com/2011/02/15/discoverer-11g-soup-to-nuts-install-of-11-1-1-3 Step 5 Number 5)

What’s New

Beats me.  The patch says “The list of bugs fixed is not available. Consult the Read Me file.” You won’t find it in the readme.  It will point you to the 11.1.1.4 Portal, Forms, Reports and Discoverer Release Documentation.  I click on release notes there, and select my platform, and it brings me to the Oracle Fusion Middleware Release Notes (11gR1 or 11.1.1) for Linux x86.  I gave up.  I’m going to go with Discoverer 11.1.1.4 is awesome, that’s what’s new.

Edit 4/27/11

It turns out, I had to revert to 11.1.1.3 because some of my reports broke in 11.1.1.4.  This issue revolved around 11.1.1.4’s apparent inability to understand date parameters (in some workbooks) and I wasn’t about to re-write reports for the sake of this lousy upgrade.  All I can say is it was unexpected (I’d never had a problem with a minor skip release like that before) and I recommend you test all your workbooks before upgrading.

February 15, 2011

Discoverer 11g: Soup to Nuts install of 11.1.1.3

Filed under: 11g, Discoverer — kkempf @ 11:43 am

Why Soup to Nuts?

I was looking at my statistics on my most visited pages, and the entries on Discoverer 11g installation and Enterprise Manager 11g installation are by far tops.  I was using my own entries as a guide to re-install Discoverer 11g the other day (it needed to be relocated to a new server) and I realized it was a bit confusing, with important information in several different posts.

What’s New?

Details regarding how to change the default Discoverer 11g client timeout (60 mins), how to install Discoverer on a 64-bit platform, and applying the January 2011 CPU.  Other than that, I’m mostly just smoothing wrinkles in the walk through, and putting it all in one place.  I’m not going to detail 11.1.1.2 as it’s just a subset of 11.1.1.3 and you can find the information elsewhere on the blog.

Environment Variables/Paths I reference

MIDDLEWARE_HOME=Middleware Install Location (xxx/xxx/xxx/Middleware)

DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain

WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3

ORACLE_HOME=Middleware/as_1

ORACLE_INSTANCE=Middleware/asinst_1

Step 1: Weblogic

The first step is to determine, download and install the appropriate version of WebLogic server.  According to my Oracle technology salesperson (as opposed to the 11i Apps salesperson), this is free as we are paying for Discoverer.  I encourage you to check your license terms before you assume you’re entitled to this product, as much has changed since the “old days” of 4i.
You can find the downloads at http://www.oracle.com/technetwork/middleware/weblogic/downloads/wls-main-097127.html  Note, my experience is only installing on Linux, but I’ve done both 32-bit and 64-bit installs so I can go through those.  This is by far the easiest part of the whole install.

Ensure that your WebLogic version lines up with your planned Discoverer install

Oracle Fusion Middleware 11.1.1.3.0 is certified (only) with WebLogic Server 10.3.3 and Apps 11i

If you’re running the 32-bit installer, you just execute the downloaded .bin file; it contains java already and is an executable binary.  If you’re installing the 64-bit version, you’re getting a jar file which isn’t platform specific.  You need to start the installer by using syntax: java -jar wls1033_generic.jar or the like.

There’s really nothing to the install.  I went with defaults, although I’m not sure I needed Oracle Coherence, I figured it wasn’t worth the effort of researching it much further.  There’s only one last item of note here: uncheck and don’t run the quickstart when it’s done laying down the files.

Finally, if you screw up, don’t like the install location, or change your mind, there’s an oui uninstaller buried in there somewhere.  That still leaves a mess on the filesystem (and inventory, in my experience) so the other option is to rm -rf the whole install directory and clean up the oracle inventory ($INVENTORY_HOME/oraInventory/ContentsXML/inventory.xml) to remove references to the install you just did.

That’s it.  Check the box, you’re done with the WLS install for now.  Once all the pieces are installed, there’s a few critical files you may wish to manually edit (regarding starting services) but I’ll go over that later.

Step 2: Discoverer Fusion Installation

Once again, you need to pull the install for the appropriate version of Discoverer.  You can find the downloads at http://www.oracle.com/technetwork/middleware/downloads/fmw-11-download-092893.html towards the bottom of the page.

Since I’m only concerned with 11.1.1.3, pull Portal, Forms, Reports and Discoverer (11.1.1.2) and Portal, Forms, Reports and Discoverer (11.1.1.3).  The 11.1.1.3 installer is only an upgrade from 11.1.1.2, so you have to run both of them.  Note that there’s also the Windows-only Discoverer Desktop and Discoverer Administrator (11.1.1.3) which you will probably need, as well as the Repository Creation Utility (RCU) 11.1.1.3.3.

Start with the Fusion Middleware (FMW) 11.1.1.2 install.  Simply run the installer and follow the bouncing ball, but ensure you select software only install.  There’s no sense trying to configure 11.1.1.2, you’re not going to use it.   When it’s done, run the 11.1.1.3 installer.

I don’t recall which installer (11.1.1.2 or 11.1.1.3) asks for what, but here’s what’s of note during the questions it asks.

  • It will want to know where your Fusion Middleware home is.  Point it to the WLS 10.3.3 install home.
  • Remember the password you provide for the weblogic user, it’s important
  • Stick with the defaults, unless you’re an experienced Weblogic user, regarding Oracle Home name (as_1), Oracle Instance name (asinst_1), and the domain name (ClassicDomain).
  • Under components to install, uncheck everything except Oracle Discoverer and Web Cache. Note there’s a checkbox for Clustered there also.  I unchecked it, have no idea what checking it leads to but it wasn’t applicable to me.

At this point, both installers should be done, but nothing should be configured.

Step 3: Repository Creation Utility (RCU)

The RCU just creates a few tablespaces, new users, and objects in the database.  I don’t know for certain if it uses your existing repository, but it seems that it does.  To launch the RCU, unzip the file, and run the executable called rcu which is in the bin directory.

Of note to the questions it asks:

  • The only thing you need checked on the entire Select Components screen is Discoverer.  (Under Portal and BI-> Discoverer)
  • Remember your new schema user and passwords, they will be required during the configuration of Discoverer and Weblogic.  I recommend SID_DISCOVERER (PROD_DISCOVERER)

Step 4: Configuration

Now you will need to run $MIDDLEWARE_HOME/as_1/bin/config.sh to complete the Discoverer setup.

Most of this should be easy to answer by now, but of note are a few things:

  • Under Specify Schema, use the SID_DISCOVERER (PROD_DISCOVERER) schema you set up with the RCU
  • Unless you’re using OID, uncheck Use Application Level Identity Store.  I missed this once (I don’t use OID) and troubleshooting the startup was impossible.  I had to reinstall the whole stack.
  • I recommend using Auto Port configuration if possible
  • At the end of the config.sh GUI, it displays all ports and URLS.  I recommend you save this file to the OS as it’s a handy reference for how to get to the WLS server and Discoverer login page.
  • Follow note 1073963 which details some profile option values which you need to change in 11i to accommodate Discoverer 11g.

Step 5: Fixes

1.  Fix the webcache bug (doc 1113163.1) “Failed to assign port 8090: Address family not supported by protocol”if you hit it

  • Edit the file $MIDDLEWARE_HOME/asinst_1/config/WebCache/webcache1/webcache.xml
  • Add <IPV6 ENABLED=”NO”/> tag below the </MULTIPORT> tag

2.  I also hit a common webcache problem this time, where the webcache wouldn’t start because webcached permissions hadn’t been set right:

  • Navigate to $MIDDLEWARE_HOME/as_1/webcache/bin
  • As root, run webcache_setuser.sh setroot (username) where username is the user who installed the webcache
  • Permissions should look like this when you’re done: -rwsr-x— 1 root    dba 6796777 Dec  3 11:32 webcached

3.  Copy/link the approriate (8.0.6 home for 11i installs) tnsnames. ora file to 2 locations with a command like this (substitute cp for ln if you want to simply copy the file, obviously)

  • cd $MIDDLEWARE_HOME/asinst_1/config
  • ln -s $8.0.6_HOME/network/admin/SID_host/tnsnames.ora
  • cd $MIDDLEWARE_HOME/as_1/network/admin4
  • ln -s $8.0.6_HOME/network/admin/SID_host/tnsnames.ora

4.  Change the Discoverer default timeout

  • Edit the attribution Timeout in ORACLE_INSTANCE/config/PreferenceServer/Discoverer_instance-name/pref.txt
  • Run ORACLE_INSTANCE/Discoverer/Discoverer_instance-name/util/applypreferences.sh

5. Hard code your password in the startup/shutdown scripts

  • Edit $DOMAIN_HOME/bin/startManagedWebLogic.sh and stopManagedWebLogic.sh
  • add entries for:
    • WLS_USER=”weblogic”
    • export WLS_USER
    • WLS_PW=”password”
    • export WLS_PW

Step 6: Starting/Stopping

These scripts works great for me.  You may have to tweak them a bit, but it’s tested and proven.  Note the sleeps are in there because it takes time to crank some of the services, and unless they’re running the next step fails.  60 seconds is generous.

# STARTUP

export MIDDLEWARE_HOME=/u02/appdevl/devdisco/Middleware

export DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain

export WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3

export ORACLE_HOME=/u02/appdevl/devdisco/Middleware/as_1

xport ORACLE_INSTANCE=/u02/appdevl/devdisco/Middleware/asinst_1

rm -rf nohup.out

rm -rf /tmp/wls_start.log

rm -rf /tmp/start_nodemanager.log

rm -rf /tmp/start_mgdwls.log

echo “Ensure NO processes related to disco 11g are running or this will fail”

nohup $DOMAIN_HOME/bin/startWebLogic.sh -Dweblogic.management.username=weblogic -Dweblogic.management.password=yourpass > /tmp/wls_start.log &

nohup $WL_HOME/server/bin/startNodeManager.sh > /tmp/start_nodemanager.log &

echo “sleeping”

sleep 60

nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh WLS_DISCO t3://(server.domain):7001 > /tmp/start_mgdwls.log &

echo “sleeping”

sleep 60

$ORACLE_INSTANCE/bin/opmnctl startall

$ORACLE_INSTANCE/bin/opmnctl status

echo “If Discoverer doesn’t start properly, login to http://(server.domain):7001/console”

echo “From the home page, click servers (Under Environment), then the control tab.  Check WLS_DISCO then click the start button below the checkbox”

# Shutdown

#!/bin/bash

export MIDDLEWARE_HOME=/u02/appdevl/devdisco/Middleware

export DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain

export WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3

export ORACLE_HOME=/u02/appdevl/devdisco/Middleware/as_1

export ORACLE_INSTANCE=/u02/appdevl/devdisco/Middleware/asinst_1

$ORACLE_INSTANCE/bin/opmnctl stopall

$ORACLE_HOME/opmn/bin/opmnctl status

# stop the discoverer server

$DOMAIN_HOME/bin/stopManagedWebLogic.sh WLS_DISCO t3://(server.domain):7001 weblogic password

# stop the weblogic admin server

$DOMAIN_HOME/bin/stopWebLogic.sh

killall startNodeManager.sh

killall nmz

echo “There’s probably still 1 disco process running – ps -ef|grep jrockit and kill it (java)”echo “It’s probably pid:”ps -ef|grep jrockit|cut -c10-15

Step 7 CPU (optional)

I had no problem applying CPU0111 (patch 10233533), though my ORACLE_HOME required a more recent Opatch (patch 6880880) version than the installer provided.

Step 8 Navigation Reference (Default)

URLS:
Discoverer Plus: http://hostname:8090/discoverer/plusDiscoverer

Viewer: http://hostname:8090/discoverer/viewer

Weblogic Server Administration: http://hostname:7001/console

Weblogic Marketing Propoganda: http://hostname:7001

December 3, 2010

Discoverer 11g with 11i: Upgrading from 11.1.1.2 to 11.1.1.3

Filed under: 11i, Discoverer — kkempf @ 3:15 pm

conFusion strikes again

I had previously posted regarding my struggles with installing, configuring, starting and stopping Discoverer 11.1.1.2 (Discoverer 11g) against an 11i environment.  Well at least Oracle is consistent.  If you can even figure out how to actually install it, the newest version of Discoverer (11.1.1.3) has all the same problems as the prior version: Webcache, startup scripts, etc.  Bundled with a convoluted upgrade only version of 11.1.1.3 and WLS restrictions as to which version can run on which and you get: conFusion Apps!

For no particular reason…

I decided to try the latest and greatest version of Fusion for my Discoverer setup.  Really, I figured that since I hadn’t installed it in production yet, it might buy me a little more support time somewhere down the road if I could sneak in a minor upgrade.  I had to assume that a X.X.X.2 to X.X.X.3 upgrade was minor, right?  Regardless, my situation was this:  I had Discoverer 11.1.1.2 installed and working with Weblogic server 10.3.2, and had previously posted here about the trials and tribulations of getting there.

The initial hurdle

For this upgrade was figuring out how to get there.  The first step is the Weblogic server.  There is no apparent “upgrade” button on the install.  I would have preferred to upgrade in place, from 10.3.2 (which is only certified against 11.1.1.2) to 10.3.3 (which is only certified against 11.1.1.3).  The problem is that the Fusion Middleware (FMW, which contains the Discoverer install) for 11.1.1.3 is not a full install, it’s an upgrade.  One might be inclined to think you would upgrade WLS to 10.3.3 and upgrade 11.1.1.2 to 11.1.1.3 and be done.  I had to open an SR to get the right order.  Effectively, WLS 10.3.3 needs to be in its own home, then you install 11.1.1.2 without the configuration option (software install only).  Then you install 11.1.1.3.  Then you run $MIDDLEWARE_HOME/bin/config.sh to configure discoverer.  Then you pick up the pieces and fix it….

The cleanest solution I could find

In my case, I just deleted or uninstalled everything.  At first, I tried to do a clean uninstall, first by running runinstaller.sh -deinstall (only Oracle would come up with syntax like this) and also uninstall.sh from the WLS 10.3.2 home.  This left a mess in the inventory, so in the end I just did a big rm -rf on the command line and “fixed” the oracle inventory to remove any references to the Fusion homes.

  1. Clean install of WLS 10.3.3 uncheck and don’t run the quickstart when it’s done
  2. Install FMW 11.1.1.2 software only install
  3. Install FMW 11.1.1.3
  4. Run the Repository Creation Utility (RCU)
  5. Run $MIDDLEWARE_HOME/as_1/bin/config.sh to complete the configuration
  6. Fix the webcache bug (doc 1113163.1) “Failed to assign port 8090: Address family not supported by protocol”if you hit it:
    1. Edit the file $MIDDLEWARE_HOME/asinst_1/config/WebCache/webcache1/webcache.xml
    2. Add <IPV6 ENABLED=”NO”/> tag below the </MULTIPORT> tag
  7. I also hit a common webcache problem this time, where the webcache wouldn’t start even after the #6 steup: webcached permissions hadn’t been set right:
    1. Navigate to $MIDDLEWARE_HOME/as_1/webcache/bin
    2. As root, run webcache_setuser.sh setroot (username) where username is the user who installed the webcache
    3. Permissions should look like this when you’re done: -rwsr-x— 1 root    dba 6796777 Dec  3 11:32 webcached
  8. Copy/link the approriate (8.0.6 home for 11i installs) tnsnames. ora file to 2 locations with a command like this
    1. cd $MIDDLEWARE_HOME/asinst_1/config
    2. ln -s $8.0.6_HOME/network/admin/SID_host/tnsnames.ora
    3. cd $MIDDLEWARE_HOME/as_1/network/admin
    4. ln -s $8.0.6_HOME/network/admin/SID_host/tnsnames.ora

For whatever reason, the default port for Discoverer changed from 8090 to 9001:

Discoverer Plus: http://hostname:9001/discoverer/plus

Discoverer Viewer: http://hostname:9001/discoverer/viewer

Weblogic Server Administration: http://hostname:7001/console

Weblogic Marketing Propoganda: http://hostname:7001

For ease of reference, here’s the variables referenced most often in Fusion installs:

MIDDLEWARE_HOME=Middleware Install Location (xxx/xxx/xxx/Middleware)
DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain
WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3
ORACLE_HOME=Middleware/as_1
ORACLE_INSTANCE=Middleware/asinst_1

Just in case you were wondering…

Yes, it’s running fine in Google chrome, though totally unsupported.   Let’s get on the ball and certify the fastest growing browser on the planet, Oracle!

Screenshots

End Game, after quite a fight!

WLS 10.3.3, showing the admin server and discoverer services running

July 28, 2010

Not Feasible to Fix

Filed under: 11i, Discoverer — kkempf @ 1:15 pm

Stopping Discoverer 10g

Anyone familiar with Discoverer 10g on AS 10g (10.1.2.3) has probably seen the following error message when trying to use opmnctl to stop Discoverer:

opmnctl: stopping opmn and all managed processes...
================================================================================
opmn id=hostname-01:6200
    6 of 7 processes stopped.

ias-instance id=BI10g_PROD.hostname-01.domain.com
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
    OC4J/OC4J_BI_Forms/default_island

Error
--> Process (pid=8331)
    time out while waiting for a managed process to stop
    Log:
    /u02/appprod/proddisco/opmn/logs/OC4J~OC4J_BI_Forms~default_island~1

For whatever reason, last maintenance windows I decided I was sick of waiting on this thing, and wanted to get rid of the ugly (but harmless) error message.   I was amused by what I found.

Bug 7357325

From My Oracle Support:

Solution
To implement the solution, please execute the following steps:

Discoverer development back-ported a fix to the second Discoverer cumulative patch on top of Oracle Application Server patchset 3 (10.1.2.3); however since then, per Bug 7357325, this workaround fix has been removed as it causes other regressions. Development has marked this as not feasible to fix in 10g. The message is harmless and can be avoided once all the Discoverer sessions exit cleanly (see above note).  We do recommend that you apply the latest cumulative patch, per Note 237607 ‘ALERT: Required and Recommended Patch Levels For All Discoverer Versions’

Goodness.  I hope they don’t cop that attitude with too many products.

July 16, 2010

Starting and stopping Discoverer 11g

Filed under: 11i, Discoverer, Support — kkempf @ 9:20 am

I’ve learned a few things about Discoverer 11g in the past few weeks.

  • Apparently, the model for Discoverer 11g on Linux is: start it once from the installer and pray it never shuts down and that you never need to restart the server.
  • Weblogic people at support don’t have any idea what Discoverer is, but they’re happy to help you troubleshoot the weblogic pieces.  In other words, nobody seems willing to embrace the whole product.  conFusion has arrived!
  • The most difficult part of a Discoverer 11g upgrade/install is figuring out how to start and stop the server

Support Woes

Document 1099996.1 explains the gist of starting/stopping Discoverer 11g on linux (via command prompt).  Not only did the Weblogic analyst I had not know about Discoverer 11g, he was also unfamiliar with Linux, and was apparently helping me troubleshoot my problems via a Windows install of Weblogic.  Seems like not such a great fit.  He was further confounded by the fact that a web conference was unavailable, as my system (OS & browser) failed the checks on their site.

Guess I’ll figure it out myself

At some point in most SRs, it becomes clear that if I really want the solution, it won’t be coming from my analyst.  If I manage to figure it out, I’ll keep the SR open to see if my analyst ever does.  Usually I get annoyed with their irrelevant questions and close the SR as “Determined solution myself”.  Somehow, those SRs are always “Hard Closed” by the analyst as “Solution Provided”.   This is one of those cases.

The gist of the problem starting and stopping all the pieces of Discoverer 11g on Linux is that there’s no good documentation.  Through trial and error, I got this thing working.  There were major problems:

  • The scripts laid out in doc 1099996.1 are not like normal Oracle scripts which automatically run in the background.  They have to be run nohup (no hangup) and with & (background).  This isn’t a problem per se, except some of them are interactive.  Meaning some of these scripts ask for things like username and password.  This doesn’t work nohup’d and backgrounded.
  • There’s too many scripts.  4 scripts to start discoverer; in 10g there were 2 (opmnctl startall and emctl start iasconsole).  It’s a sorry step in the wrong direction.
  • For whatever reason, there’s a t3 in front of the URL argument passed to many of the scripts.  I have no idea why, but it’s required.
  • /yourpath/Middleware/user_projects/domains/ClassicDomain/bin/startManagedWebLogic.sh requires you to hard code values in their script.  Yes, you need to vi the file and put your username and password in the script under WLS_USER and WLS_PW or it will not start except in the foreground on your screen (interactively).  If you do this, you cannot ever close that screen, or Discoverer stops.
    • WLS_USER=”weblogic”
    • WLS_PW=”yourpassword
    • To add insult to injury, the script exports these values, meaning they’re available variables to the host.

Startdisco.sh

#!/bin/bash

export MIDDLEWARE_HOME=/u02/appdevl/devdisco/Middleware

export DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain

export WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3

export ORACLE_HOME=/u02/appdevl/devdisco/Middleware/as_1

export ORACLE_INSTANCE=/u02/appdevl/devdisco/Middleware/asinst_1

rm -rf nohup.out

rm -rf /tmp/wls_start.log

rm -rf /tmp/start_nodemanager.log

rm -rf /tmp/start_mgdwls.log

echo “Ensure NO processes related to disco 11g are running or this will fail”

nohup $DOMAIN_HOME/bin/startWebLogic.sh -Dweblogic.management.username=weblogic -Dweblogic.management.password=disco11g > /tmp/wls_start.log &

nohup $WL_HOME/server/bin/startNodeManager.sh > /tmp/start_nodemanager.log &

echo “sleeping”

sleep 60

nohup $DOMAIN_HOME/bin/startManagedWebLogic.sh WLS_DISCO t3://(server.domain):7001 > /tmp/start_mgdwls.log &

echo “sleeping”

sleep 60

$ORACLE_INSTANCE/bin/opmnctl startall

$ORACLE_INSTANCE/bin/opmnctl status

echo “If Discoverer doesn’t start properly, login to http://(server.domain):7001/console”

echo “From the home page, click servers (Under Environment), then the control tab.  Check WLS_DISCO then click the start button below the checkbox”

Stopdisco.sh

#!/bin/bash
export MIDDLEWARE_HOME=/u02/appdevl/devdisco/Middleware
export DOMAIN_HOME=$MIDDLEWARE_HOME/user_projects/domains/ClassicDomain
export WL_HOME=$MIDDLEWARE_HOME/wlserver_10.3
export ORACLE_HOME=/u02/appdevl/devdisco/Middleware/as_1
export ORACLE_INSTANCE=/u02/appdevl/devdisco/Middleware/asinst_1
$ORACLE_INSTANCE/bin/opmnctl stopall
$ORACLE_HOME/opmn/bin/opmnctl status
# stop the discoverer server
$DOMAIN_HOME/bin/stopManagedWebLogic.sh WLS_DISCO t3://(server.domain):7001 weblogic disco11g
# stop the weblogic admin server
$DOMAIN_HOME/bin/stopWebLogic.sh
killall startNodeManager.sh
killall nmz
echo “There’s probably still 1 disco process running – ps -ef|grep jrockit and kill it (java)”
echo “It’s probably pid:”
ps -ef|grep jrockit|cut -c10-15

June 29, 2010

Discoverer 11g Installation on 11i

Filed under: 11i, Discoverer — kkempf @ 3:33 pm

Another Upgrade for the sake of Upgrading

Looking at my list of things to do, I recalled that Discoverer 10g (10.1.2.3) was on the bubble for upgrade to Discoverer 11g.  If you read here regularly, you’ll recall  that originally, Oracle planned to kill support for 10.1.2.3 in December of 2010, but something changed and they gave AS 10.1.2.x a 1 year reprieve (page 8).  I have no particular inclination to perform this upgrade, except that it’s just another “thing” to do sooner or later, and I had the time to play with it, in earnest, this week.

Product Certification

Like other Fusion entities (EM 11g and possibly R12, sooner than you’d think), Discoverer 11g ditched Oracle application server and moved to Weblogic Server.  Discoverer 11g comes in 3 flavors at the moment: 11.1.1.1, 11.1.1.2 and 11.1.1.3.  Per Oracle support, the certified version combinations are:

  • Oracle Fusion Middleware 11.1.1.1.0 is certified with WebLogic Server 10.3.1
  • Oracle Fusion Middleware 11.1.1.2.0 is certified with WebLogic Server 10.3.2
  • Oracle Fusion Middleware 11.1.1.3.0 is certified with WebLogic Server 10.3.3

When you add E-Business Suite 11i (or R12), you’re immediately stuck at 11.1.1.2 (as of this moment).  Late breaking news, 11.1.1.3.0 is now certified

Installation Overview

One of my major gripes with (con)Fusion 11g, is that the installs are so convoluted and peculiar.  In the case of Discoverer 11g, you first have to install Weblogic (10.3.2).  This actually went off without a hitch, and simply provides the framework from within which Discoverer can run.

Next you have to run the Repository Creation Utility (rcu) against your E-Business suite RDBMS.  This is like a mini-install, which effectively just forms a database connection and creates some new tablespaces and upgrades your repository to make it “11g ready”.  What, exactly, it upgrades, is a mystery to me.  Finally, you have to install Discoverer, but the installer includes everything but the kitchen sink.  I kid you not, here’s what’s in the download:

  • HTTP Server
  • WebCache
  • Portal
  • Forms Services
  • Forms Builder
  • Reports Services
  • Report Builder/Compiler
  • Discoverer Administrator
  • Discoverer Plus
  • Discoverer Viewer
  • Discoverer Services
  • Discoverer Desktop
  • Enterprise Manager Fusion Middleware Control

After running the Discoverer install, I’m required to apply patch 9387227 for 11i interoperability.  Of course, this won’t  work until I’ve applied patch 6880880 – you got it, the latest OPatch to the Oracle Home.  You can find Discoverer, the Repository Creation Utility and Weblogic here.   Interestingly, the RCU and  Weblogic are at the end under “Required Additional Software”.   To get 9387227 or 6880880 you can just search from Metalink.

So in the end, to install Discoverer, it’s Weblogic Server 10.3.2 (1 CD),  RCU (1 CD), Discoverer (4 CDs), and 2 one-off patches.  I wish this was bundled in some understandable way, but Fusion appears to be a patchwork of installs in competition with the labyrinth of Minos for clarity.

Weblogic 10.3.2 Installation

Nothing difficult here.  Just choose your paths and follow the bouncing ball, so to speak.  Just remember not to check the Quick Launch at the end.  The screen shots below depict 10.3.3; this is because I forgot to take screen shots during the 10.3.2 install.  They’re virtually identical from a GUI standpoint, I almost don’t even need to include any of them.

Launching the Weblogic Installer from the command line

Welcome Screen

Tell the installer where the Middleware home will reside

Don't call me; I'll call you

Just choose "Typical"

Give it a path. Coherence is new.

Here's the gigabyte of "stuff" I'm going to install

Happily working at installing

Repository Creation Utility

Just figuring out how to launch this thing was annoying.  When unzipped, it creates a folder called rcuHome, which has about 25 directories in it.  From the bin directory, run rcu.  The options and required information is pretty self explanatory until you get to step 3 called “Select Components”.  Under Portal and BI, you select Discoverer (only) and proceed.  Shortly after this time it connects to your ERP database, creates a few tablespaces owned by a schema owner you designated, and ultimately unceremoniously dumps you back to the command prompt.

Launching the Repository Creation Utility

Welcome Screen

Create (as opposed to drop) the components

Provide database connectivity information

Warnings, verifications, etc

You only need to select Discoverer

New Schema Passwords

New Tablespaces will be created

Final look before it runs

Discoverer

First of all, could the versioning be more confusing?  While someone at Oracle may think 11.1.1.2.0 is a great name for a version, I personally do not.  How about Discoverer 11gR1.2?  Or, perish the thought, just Discoverer 11gR2?  Are there really going to be that many versions of this thing?  Alright, once again, this install was largely “follow the bouncing ball”.   I purposely chose the defaults, and in the process created an near indiscernible mess on the operating system between ORACLE HOME being called as_1 and the ORACLE_INSTANCE being called asinst_1.  I’d urge you to try using something more meaningful.  I did make the weblogic domain name Discoverer, as I suppose, in theory, it could host multiple applications.  Under components, once again I chose to deselect everything I could (including Enterprise Manager for Fusion) and choose only Oracle Discoverer and the Oracle Web Cache.   I used auto port configuration and unchecked the “Use Application Level Identity Store” and we don’t use OID for authentication.  After running root.sh at the end, it said it was done successfully.

Launching the Discoverer Installer

Choose Install & Configure

System Prerequisite Checks

Provide the weblogic domain information you wish

Sign up for Oracle spam

Provide details on the installation location(s) and names

Choose Discoverer and web cache, uncheck clustered if applicable. Enterprise Manager at your discretion

Choose Discoverer and web cache, uncheck clustered as applicable. Enterprise Manager at your discretion.

Let Oracle figure out the ports..

Create a connect string: myserver.mycompany.com:1521:DEV or the like

OID integration, which I don't have to worry about

Installation Summary

Installation in Progress

It's not an install without root.sh

Post install configurations

Successful install

Helpful summary of locations and ports

More of helpful summary with URLs

OPatch & Interop Bug

These installed without incident, though the hardest part was figuring out how to set environmental variables for the new Discoverer 11g environment.  For what it’s worth,

the MIDDLEWARE_HOME is the base Middleware directory
the ORACLE_HOME is a subdirectory Middleware/as_1
the ORACLE_INSTANCE is subdirectory Middleware/asinst_1

Webcache Bug

I always hit a problem with the webcache.  The weblogic server was happily up and running, but when I went to the URL for discoverer, I got nothing.  There was some vague message to the effect of:

Failed to assign port 8090: Address family not supported by protocol

After a surprisingly responsive and competent analyst helped me for the better part of the day, I realized I was hitting a bug spelled out in ID 1113163 (which was supposedly only relevant to Suse Linux).  Effectively, there was a problem out of the box with IPv6 (which I didn’t have enabled on the host).  After adding <IPV6 ENABLED=”NO”/> tag below the </MULTIPORT> tag in the webcache.xml file, all was good and I was finally able to log in.

TNS conFusion

Since I started working with Oracle, the tnsnames. ora file has gone in $ORACLE_HOME/network/admin.  So Discoverer requires it in the instance home:  Middleware/asinst_1/config.  Makes sense to me.  Until you put in a valid entry for your ERP database, you cannot connect.  I put the file under the Middleware/as_1/network/admin (aka ORACLE HOME) and made a symbolic link to it in the Middleware/asinst_1/config directory.

Default URLs

Discoverer Plus: http://hostname:8090/discoverer/plus

Discoverer Viewer: http://hostname:8090/discoverer/viewer

Weblogic Server Administration: http://hostname:7001/console

Weblogic Marketing Propoganda: http://hostname:7001

Initial Impressions

Aside from the capacity to remember some of the login information and save it so I don’t have to type everything about my connection every time I go there, there appears to be no fundamental difference between Discoverer 10g and Discoverer 11g.  This truly was an upgrade merely for the sake of upgrading, my favorite kind. </sarcasm>

 

Creating connections is an improvement over 10g

Saved Session

 

Profile options

I should note, there’s a bunch of profile option values and autoconfig context file values which you may wish to change.  Realistically, I’m not sure how much they affect, but you should follow note 1073963.

April 14, 2010

Discoverer EUL exports fails on libpthread.so.0

Filed under: Discoverer, Linux — kkempf @ 8:49 am

The export fails

I noticed a cron job which ran a nightly export of the Discoverer End User Layer schema was failing recently, and upon investigation the exact error looked like this:

java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory

To be precise, the exact point of failure is in the call to $DISCOVERER_HOME/bin/eulapi

What the script is, and why I run it

I don’t know how relevant any of this is to anyone anymore, as Discoverer use is declining, and the export is pretty much just insurance for me.  The main reason I ever started the job was during development/migration of all of our reports from 4i to 10g; it was a nice way to be able to quickly revert to “yesterday” without affecting the rest of 11i.   I run the job as a cron of a parameterized script on linux; which looks like this in the crontab:

# run a nightly export of the Discoverer EUL
0 21 * * * /scratch/oracle/dba/discoverer/exportEUL.sh -sid PROD -eulapi /u02/appprod/proddisco/bin > /dev/null

exportEUL.sh

#!/bin/ksh
##############################
# Oracle Discoverer EUL Export
##############################
# Run this script as a cron only from the discoverer node; it will fail elsewhere
# default EUL
EUL=eul_us
if [ $# -eq 0 ]
then
echo "Usage:  exportEUL.sh"
echo "    -sid      (DEV)                            Instance"
echo "    -eul      (eul_us)                         EUL name"
echo "    -eulapi   (/u02/appmis/misora/iAS10g/bin)  Path to eulapi binary"
echo
exit 1
fi
for parm in `echo $*`
do
if [ $1 = '-sid' ]
then
shift
SID=$1
elif [ $1 = '-eul' ]
then
shift
EUL=$1
elif [ $1 = '-eulapi' ]
then
shift
EULAPI=$1
fi
if [ $2 ]
then
shift
fi
done
# lower case SID
LOWERSID=`echo $SID|tr [A-Z] [a-z]`
PASSW=`cat /scratch/home/ora${LOWERSID}/.discoverer`
CONNECT=sysadmin/$PASSW\@${SID}
TODAY=`date +%d%m%y`
EXPORTFILE=/scratch/oracle/dba/discoverer/exp${SID}_${TODAY}.eex
LOGFILE=/scratch/oracle/dba/discoverer/exp${SID}_${TODAY}.log
${EULAPI}/eulapi -connect ${CONNECT} -apps_user -apps_responsibility "System Administrator" -eul ${EUL} -export ${EXPORTFILE} -all -log ${LOGFILE}
zip ${EXPORTFILE}.zip ${EXPORTFILE}
rm ${EXPORTFILE}
zip ${LOGFILE}.zip ${LOGFILE}
rm ${LOGFILE}
find /scratch/oracle/dba/discoverer/*PROD*.zip -mtime +31 -exec rm -rf {} \;
find /scratch/oracle/dba/discoverer/*DEV*.zip -mtime +7 -exec rm -rf {} \;

The Fix

I found this on MOS, in ID 729952.1.  Well most of this.  #4, below, was wrong in the document online as it implied applypreferences.sh was in $ORACLE_HOME/discoverer, but I added the full (and correct) path for clarity.  It’s about the cheesiest fix I’ve seen in awhile, but it works.  Anytime I’m asked to “pound out” a line of a delivered shell script, it feels wrong, but in the end, it’s no different than all the fixes you used to have to make after an autoconfig run to various 11i files which got messed up before the templates finally got in order.

  1. Navigate to:  $ORACLE_HOME/discoverer.
  2. Backup the discwb.sh environment script file.
  3. Remove “export LD_ASSUME_KERNEL=2.4.19” from the discwb.sh file.
  4. You can now execute:  $ORACLE_HOME/discoverer/util/applypreferences.sh

The Cause

Unknown, but my suspicion is that it was either an OS upgrade or a Discoverer CP patch.

April 9, 2010

A dose of good news from Oracle?

Filed under: Discoverer — kkempf @ 3:00 pm

A few months ago, I was grumbling about the end of life for Discoverer 10g, which was occurring without a replacement certification in sight for 11i.  Since then, Discoverer 11g has been certified, and it appears that iAS/Application Server/(con)Fusion/Discoverer 10g has received a 1 year reprieve from the powers that be at Oracle via a free extension of premier support until December 2011. The .pdf states:

“Includes: BPEL, InterConnect, B2B, Business Activity Monitoring, Discoverer, Reports, Forms, Portal, Internet Directory, Single Sign-On, Certificate Authority, Containers for J2EE, Web Cache and Developer Suite (Designer, Forms Builder, Reports Builder & Discoverer Administrator)”.

It’s less than clear, and it looks like Steven Chan minces no words when he says he’s not allowed to answer questions or discuss the issue much.  His recommendation is to log an SR to ask specific questions; my personal recommendation, in the case of Discoverer, is just to get off of Discoverer 10g and go to 11g.

In related news, I’ve seen a few demos of the OBIEE plus product.  Wow, no wonder Discoverer has been left to die on the proverbial vine.  This is a polished, feature-rich product which is eye candy for non-IT folks.  It ain’t cheap.

Heading to OAUG next week, will be my first time there.  Might make for some interesting blog fodder.

December 3, 2009

Discoverer 10g with Ebuisness Suite 11i (CP6 available)

Filed under: 11i, Discoverer, Linux — kkempf @ 10:44 am

I’m still running Discoverer 10g…we’re not sophisticated enough or rich enough to justify OBIEE yet.  I noticed in Note 237607.1 that Discoverer 10g CP6 (8746296) was available for Windows, Solaris and HP-UX a few weeks back; now they’ve got the Linux x86 version available.   It’s as simple as OPatch gets: download it, unzip it, stop discoverer, run opatch, start discoverer.

Version before and after the patch was: 10.1.2.55.26; apparently this doesn’t change with CP’s

I looked up the Statement of Direction for Oracle Discoverer 10g, as it’s always been a bit of a red-headed stepchild in the Oracle inventory (more so with the advent of OBIEE).  Looks like 10gR2 (10.1.2) loses premier support in 1 year, and extended support in 2 years.  Not sure if that means it will be kept on life support on an exception basis for Ebusiness Suite users, or if we’ll see a cert for Discoverer 11g with 11i.

Incidentally, about a year ago we moved from desktop-based Discoverer 4i to 100% web based use of Discoverer 10g.  From an administration standpoint, it’s a dream.  We intentionally decided not to let users have the 10g Discoverer plus desktop client due to the hassles of patches, tnsnames changes, etc, and the result is that I only think about Discoverer a few times a year when various patchsets come out.

Discoverer 10g with 11i

Create a free website or blog at WordPress.com.