Kevin Kempf's Blog

October 23, 2014

Running a shell script via concurrent request

Filed under: 11i, Linux, Mobile Supply Chain, R12.2 — kkempf @ 8:55 am

shell

The Problem

Mobile Supply Chain (MSCA) in 11i and R12 is a rather unreliable product as I discussed here.  It tends to cease working for no reason; I’ve blogged about this before so I won’t dwell on how ridiculous that is.  While I have Linux/OS cron jobs to bounce the service entirely once per day, and rotate the ports twice per day, it’s still not enough.  Invariably, the dispatcher will stop fielding connections at 2am on Sunday (2 hours after the bounce), when I get a phone call and I have to go VPN in and unscrew it.  It is in my own self-interests, then, that I set out to create a concurrent request which would cleanly kill/restart MSCA from inside the Ebusiness Suite, and the reason behind this post.  Incidentally, one added benefit of using a concurrent process is that I can leverage Oracle responsibilities and security, ensuring not just anybody can bounce the MSCA telnet server.

Nothing New Here

I suppose I’ve always known there was a host option when you defined a concurrent program, I just never knew if it worked.  I’m in a weird sort of down-time before our R12.2 go-live and find myself without much to work on for a few days, so I’m “playing” in R12.2.  There’s nothing revolutionary about this post, it’s just my goal to walk from beginning to end through the process of running a host script as a concurrent process, and provide some in depth/meaningful context as to why you’d like to do that.  In fact I’d like to acknowledge the Oracle Ebizsuite blog here which helped me understand how to go about setting this up, as well as Doc ID 1594853.1.

From the Apps to the OS

I’ll walk the setup starting within the apps.  This bit hasn’t changed any from 11i to R12.

Concurrent Program Executable

As sysadmin, define the Executable via Concurrent->Program->Executable

CP Executable

Incidentally, you need to define an application in the 3rd line; we used our custom schema.

Concurrent Program Definition

Now define the concurrent program via Concurrent->Program->Define

Incidentally, you need to define an application in the 3rd line; we used our custom schema.  The Executable in the 5th field is the one you defined in the prior step.  Note I have no parameters, but if you wanted to you could add them here.  We’ll go over how the shell sees these in a minute.

CP Define

Assign a Request Group

Go to Security->Responsibility->Request and figure out what group can run this concurrent request.  Or create a group, I suppose.

Sec Respons Request

Write Your Script

Things to know about the script

  • appsuser is the OS user who started MSCA out of $ADMIN_SCRIPTS_HOME via mwactrlwrpr.sh or the like
  • kkempf@myemail.com references: I want to know when someone runs this request, or when the dispatcher port isn’t free for some reason to restart the dispatcher (in essence, meaning the script failed)
  • System vs. User parameters: I didn’t see any way to make it cleaner than was done over here, so I pulled it in so you can see the parms
    • $0: The shell script being run
    • $1: apps/{your apps password in plain text!!!}  HINT: Don’t put this in the concurrent request, or the person who just ran this CR can see the apps password
    • $2: unsure; possibly OS PID?
    • $3: Apps userid who ran the job
    • $4: Concurrent Request ID
    • $5: First User parameter (in my case, there are none)
    • $6-$n: Additional User parameters
  • 11i vs. R12 parms: I had two versions because they moved where the port number of the individual telnet servers were between 11i and 12 so awk changed slightly to get the right spot
    •     #DTL=`ps -fu appsuser|grep telnet|grep -v "grep"|awk '{print $15}'`   # 11i
          DTL=`ps -fu appsuser|grep telnet|grep -v "grep"|awk '{print $17}'`   #  R12.2
  • Seems like it need the .prog extension to be called e.g. scriptname.prog
  • If you try to use this in 11i you will need to define $ADMIN_SCRIPTS_HOME variable via .bash_profile or something as it’s a new one in R12.  Or you can just hardcode the path to the mwactlwrpr.sh script
  • I put the script in $CUSTOM_TOP/bin
  • Script walk through
    • It first echoes back your parameters
    • Next it issues the command mwactlwrpr.sh stop to attempt to gracefully shut down the MSCA telnet server.  This will fail to stop the dispatcher.
    • It emails me that the concurrent request was run, so I can be aware of how often this is needed, and who is doing it
    • Sleep 30 seconds so the mwactlwrpr.sh command can try to do it’s job.  Then figure out the PID for MWADIS and kill it
    • Next figure out the remaining telnet processes and kill them.  This doesn’t normally execute
    • Sleep 30 seconds and then issue mwactlwrpr.sh start.  Tee the output of this command to a temp file and scan for the words “is not free to start the dispatcher”
      • tee /tmp/mwa_alert.txt
    • If you see that it’s not free to start the dispatcher, email me again, this means there’s a bigger problem which needs my attention

bounce_mwa.prog

#!/bin/bash
# mwa_bounce.sh   06/24/13 kmk

echo "Following are System Parameters"
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
p0=$0
p1=$1
p2=$2
p3=$3
p4=$4
echo "1st System Parameter    :"$p0
#echo "2nd System Parameter    :"$p1
echo "3rd System Parameter    :"$p2
echo "4th System Parameter    :"$p3
echo "5th System Parameter    :"$p4
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
echo "Following are User Parameters  "
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"
#u1=$5
#echo "1st User Parameter    :"$u1
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~"


echo "Stopping MWA Server via "$p0
echo
nohup $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh stop $p1
echo $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh stop APPS/password

mail -s "Concurrent Request "$p4" Submitted to Bounce MWA by "$p3 kkempf@myemail.com < /dev/null

sleep 30

# Kill MWADIS

for pids in `ps -fu appsuser|grep MWADIS|grep -v "grep"|awk '{print $2}'`
do
    echo "forcing kill on MWADIS PID ${pids}"
    kill -9 ${pids}
done

# Kill telnet
for pids in `ps -fu appsuser|grep telnet|grep -v "grep"|awk '{print $2}'`
do
    #DTL=`ps -fu appsuser|grep telnet|grep -v "grep"|awk '{print $15}'`   # 11i
    DTL=`ps -fu appsuser|grep telnet|grep -v "grep"|awk '{print $17}'`   #  R12.2
    echo "forcing kill on PID ${pids} for ${DTL}"
    kill -9 ${pids}
done

sleep 30

echo "Starting MWA Server via "$p0
echo $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh start APPS/password
echo
rm -rf /tmp/mwa_alert.txt
touch /tmp/mwa_alert.txt
nohup $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh start $p1 |tee /tmp/mwa_alert.txt
  
# if the dispatcher isn't free, email me

MWAERROR=`cat /tmp/mwa_alert.txt | grep "is not free to start the dispatcher"`

if [ ! -z "$MWAERROR" ]
then
  echo $MWAERROR | mail -s "MWA Error" kkempf@myemail.com
fi

Why Didn’t I Just Use Oracle’s Vanilla Scripts?

In short, because they don’t work.  They start MSCA fine, but in 12.2 (at least as of the AD delta 5) they don’t actually stop the processes correctly.  It stops the individual ports, but not the dispatcher.  The dispatcher is like some bad villain in an action movie that just refuses to die:

$ ps -ef|grep oratest
...                                                                                                                                                                                                                                    
oratest  29516     1  0 Oct21 ?        00:01:26 /u02/apptest/fs2/EBSapps/comn/util/jdk32/bin/java -Dxdo.xliff.source=EBS -DCLIENT_PROCESSID=29516 -Doracle.apps.mwa=/u02/apptest/fs2/EBSapps/appl/mwa/12.0.0 -Doracle.apps.jrad.mmd=/u02/apptest/fs2/FMW_Home/Oracle_EBS-app1/applications/oacore/html/jrad -Doracle.apps.inst=/u02/apptest/fs2/inst/apps/TEST_host -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10260
oratest  29540     1  0 Oct21 ?        00:01:24 /u02/apptest/fs2/EBSapps/comn/util/jdk32/bin/java -Dxdo.xliff.source=EBS -DCLIENT_PROCESSID=29540 -Doracle.apps.mwa=/u02/apptest/fs2/EBSapps/appl/mwa/12.0.0 -Doracle.apps.jrad.mmd=/u02/apptest/fs2/FMW_Home/Oracle_EBS-app1/applications/oacore/html/jrad -Doracle.apps.inst=/u02/apptest/fs2/inst/apps/TEST_host -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10262
oratest  29567     1  0 Oct21 ?        00:01:25 /u02/apptest/fs2/EBSapps/comn/util/jdk32/bin/java -Dxdo.xliff.source=EBS -DCLIENT_PROCESSID=29567 -Doracle.apps.mwa=/u02/apptest/fs2/EBSapps/appl/mwa/12.0.0 -Doracle.apps.jrad.mmd=/u02/apptest/fs2/FMW_Home/Oracle_EBS-app1/applications/oacore/html/jrad -Doracle.apps.inst=/u02/apptest/fs2/inst/apps/TEST_host -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10264
oratest  29584     1  0 Oct21 ?        00:00:00 /bin/sh -f /u02/apptest/fs2/inst/apps/TEST_host/admin/scripts/mwactl.sh start_dispatcher
oratest  29589 29584  0 Oct21 ?        00:00:01 /u02/apptest/fs2/EBSapps/appl/mwa/12.0.0/bin/MWADIS
$ mwactlwrpr.sh stop apps/apps

Stopping MWA Servers and the dispatcher ....
Stopping MWA Server on Port number: 10260
Stopping MWA Server on Port number: 10262
Stopping MWA Server on Port number: 10264
Stopping MWA Dispatcher on Port number: 10830
$ ps -ef|grep oratest
...                                                                                                                                                                                
oratest  29584     1  0 Oct21 ?        00:00:00 /bin/sh -f /u02/apptest/fs2/inst/apps/TEST_host/admin/scripts/mwactl.sh start_dispatcher
oratest  29589 29584  0 Oct21 ?        00:00:01 /u02/apptest/fs2/EBSapps/appl/mwa/12.0.0/bin/MWADIS

$  mwactl.sh stop_dispatcher apps/apps
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002]
Telnet dispatcher shut down successfully.
$ ps -ef|grep oratest
...                                                                                                                                                                                        
oratest  29584     1  0 Oct21 ?        00:00:00 /bin/sh -f /u02/apptest/fs2/inst/apps/TEST_host/admin/scripts/mwactl.sh start_dispatcher
oratest  29589 29584  0 Oct21 ?        00:00:01 /u02/apptest/fs2/EBSapps/appl/mwa/12.0.0/bin/MWADIS

Final Steps

Go to $CUSTOM_TOP/bin or wherever you have your shell script

Sym link fndcpesr to the name of the shell script minus the .prog extension:

ln

Confession: I have no idea how you’re supposed to do this in R12 and what I could find on My Oracle Support only covered through R12.1 (where there is only one filesystem).  I don’t know how you’re supposed to handle the dual filesystems, and you can clearly see my sym link is hard coded to fs2.  There has to be a more elegant way to do this, just not sure what it is at the moment.

If I do an ls -ltr on my directory, here’s what it looks like:

ls ltr

Concurrent Request

Here’s the log (there is no output) from my concurrent request.

CR Log

Soapbox

I found this to be a fun diversion, but I like writing shell scripts as much as SQL, and I like figuring out ways to do things which are more “correct”.  All that said, shame on the Oracle MSCA team for delivering such a half-baked product.  The R12.2 version of MSCA is plagued by a host of problems, most of which I have active SR’s open for:

  • Constant issues with tabbing not working in the MSCA GUI
  • Really, your stop script doesn’t stop your process?  Even when I specify stop_dispatcher?
  • How is it that adstrtal.sh doesn’t start MSCA, but it used to in 11i, and R12 has been out for some time?  I could say the same for adstpall.sh…
  • Major, quirky Java compatibility issues with the MSCA GUI
  • For the love of Pete, please don’t tell me that the tabbing issues are known to be resolved in JRE 1.7.23.  While your MSCA might work with 7.23, that’s not a real-world option as everyone else at Oracle is telling me to be on the latest version of Java and this PC also has to connect to Oracle forms.

October 17, 2014

Querying your 11i/R12 Context File from Oracle SQL*Plus

Filed under: 11i, R12.2 — kkempf @ 8:53 am

 

contextfile

Peeling back the layers of the context file, precipitated by MSCA

We all know that the context file holds information about virtually everything on the applications tier, and how important it is.  As a result of our upgrade effort from 11i to R12.2, I started getting incessant calls wanting to know that current dispatcher port for MSCA. While this post will be specific to MSCA as the reason for having to figure out how to do this query, the principles here can be used to obtain any value from the context file via SQL*Plus, for any reason.

Background

I should note a few things about the MSCA product for those (presumably the majority of you) who are unfamiliar with it.  The MSCA product is essentially a telnet front end, which runs on the application server on a specific port.  Users can either use telnet to connect to it, or the MSCA GUI client.  Either one requires a hostname and port.  MSCA actually runs on many ports, managed by a dispatcher to reduce the load on any one port.  So ideally, users connect to the dispatcher.  Thus one would think that when I created a new environment, it would be easy to just inform everyone of the port to connect to and the calls would end.  In my case, there’s two problems with that.  First, there’s a known R12.2 bug which switches the dispatcher port during an online patch session.  So if you’re running on filesystem1, it might be 10300, but if you’re on filesystem2, 10302.  Second, MSCA doesn’t like to stay running.  So it’s helpful for a user to confirm they’re trying to connect to the correct port before calling.

The good part

The bottom line is the query to accomplish this is as follows; the table fnd_oam_context_values stores entire context files as clobs in a field called text.

select
extractvalue( xmltype( text ),’//mwaDispatcherPort‘ )
from
fnd_oam_context_files
where
status = ‘S’
and name = ‘SID_hostname
and last_synchronized =
(select
max( last_synchronized )
from
fnd_oam_context_files
where
status = ‘S’
and name = ‘SID_hostname
) ;

Obviously, when you do it, you’re going to change a few things:

  • mwaDispatcherPort is the tag from the $CONTEXT_FILE you’re looking for the value of.  Examples include oa_context_name, jinit_ver_dot, platform, or whatever you’re looking for.
  • SID_hostname is the database SID followed by an underscore and the applications tier hostname.

Why the subquery?  There are many context files stored in this field as clobs.  You want the most recent one (presumably, tweak as desired, based on last_synched) and you want one which has a value of S (synched? not 100% sure about this column, just figured it out through trial and error)

Incidentally, there is a table called fnd_env_context which holds some, but not all of the values contained in the context file.

End Result

We’re gunning for a centralized APEX report which can query the databases and return the relevant port numbers for users, as appropriate, depending on their apps responsibility.  So they can stop calling me so I can blog more.

September 5, 2014

11i Apache v 1.0.2.2.2 and Symbolic Links

Filed under: 11i — kkempf @ 10:48 am

asf-logo

Minding my own business

While cloning 11i environments to try to get to R12.2, I ran into this error when logging in after completing the clone:

Forbidden

You don’t have permission to access /OA_HTML/AppsLocalLogin.jsp on this server.

Wow that’s pretty serious looking.  What did I forget?  Bad values in the context file for ssl certs?  No.  Talking ssl over a port not allowed by our proxy server?  No.  It turns out, I got a little creative in trying to re-arrange environments (on an old 32-bit front end Oracle Linux 5 box) and I built this particular 11i front end off of a symbolic link in Linux pointing to an unused 11i disk path (which has already been converted to R12, on a 64-bit front end Oracle Linux 6 box).  Sometimes you have to do strange things in the name of expediency, and also because the guy who allocates disk from the SAN is out today.

 

The fix

Edit your context file in $APPL_TOP/admin, and change s_options_symlinks from Options -FollowSymLinks to Options +FollowSymLinks.  This writes to httpd.conf and httpd_pls.conf under your IAS Home.

Shut down your services, run autoconfig and restart services.

Use at your own caution, as it’s probably violating 10 different security rules, but I don’t care I am throwing this environment away within 3 days.

June 10, 2014

Let the games begin!

Filed under: 11i, R12.2, Ubuntu — kkempf @ 8:34 am

R12.2.3

R12.2.3 Testing has begun

R12.2 changes a LOT of things, especially for an applications DBA.  We’re underway with our upgrade plans to 12.2, and I was happy to see if you have the patience to make all the pieces work (Chrome, Ubuntu, Java, PopUp Blocker, Chrome Plug-In for Oracle R12) you can actually launch forms from Ubuntu.

Initial Upgrade Impressions

There’s a lot of work to upgrade from 11i to 12.2.3.  The upgrade documents are extensive, and not light reading.  Customizations are a bugger to bring over, the whole landscape has changed with edition based redefinition.  Much of this revolves around the changes to the techstack (Weblogic), and online patching.  The new patching utility, adop, works in conjunction with other old favorite ad utilities, but it’s not terribly intuitive.  There was something reassuring watching my workers progress and run; maybe there’s a way to do it and I just haven’t figured out the appropriate CLI switch.  You can still look at them via adctrl, but it’s not the same, I don’t get the reassurance of knowing I only have 123,000 jobs left to finish.  Mostly, though, things look the same.  More to follow.

April 16, 2014

Password Reset Error

Filed under: 11i, Cloning, R12 — kkempf @ 12:31 pm

Just minding my business

I received an email saying the following error occurs in an 11i cloned instance.  Mind you, this is because we added users to the environment, and by rule Oracle requires them to change their password the first time they login so the password is different than what the system administrator assigned.

Internet Explorer error message:

jsp

 

I’ll spell out a few keywords to the search engines can index this one: AppsChangePassword.jsp java.lang.NoClassDefFoundError JSP Error

If you’re running Chrome, here’s your error message

 

 

 

 

That Was Supposed to be a Joke

From Chrome, I just got a white page, no error, no nothing.  I had to debug this from IE, which just pains me.  Oh yeah, I’m not supposed to use Chrome for Ebusiness suite, even though it works fine and I’ve been using it from a Linux desktop for 8 years now.  Incidentally, there’s a plug in for Chrome to make R12 work, it turns out it does work fine though I have some privacy concerns… you can find it here.

The Fix

First I tried to bounce apache, but that did nothing, so I opened an SR.  I think I found this fix on MOS before the analyst gave it to me, but it didn’t seem like a great fit based on the description in the doc so I didn’t do it until he told me to.  Hat tip to MOS and the ATG team, they identified a fix for an issue quickly and without asking me 10 irrelevant questions first.  Anyways, run this script

$JTF_TOP/admin/scripts/ojspCompile.pl –compile -s ‘AppsChangePassword.jsp’ –flush

December 5, 2013

Cleaning up after Autoconfig

Filed under: 11i, Linux — kkempf @ 12:01 pm

shell

This post could also be called “writing a tiny shell script to notify me of a difference in a config file”.  Short and sweet 2nd post today, I love shell scripting!

Whenever I run autoconfig, it reverts back a value for my Mobile Supply Chain dispatcher port to a value stored in the database somewhere which is wrong.  If I dig into the log, it’s complaining that the port I want it to change to is in use.  Duh, it’s running.  Except it complains when it’s not running also.  The dispatcher is downright critical to getting into Mobile Supply Chain, and Oracle Support is being their usual helpful self.  Except now as of Dec 1, new for 11i: extra ignore!

Let’s assume you have a “good” version of a config file sitting next to one autoconfig wants to keep changing.  This script will email you the differences if they exist.  It assumes you have sendmail setup on your linux host.

cat checkmwacfg.sh

MWACHECK=`diff --brief $MWA_TOP/secure/mwa.cfg $MWA_TOP/secure/mwa.cfg.good|grep -woF differ`
if [ $MWACHECK ]
then
  diff $MWA_TOP/secure/mwa.cfg $MWA_TOP/secure/mwa.cfg.good | mail -s "$MWA_TOP/secure/mwa.cfg file is different" kempf@myemailaddress.com
fi

Crontab it up and you’re done:

# email me bad MWA config file in $MWA_TOP/secure once a day
0 16 * * * /(path to script)/checkmwacfg.sh

Good old ADI

Filed under: 11i — kkempf @ 9:42 am

Applications Desktop Integrator

I can’t really defend the fact that we’re still using ADI, running on 11i.  Some lines of business are resistant to change.  I recently ran into an issue after upgrading to RDBMS 11.2.0.4 and pushing the techstack to Autoconfig U.  It was a bugger, because accounting couldn’t publish their Financial Statements (FSG’s), and to be honest, while the truth resided on Oracle’s Support Site (or MOS), it wasn’t easy to find.

What is ADI?

For those who are blissfully unaware of this tool, it’s essentially a desktop Excel “plug-in” which connects to the database and the 8.0.6 techstack via the RDBMS listener or 8.0.6 listener on the applications tier (depending on what they’re doing).  In a word, it allows accountants to do their business in Excel, where they prefer to work, and then push data to the EBS.  It’s technically only certified for Windows XP, and Oracle has said it’s dead, to be replaced by Web ADI for uploading journal entries and Oracle Report Manager for publishing.  It seems accountants tend to like things just like they were in 1970, so that’s where we’re at.

New Security

As a part of moving to 11.2.0.4, some part of the applications tier packages (I heavily suspect Autoconfig U, but there were other patches), Oracle implemented a “security feature” which in effect just tightened down the 8.0.6 listener.  Generically, you can find the problem description in MOS note 291897.1.  What really happens is that after autoconfig, a protocol.ora file gets landed in your 8.0.6 $TNS_ADMIN directory.  This protocol.ora file has 2 lines (or at least mine did) generated by autoconfig:

tcp.validnode_checking = yes
tcp.invited_nodes = (<database node IP>, <apps tier IP>, <apps tier IP>)

Behavior

As a result of this change, and a regression test “miss”, accounting could log into ADI and post journals, but when they went to publish reports they received this awesome Oracle message:

adi error

I’m typing it out so google can crawl it: An error occurred while attempting to establish an Applications File Server connection.  There may be a network configuration problem, or the TNS listener may not be running.

The Quick Fix

# pound out the 2 lines in protocol.ora, and restart the 8.0.6 listener with adalnctl.sh stop/start.

The Better Fix

Log into 11i as system administrator and change profile option value SQLNet Access to value: ALLOW_ALL at the site level.  In theory this lets you survive an autoconfig run.

The Most Correct Fix (Grudgingly Admitted)

Figure out who needs access, and add them via OAM.  See MOS 281758.1 for details, look under Managed SQL*Net Access from Hosts

Soapbox

Why does Oracle wait 10 years and then change something like this in the terminal release?  Because they can.  While I admit it would be ideal to name every PC from which an accountant could log in to publish a report, the truth is I don’t know when they might get a new machine, or who might be doing it from home over VPN, etc.  It’s a good idea, but somewhat impossible to administer, and I’d venture to guess irrelevant for most installations.  Why not add the feature, but by default DISABLE it?  Just saying.

July 9, 2013

Oracle Mobile Supply Chain Applications (MSCA) : One month after go-live

Filed under: 11i — kkempf @ 10:25 am

brokenc

MSCA vs. Highjump

We replaced a rather over-customized and pretty unstable 3rd party software called (formerly 3M) Highjump with Oracle’s own Mobile supply chain.  In the end, I think the only gain was that now we’re not running the implementation of mobile on a Windows server.  MSCA sits on the 11i application server, and runs* there.  Out of the box, MSCA is rather useless, most transactions require so much irrelevant or derivable (is that a word?) input that by the time you get through the mobile form you feel like you ran a half-marathon.  That said, it does work, it’s faster and more native than a 3rd party bolt-on, and we can lean on Oracle support when we have problems.

* depending upon your definition of run

Failures in Unexpected Places (aka AUTHENTICATION FAILURE)

So I went out to Wikipedia, to confirm that telnet is (slightly) older than I am.  What amazes me is that for a protocol that old, Oracle has managed to completely botch implantation, and yet still charge us for the pleasure of having to deal with it.  Here’s where MSCA is really frustrating.  They can’t keep 3 telnet threads and a round-robin dispatcher (or listener) running right.  Yes, we’re on 11i and thus we’re behind, but it runs off of Java 6 and as I mentioned telnet has been around awhile.  I opened an SR with Oracle on this issue.  The analyst confirmed I was on the latest version of code, then proceeded to tell me most customers bounce the MWA dispatcher and the telnet threads every shift.  Really?  That’s the support answer?  The tech even agreed that it was a bad answer, but the best they could do:

Hi Kevin

I agree with your comment about 198543.1 . When I saw this first time I had same impression but over the years now I have seen almost every customer do start once a day so it is not strange 
any more for me . Base on information you gave you already restarting which is good only other suggestion is increase ports (MWA servers ) if you look at 567214.1 suggestion would be 4 servers 
for 60 users

Add this to the flakiness of the Java MSCA GUI client, and you have a product that’s well implemented and cemented into Oracle behind the scenes, but is an unmitigated disaster to the users.

For whatever reason, every X hours, one of the telnet ports decides to stop working.  When you happen to get that port via round robin (or go to it directly) you get AUTHENTICATION FAILURE messages on the client.  When they get a hold of me to fix it, I first try to issue a graceful stop of the port via $MWA_TOP/bin/mwactl.sh -login apps/password -stop PORT#.  This always returns AUTHENTICATION FAILURE.  So graceful just went out the window.  Now I have to ps -ef |grep PORT# to find the PID on the command line and kill it, then use $MWA_TOP/bin/mwactl.sh start PORT# to restart it.  Really, really lame.

This has happened often enough that I’m going to have to script around it in a rather elaborate and spectacular fashion.  It’s better to script by day, than answer the phone at night, I always say.   In essence, the script will have to do the following on some regular basis:

  • try to stop each port gracefully and see if it returns AUTHENTICATION FAILED
  • if it does get AUTHENTICATION FAILED, grep out the PID for that particular telnet server and kill -9 it
  • restart that telnet server I just killed via MWACTL.sh

The Dispatcher

The one thing this doesn’t do is account for the dispatcher.  When that goes South, it’s game over.  Basically you can find the dispatcher PID by ps -ef|grep’ing for MWADIS.  I have, however, had cases where the MWADIS process is not running but when I use MWACTL.sh to start the dispatcher (only) it tells me it’s already running.  netstat -anp doesn’t see it listening on the assigned port.  It’s really a mess I hope doesn’t pop up in production.

Impressions

While I’m aware that 11i is showing its age, Oracle has still failed to provide a functional incentive for my organization to migrate to R12.  There’s financial reasons (“We’re gonna charge you more!”) there’s techstack reasons (I’d really like to be running on a web server released in the past decade, and I’d like to be running the front end on 64-bit linux) but functionally, we’re not getting anything.  After implementing MSCA, and the 6-months of headache preceding it, I’m in NO hurry to get on the R12 bandwagon.  Why do I mention this?  Because, of course, the answer to my complaints about the stability of the telnet server will be “get on the latest release”.  In my opinion, Oracle has had 40 years to figure out how to make telnet stable, and you’re trying to tell me they just go it in R12?

 

June 18, 2013

Going live on MSCA

Filed under: 11i, Linux, Mobile Supply Chain, Utilities — kkempf @ 3:20 pm

index

Oracle Mobile Supply Chain Application is live!

So we’re live on MSCA now. All things considered it went fine; from a technical perspective the issue which surprised me the most was the instability of the mwa server on the apps tier. I set up a dispatcher with 3 ports (plenty far apart) and at 5am I got a call from Ireland saying that there was only one device in the whole plant which could still connect.

AUTHENTICATION FAILED

The users weren’t getting their password wrong every time, though the message is the same.  Everyone got this “AUTHENTICATION FAILED” message. I’d seen this before; basically it meant that the port was hosed.

The behavior was such that even trying to do a stop_force against a “dead port” returned the message AUTHENTICATION FAILED.  Here’s a snip of my command line sequence to fix it in the middle of the night; basically I tried to gracefully shut down ports 10200, 10300 and 10400. 10300 and 10400 wouldn’t play nicely, so I had to find their PID and kill -9 on them, then restart all 3 ports:

$ cd $MWA_TOP/bin 
$ ps -ef|grep 10200 
oraprod  17027     1  0 Jun16 ?        Sl     0:48 /opt/jdk/bin/java -DPID=17021 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10200 
oraprod  21554 19963  0 05:00 pts/3    S+     0:00 grep 10200 
$ ps -ef|grep 10300 
oraprod  17059     1 30 Jun16 ?        Sl   742:11 /opt/jdk/bin/java -DPID=17053 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10300 
oraprod  21580 19963  0 05:00 pts/3    S+     0:00 grep 10300 
$ ./mwactl.sh -login apps/password stop_force 10200 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Telnet server shut down successfully. 
$ ./mwactl.sh -login apps/password stop_force 10300 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Error: ServerManagerListener returned 'AUTHENTICATION_FAILED' 
mwactl: Error shutting down Telnet server 
$ ./mwactl.sh -login apps/password stop_force 10400 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Error: ServerManagerListener returned 'AUTHENTICATION_FAILED' 
mwactl: Error shutting down Telnet server 
$ ps -ef|grep 10300 
oraprod  17059     1 30 Jun16 ?        Sl   742:57 /opt/jdk/bin/java -DPID=17053 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10300 
oraprod  22006 19963  0 05:01 pts/3    S+     0:00 grep 10300 
$ kill -9 17059 
$ ps -ef|grep 10400 
oraprod   7845     1  0 Jun17 ?        Sl     0:59 /opt/jdk/bin/java -DPID=7843 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10400 
oraprod  22063 19963  0 05:01 pts/3    S+     0:00 grep 10400 
$ kill -9 7845 
$ ps -ef|grep 10200 
oraprod  22169 19963  0 05:01 pts/3    S+     0:00 grep 10200 
$  ./mwactl.sh start 10200
Created server socket : listening on port 10200 
Server startup is successful.
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
$  ./mwactl.sh start 10300
Created server socket : listening on port 10300 
Server startup is successful.
$  ./mwactl.sh start 10400 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
$ Created server socket : listening on port 10400 
Server startup is successful.

mwactl.sh to the rescue

There’s no mwactl.sh status.  If there were, I could log in to the port, get the AUTHENTICATION FAILED error, kill -9 that PID and restart the telnet server on that same port.  So I have to do a “soft shutdown”.  Basically, I wrote a shell script to run every X hours via cron (I’ll start with 12).  It will bring up 3 new ports, and the old 3 ports will gracefully die (meaning, they’ll die after all users have disconnected).

Script to cycle ports with mwactl.sh gracefully

One note: the PASSWD line needs to be customized to your 8.0.6 Apache home.  The wdbsvr.app file contains a plain-text readable copy of your apps password.  At least in 11i.  You can also just hard code the value, but I’ll leverage Oracle’s security hole to my advantage and not propogate random hard coded passwords I have to change everywhere when I need to change the apps password.

#!/bin/bash
# mwa_port_roatate.sh   06/18/13 kmk

# Port Pool 1 : 10200, 10205, 10210, 10215, 10220, 10225

# Port Pool 2 : 10250, 10255, 10260, 10265, 10270, 10275

# mwa.TelnetServer=localhost.com:10200,10205,10210,10215,10220,10225,10250,10255,10260,10265,10270,10275

# Dispatcher: 10800

. /scratch/home/oraprod/env/PROD_APPS
  PASSWD=`cat /u02/appprod/prodias/iAS/Apache/modplsql/cfg/wdbsvr.app|grep -m1 password|sed -re 's/(^.+= )//'`
  USERNAME=`whoami`

if [ -f /tmp/mwa_port_pool1 ]
then
  # Port pool 1 is active, touch port pool 2 file and remove port pool 1 file
  rm -rf /tmp/mwa_port_pool1
  touch /tmp/mwa_port_pool2
 
  echo "====================STARTING=2=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Starting Port Pool 2" >> /tmp/mwa_port.log
  
  # start port pool 2
  echo "10250" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10250 >> /tmp/mwa_port.log 2>&1 &
  echo "10255" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10255 >> /tmp/mwa_port.log 2>&1 &
  echo "10260" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10260 >> /tmp/mwa_port.log 2>&1 &
  echo "10265" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10265 >> /tmp/mwa_port.log 2>&1 &
  echo "10270" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10270 >> /tmp/mwa_port.log 2>&1 &
  echo "10275" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10275 >> /tmp/mwa_port.log 2>&1 &

  sleep 10

  echo "====================STOPPING=1=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Stopping Port Pool 1" >> /tmp/mwa_port.log
  
  # stop port pool 1
  echo "10200" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10200 >> /tmp/mwa_port.log 2>&1 &
  echo "10205" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10205 >> /tmp/mwa_port.log 2>&1 &
  echo "10210" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10210 >> /tmp/mwa_port.log 2>&1 &
  echo "10215" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10215 >> /tmp/mwa_port.log 2>&1 &
  echo "10220" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10220 >> /tmp/mwa_port.log 2>&1 &
  echo "10225" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10225 >> /tmp/mwa_port.log 2>&1 &
 
  sleep 300

  for pids in `ps -fu ${USERNAME}|egrep '(10200)|(10205)|(10210)|(10215)|(10220)|(10225)'|grep -v grep|awk '{print ($2)}'`
  do
      echo "forcing kill on PID ${pids} (LIVE)" >> /tmp/mwa_port.log 2>&1 &
      kill -9 ${pids}
  done

else
# mwa.TelnetServer=localhost.com:10200,10205,10210,10215,10220,10225,10250,10255,10260,10265,10270,10275
  rm -rf /tmp/mwa_port_pool2
  rm -rf /tmp/mwa_port_pool1
  touch /tmp/mwa_port_pool1

  echo "====================STARTING=1=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Starting Port Pool 1" >> /tmp/mwa_port.log
  
  # start port pool 1
  echo "10200" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10200 >> /tmp/mwa_port.log 2>&1 &
  echo "10205" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10205 >> /tmp/mwa_port.log 2>&1 &
  echo "10210" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10210 >> /tmp/mwa_port.log 2>&1 &
  echo "10215" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10215 >> /tmp/mwa_port.log 2>&1 &
  echo "10220" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10220 >> /tmp/mwa_port.log 2>&1 &
  echo "10225" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10225 >> /tmp/mwa_port.log 2>&1 &

  sleep 10

  echo "====================STOPPING=2=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Stopping Port Pool 2" >> /tmp/mwa_port.log
  
  # stop port pool 2
  echo "10250" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10250 >> /tmp/mwa_port.log 2>&1 &
  echo "10255" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10255 >> /tmp/mwa_port.log 2>&1 &
  echo "10260" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10260 >> /tmp/mwa_port.log 2>&1 &
  echo "10265" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10265 >> /tmp/mwa_port.log 2>&1 &
  echo "10270" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10270 >> /tmp/mwa_port.log 2>&1 &
  echo "10275" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10275 >> /tmp/mwa_port.log 2>&1 &

  sleep 300

  for pids in `ps -fu ${USERNAME}|egrep '(10250)|(10255)|(10260)|(10265)|(10270)|(10275)'|grep -v grep|awk '{print ($2)}'`
  do
      echo "forcing kill on PID ${pids} (LIVE)" >> /tmp/mwa_port.log 2>&1 &
      kill -9 ${pids}
  done

fi

# cool command to return PIDS for all telnet servers
#
# ps -ef|egrep '(10200)|(10300)|(10400)|(10500)|(10600)|(10700)'|grep -v grep|awk '{print ($2)}'

February 12, 2013

Running the Oracle Mobile Supply Chain (MSCA) GUI on Ubuntu 12.1

Filed under: 11i, Ubuntu — kkempf @ 1:34 pm

MSCA Overview

Oracle’s 11i Mobile Supply Chain (which is itself a small piece of their warehouse application) is essentially a telnet version of the forms relevant to certain business areas.  Why, you might ask?  As a manufacturer, we find it rather convenient to be able to perform Work in Process and Shipping transactions from a hand-held wireless device, instead of being tied down to a PC with a browser logged into Oracle professional forms.  In short, it’s a small footprint, narrowed-down version of 11i forms.  My original post is here, but it’s a bit dated.

Patches

What would an Oracle product implementation be without a bunch of patches?  To get this product cooking right, you need to put in 5712178 (allows you to start/stop the applications tier telnet server via adstrtal.sh/adstpall.sh) and 8305261 (fixes a bug which didn’t allow you to actually stop the telnet server via adstrtal.sh/adstpall.sh).

Why Ubuntu?

You can find plenty of articles about how to install the MSCA client on Windows.  But I don’t like to run Windows; it’s clumsy and tries too hard to out-think me (not hard to do).  Ubuntu is a nice compromise, because it is still linux, but has good desktop support (drivers for wifi and displays come to mind).  I especially like being able to right click and open a terminal (sudo apt install nautilus-open-terminal) and type sqlplus apps@prod and be able to connect to my database directly.  More on installing the Oracle 11g client in the next post.

Oracle’s GUI

It’s not pretty.  It’s basically a java GUI to wrap around a telnet session.  But it is much nicer than telnet.  Incidentally, don’t hit the tab key, ever, when you’re using it.  Despite the fact that it’s almost second nature to do so, it’s not supported unless you’re using java 1.1.8 (yep, the one from 1999) and if you do hit tab, the GUI gets totally hosed.  I don’t know how to explain it better.  Oracle is aware of the bug, but since they haven’t solved it since 1.3 came out, I don’t have a lot of faith a patch is eminent.

Download the MSCA patch

Basically, to use this GUI you need to pull patch 4205328.  There is an updated version of this patch which I could not get to work (more on that later).  Extract the zip file and get the j4205328.zip file, that’s all you need.

Setup your environment

Create an MWA install directory.  In my case, I made /usr/local/oracle/msca/lib and /usr/local/oracle/msca/log.  Then I created a runMSCA.sh script which looks like this (you may need to tweak the JAVA_HOME to match your settings):

JAVA_HOME=/opt/java/jre1.6.0_38
#JAVA_HOME=/opt/java/jdk1.6.0_38
MWA_GUI_TOP=/usr/local/oracle/msca
$JAVA_HOME/bin/java -classpath $MWA_GUI_TOP/lib/j4205328.zip oracle.apps.mwa.awt.client.StartGUI

Note that you can point to either a JRE or JDK top; either work.  chmod +x your script and you’re basically done.

Launch it

In my case, runMSCA.sh is on my Ubuntu Desktop.   So I double clicked it

runMSCA.sh

Click Run

Now you get to the host screen

host login

Specify the Site Name, Host name and port

This can be defaulted through MSCA configuration

This can be defaulted through MSCA configuration

Login with your normal apps 11i credentials

Login with your normal apps 11i credentials

Select an MSCA responsibility

Select an MSCA responsibility

menu items

Ready for business!

Updated GUI versions

While I know we got the newer version of the GUI to run on Windows, it wouldn’t cooperate with me on Ubuntu.  The newer patch is 12780257 which supposedly fixed some bugs (curiously, though, not the tabbing issue!).  When I tried to put my launcher together, it crashed and burned

msca2.sh

Seems straight forward

 

msca2.sh_error

Conclusion

You can run the GUI from Ubuntu (or any Linux version which has a JDK/JRE available) with a little improvisation.  Perhaps I made a simple mistake on the newer version, if so I’d love to hear about it.  In the meantime, I can at least regression test mobile apps from my preferred desktop!

 

 

 

Older Posts »

Blog at WordPress.com.