Kevin Kempf's Blog

June 18, 2013

Going live on MSCA

Filed under: 11i, Linux, Mobile Supply Chain, Utilities — kkempf @ 3:20 pm

index

Oracle Mobile Supply Chain Application is live!

So we’re live on MSCA now. All things considered it went fine; from a technical perspective the issue which surprised me the most was the instability of the mwa server on the apps tier. I set up a dispatcher with 3 ports (plenty far apart) and at 5am I got a call from Ireland saying that there was only one device in the whole plant which could still connect.

AUTHENTICATION FAILED

The users weren’t getting their password wrong every time, though the message is the same.  Everyone got this “AUTHENTICATION FAILED” message. I’d seen this before; basically it meant that the port was hosed.

The behavior was such that even trying to do a stop_force against a “dead port” returned the message AUTHENTICATION FAILED.  Here’s a snip of my command line sequence to fix it in the middle of the night; basically I tried to gracefully shut down ports 10200, 10300 and 10400. 10300 and 10400 wouldn’t play nicely, so I had to find their PID and kill -9 on them, then restart all 3 ports:

$ cd $MWA_TOP/bin 
$ ps -ef|grep 10200 
oraprod  17027     1  0 Jun16 ?        Sl     0:48 /opt/jdk/bin/java -DPID=17021 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10200 
oraprod  21554 19963  0 05:00 pts/3    S+     0:00 grep 10200 
$ ps -ef|grep 10300 
oraprod  17059     1 30 Jun16 ?        Sl   742:11 /opt/jdk/bin/java -DPID=17053 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10300 
oraprod  21580 19963  0 05:00 pts/3    S+     0:00 grep 10300 
$ ./mwactl.sh -login apps/password stop_force 10200 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Telnet server shut down successfully. 
$ ./mwactl.sh -login apps/password stop_force 10300 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Error: ServerManagerListener returned 'AUTHENTICATION_FAILED' 
mwactl: Error shutting down Telnet server 
$ ./mwactl.sh -login apps/password stop_force 10400 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
Error: ServerManagerListener returned 'AUTHENTICATION_FAILED' 
mwactl: Error shutting down Telnet server 
$ ps -ef|grep 10300 
oraprod  17059     1 30 Jun16 ?        Sl   742:57 /opt/jdk/bin/java -DPID=17053 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10300 
oraprod  22006 19963  0 05:01 pts/3    S+     0:00 grep 10300 
$ kill -9 17059 
$ ps -ef|grep 10400 
oraprod   7845     1  0 Jun17 ?        Sl     0:59 /opt/jdk/bin/java -DPID=7843 -Doracle.apps.mwa=/u02/appprod/prodappl/mwa/11.5.0 -mx512m -ms128m oracle.apps.mwa.presentation.telnet.Listener 10400 
oraprod  22063 19963  0 05:01 pts/3    S+     0:00 grep 10400 
$ kill -9 7845 
$ ps -ef|grep 10200 
oraprod  22169 19963  0 05:01 pts/3    S+     0:00 grep 10200 
$  ./mwactl.sh start 10200
Created server socket : listening on port 10200 
Server startup is successful.
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
$  ./mwactl.sh start 10300
Created server socket : listening on port 10300 
Server startup is successful.
$  ./mwactl.sh start 10400 
MWA Telnet Server Release: 1.0.8.4 [December 12th 2002] 
$ Created server socket : listening on port 10400 
Server startup is successful.

mwactl.sh to the rescue

There’s no mwactl.sh status.  If there were, I could log in to the port, get the AUTHENTICATION FAILED error, kill -9 that PID and restart the telnet server on that same port.  So I have to do a “soft shutdown”.  Basically, I wrote a shell script to run every X hours via cron (I’ll start with 12).  It will bring up 3 new ports, and the old 3 ports will gracefully die (meaning, they’ll die after all users have disconnected).

Script to cycle ports with mwactl.sh gracefully

One note: the PASSWD line needs to be customized to your 8.0.6 Apache home.  The wdbsvr.app file contains a plain-text readable copy of your apps password.  At least in 11i.  You can also just hard code the value, but I’ll leverage Oracle’s security hole to my advantage and not propogate random hard coded passwords I have to change everywhere when I need to change the apps password.

#!/bin/bash
# mwa_port_roatate.sh   06/18/13 kmk

# Port Pool 1 : 10200, 10205, 10210, 10215, 10220, 10225

# Port Pool 2 : 10250, 10255, 10260, 10265, 10270, 10275

# mwa.TelnetServer=localhost.com:10200,10205,10210,10215,10220,10225,10250,10255,10260,10265,10270,10275

# Dispatcher: 10800

. /scratch/home/oraprod/env/PROD_APPS
  PASSWD=`cat /u02/appprod/prodias/iAS/Apache/modplsql/cfg/wdbsvr.app|grep -m1 password|sed -re 's/(^.+= )//'`
  USERNAME=`whoami`

if [ -f /tmp/mwa_port_pool1 ]
then
  # Port pool 1 is active, touch port pool 2 file and remove port pool 1 file
  rm -rf /tmp/mwa_port_pool1
  touch /tmp/mwa_port_pool2
 
  echo "====================STARTING=2=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Starting Port Pool 2" >> /tmp/mwa_port.log
  
  # start port pool 2
  echo "10250" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10250 >> /tmp/mwa_port.log 2>&1 &
  echo "10255" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10255 >> /tmp/mwa_port.log 2>&1 &
  echo "10260" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10260 >> /tmp/mwa_port.log 2>&1 &
  echo "10265" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10265 >> /tmp/mwa_port.log 2>&1 &
  echo "10270" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10270 >> /tmp/mwa_port.log 2>&1 &
  echo "10275" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10275 >> /tmp/mwa_port.log 2>&1 &

  sleep 10

  echo "====================STOPPING=1=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Stopping Port Pool 1" >> /tmp/mwa_port.log
  
  # stop port pool 1
  echo "10200" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10200 >> /tmp/mwa_port.log 2>&1 &
  echo "10205" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10205 >> /tmp/mwa_port.log 2>&1 &
  echo "10210" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10210 >> /tmp/mwa_port.log 2>&1 &
  echo "10215" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10215 >> /tmp/mwa_port.log 2>&1 &
  echo "10220" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10220 >> /tmp/mwa_port.log 2>&1 &
  echo "10225" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10225 >> /tmp/mwa_port.log 2>&1 &
 
  sleep 300

  for pids in `ps -fu ${USERNAME}|egrep '(10200)|(10205)|(10210)|(10215)|(10220)|(10225)'|grep -v grep|awk '{print ($2)}'`
  do
      echo "forcing kill on PID ${pids} (LIVE)" >> /tmp/mwa_port.log 2>&1 &
      kill -9 ${pids}
  done

else
# mwa.TelnetServer=localhost.com:10200,10205,10210,10215,10220,10225,10250,10255,10260,10265,10270,10275
  rm -rf /tmp/mwa_port_pool2
  rm -rf /tmp/mwa_port_pool1
  touch /tmp/mwa_port_pool1

  echo "====================STARTING=1=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Starting Port Pool 1" >> /tmp/mwa_port.log
  
  # start port pool 1
  echo "10200" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10200 >> /tmp/mwa_port.log 2>&1 &
  echo "10205" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10205 >> /tmp/mwa_port.log 2>&1 &
  echo "10210" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10210 >> /tmp/mwa_port.log 2>&1 &
  echo "10215" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10215 >> /tmp/mwa_port.log 2>&1 &
  echo "10220" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10220 >> /tmp/mwa_port.log 2>&1 &
  echo "10225" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh start 10225 >> /tmp/mwa_port.log 2>&1 &

  sleep 10

  echo "====================STOPPING=2=====================================" >> /tmp/mwa_port.log
  date >> /tmp/mwa_port.log
  echo "Stopping Port Pool 2" >> /tmp/mwa_port.log
  
  # stop port pool 2
  echo "10250" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10250 >> /tmp/mwa_port.log 2>&1 &
  echo "10255" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10255 >> /tmp/mwa_port.log 2>&1 &
  echo "10260" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10260 >> /tmp/mwa_port.log 2>&1 &
  echo "10265" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10265 >> /tmp/mwa_port.log 2>&1 &
  echo "10270" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10270 >> /tmp/mwa_port.log 2>&1 &
  echo "10275" >> /tmp/mwa_port.log
  nohup $MWA_TOP/bin/mwactl.sh -login apps/${PASSWD} stop 10275 >> /tmp/mwa_port.log 2>&1 &

  sleep 300

  for pids in `ps -fu ${USERNAME}|egrep '(10250)|(10255)|(10260)|(10265)|(10270)|(10275)'|grep -v grep|awk '{print ($2)}'`
  do
      echo "forcing kill on PID ${pids} (LIVE)" >> /tmp/mwa_port.log 2>&1 &
      kill -9 ${pids}
  done

fi

# cool command to return PIDS for all telnet servers
#
# ps -ef|egrep '(10200)|(10300)|(10400)|(10500)|(10600)|(10700)'|grep -v grep|awk '{print ($2)}'
Advertisements

Blog at WordPress.com.