Kevin Kempf's Blog

March 9, 2015

R12.2 Apache won’t start up

Filed under: Linux, Oracle Linux, R12.2, Weblogic — kkempf @ 9:45 am

magnifying-glass-clipart-biy5E46iL

Mondays

I had to bounce my R12.2 non-production front end this weekend to switch kernels around.  I restarted services, but admit I didn’t pay much attention to them as it wasn’t PROD.  We run multiple instances of R12.2 on different port pools but on one host (well 2, one for the databases, one for the application servers). We do this not because I like it, but because we don’t have infinite budgets.  Regardless, I get a complaint that our audit environment isn’t working.  Sure enough, the login is unavailable, and Weblogic says the web tier process EBS_web_AUDIT is down.

Identify the problem: To the command line!

I try to simply start it in Weblogic but it errors out, and there’s far too many log links to click for this admin to stay in a GUI.   I run adapcctl.sh start and get:

You are running adapcctl.sh version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance …

adapcctl.sh: exiting with status 204

Follow the bouncing ball

adapcctl.sh tells me to check my log file: $INST_TOP/logs/appl/admin/log/adapcctl.txt, and here’s what it tells me:

ias-instance id=EBS_web_AUDIT_OHS1
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
  EBS_web_AUDIT/OHS/OHS/

Error
--> Process (index=1,uid=1810789619,pid=11169)
  failed to start a managed process after the maximum retry limit
  Log:
  /u02/appaudit/fs1/FMW_Home/webtier/instances/EBS_web_AUDIT_OHS1/diagnostics/logs/OHS/EBS_web_AUDIT/console~OHS~1.log


03/09/15-09:52:40 :: adapcctl.sh: exiting with status 204
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address [::]:10059
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address 0.0.0.0:10059
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] no listening sockets available, shutting down
[2015-03-09T09:38:34.0591-04:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] Unable to open logs

Closing in

So it appears that my user (oraaudit) can’t get a port it wants (10059), and after a little hunting on Google I find the right syntax. I wanted to use netstat, but I couldn’t figure out the PID and lsof made it really easy:

# lsof -i:10059
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    13815 oratrain  305u  IPv6  82113      0t0  TCP (myhostname).com:10059->(myhostname).com:rds2 (ESTABLISHED)

Take down the suspect

It’s some random java process from another instance (Train). See ya.

# kill -9 13815
# lsof -i:10059

Restart Apache

adapcctl.sh start

You are running adapcctl.sh version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

Nothing to see here

This is something most apps DBAs have seen at some time or in some form. The path to follow changes, but in the end the fix is usually similar to this. Knowing with certainty that it’s OK to kill the blocking PID is always a delicate task. There’s absolutely no way you could have fixed this from within the Weblogic admin console that I can think of, short of going into the other environment and shutting it down completely.

Advertisements

Blog at WordPress.com.