EBS R12.2 Security

That light at the end of the tunnel might be a freight train

If you’re running 12.2 and considering 12.2.6 any time in the near future, you need to be aware of significant changes coming to the security requirements of EBS.  At the East Coast Oracle conference in Raleigh a few weeks back, I sat in a session by Elke Phelps (Oracle) entitled “Ready or Not: Applying Secure  Configuration to Oracle  E – Business Suite“.   Oracle EBS is getting serious, if not belligerent, about security.

Bottom line: If you don’t comply with their security recommendations, your users will not be able to log into EBS.  That’s right.  This is probably the most concerning thing about the changes: they’re not suggested.  Don’t believe me?  Check out Doc 2174164.1 section 3.1:

The Secure Configuration Console automates the security configuration process by consolidating the security configuration process onto one user interface and creating a single checkpoint entry into the system. It checks your system against 16 high-priority security configuration guidelines and makes recommendations to the system administrator to either fix or suppress the failure. Until the system administrator acknowledges these checks, users will be denied entry into the system.

Do I have you attention now?

Here’s the thing: most of the things it appears to be concerned with are either simple fixes or common sense.  While I haven’t seen it myself, I’ve been assured there is a page somewhere (in OAM?) where the apps DBA can go in and check boxes saying “I understand I’ve broken security rules, now please let me use my ERP”. Think: DEV environments.

One problem with all of this is it’s implemented in Oracle’s typical unwieldy and disparate manner.  Start by grinding your way through the 468 page Oracle Ebusiness Suite R12.2 security guide.  Let me just say it’s not a real page turner and I’ve had less reading in graduate level courses.

EBSSecConfigChecks.sql

So we move on to 2069190.1 where you can find a nifty set of scripts called EBSSecConfigChecks.zip.  This is run against your database, and comes back with somewhat actionable results.  I will admit, it found some things I wasn’t aware of, but it’s not perfect.  By the numbers:

• Check: Security Profiles: Configuration ERRORS
  • This checks POVs to see if you’re doing anything wrong.  In my case, it didn’t like a site level POV called Framework Validation Level (value was set to none).  The problem with this assessment, is that this isn’t a POV I can find in any way.  As in, I can’t query it up in forms, and when I try to use fnd_profile.save to change it, it errors.  First check and I have to open an SR.  Great start!
• Check: Security Profiles: Configuration WARNINGS
  • This appears to check and see who has the ability to run diagnostics in EBS, as well as who can attach files and personalize self service bits.  The problem is, it doesn’t bother to check and see if the EBS user with these privs is end dated.  So I wound up with a list of end-dated consultants from our R12.2 upgrade, none of which can log in, and a list of IT super users.
• Check: Security Profiles: Configuration MISSING
  • No idea, I “passed” this one
• Check: Application Users with Default Passwords
  • I’ll give Oracle credit on this one, it found some interesting things here.  I had to change my guest password, and disable autoinstall.  Hint: try to find that user.. when you get tired of that game, run FND_USER_PKG.DISABLEUSER(‘AUTOINSTALL’);
  • It also found some schemas which had the default password.  As in, all the new schemas presumably introduced by the 12.2 upgrade, so kudos to Oracle there.  Mine were ddr, dna, dpp, gmo, ibw, inl, ipm, jmp, mth, qpr and rrs.  Obviously I hadn’t run FNDCPASS against “ALL” since the upgrade, and so these were just hanging around.
• Check: DB Users With Default Passwords
  • Again, a handful out there, but if the script would exclude “EXPIRED & LOCKED” that’d be great.  Most significant was applsyspub.  I swear it used to be you were told not to change this, but it’s easy enough to fix with FNDCPASS
• Check: For excessive privs in APPLSYSPUB
  • Passed this one
• Check: Oracle Applications User Passwords Migrated to Non-Reversible Hash Password
  • Yeah, I hadn’t done this.  It’s an easy fix see 457166.1
• Check: Server Security Status
  • Passed this one
• Check: SSL Status
  • You need to be using TLS 1.2.  See 2143101.1.  If you’re not familiar with the process, welcome to a lot of work.  And now you get to keep up with an external signing authority!  Yay work!
• Check: Credit Card Encryption Status
  • Beats me, I failed this, but we don’t store this in the database, so perhaps that would be helpful to note.
• Check Status of 12.2 Security Features
  • I had a recommendations here, because I allowed unrestricted JSP access,  unrestricted redirects and a setting it didn’t like for Cookie Domain.  This is probably legitimate, but it’s going to cause havoc with our Oracle APEX implementation and will likely take time to fix.  There’s a script called txkCfgJSPWhitelist.pl in $FND_TOP/patch/115/bin which may help you (“Oracle E-Business Suite Release 12.2.6 delivers a configuration script which can assist in configuring the products in your Allowed JSP lists. Products will be turned off in the Allowed JSP family configuration files based on whether recent transactions are detected for the product. Customers are strongly recommended to configure the Allowed JSPs using this script.”).  Check out the Oracle EBS Security Guide
• Check: Users with Access to Sensitive Pages
  • Disabled by default.  I guess I pass?

Where this falls apart for me

They’re going to hold your EBS hostage until you address security gaps?  Unacceptable.  While it’s true, you’re not going to wake up one morning to users unable to log into production, it’s still a rather arrogant shift.  I happen to know my company paid a fortune for their software, but now I’m not so confident we own it.  This feels like some back room legal department at Oracle decided to get a record that “so and so” at XYZ company deliberately agreed to ignore our security recommendations, and we have it on the record because they checked boxes acknowledging they assume the risk.  Except the reciprocal is not true.  If I get hacked despite having complied with Oracle recommendations, I’m pretty sure Oracle isn’t going to write me a check to help me fix it.

Oracle is making assumptions about my environment and punishing me if I don’t comply.  What if I have no internet exposure?  What if this is a DEV environment where I don’t care or can’t afford to keep up with TLS 1.2?

Instead, a warning message (old school nag screen) at login to any user stating something like “Your corporate EBS may be at risk to security vulnerabilities, please contact your system administrator” would have been equally effective.  I’m not going to let that screen be up there for the world to see, and I will get phone calls.  But if something happens beyond my control, my users can still get into EBS.

Determining your EBS Code Level and Family Pack

Quick and Dirty

There’s lots of posts out there telling you how to determine your EBS patch levels, code levels, etc.  But whenever I google them, I can’t find the one that actually tells me what I want in a simplified way.

Here’s what matters most often to me as the Apps DBA:

select
  abbreviation
,codelevel
from
  ad_trackable_entities
where
  abbreviation in( 'ad','txk','fnd','fwk','atg_pf','icx' )
order by
  abbreviation;

 

ABBREVIATION                   CODELEVEL                                                                                                                                            
------------------------------ ------------------------------------------------------------------------------------------------------------------------------------------------------
ad                             C.7                                                                                                                                                   
atg_pf                         C.4                                                                                                                                                   
fnd                            C.4                                                                                                                                                   
fwk                            C.4                                                                                                                                                   
icx                            D.3                                                                                                                                                   
txk                            C.7                                                                                                                                                   

 6 rows selected

How to Migrate an Oracle database host in R12.2 using LVM

Painting the backdrop

I realize this post is somewhat specific to your setup, but I believe it may hold value to some folks out there so I thought I’d formalize it.  In my case, I have a rather large (1TB+) database residing on Oracle Linux (RHEL) 5 which serves as the back end for my EBS 12.2 environment.  We do not use ASM nor RAC, but do use Linux LVM (logical volume manager) to make growing disks easier.  The disk itself is on a SAN in the data center, so in my case this process involves some assistance from the systems/network folks.

I’m moving from OL5 to OL6 because support for OL5 is running down.  Let’s face it: the database works great on OL5, and I have no compelling reason to migrate it, but this is part of our lives: upgrades for the sake of upgrading.

For the purposes of this post, I’ll simply use oldhost as the OL5 hostname, and newhost as the OL6 hostname for the database, and appshost as the hostname for the applications tier.

In the olden days

Oracle used to have a bonafide methodology for migrating the database tier, in 11i it’s spelled out under DOC 338003.1.  It made sense; it said to use the tech stack on the front end to tell the application tier that there’s a new host for the database.

The R12.2 solution

I opened an SR to get the equivalent document for 12.2 and the analyst basically said “Follow Doc ID 1968231.1 to use logical hostnames, you can sort of use autoconfig but we don’t support it”.  I don’t know if that’s really the best answer, but it was all I had to go with.  I had been planning to use DNS as a safety net, not the primary vehicle of changing the database hostname.  But apparently it’s the only vehicle now, despite the fact that I enter hostname on the front end in the context file.  While I think this is a terrible answer, that’s not the point of my post so I’ll let it go.

LVM Setup (oldhost)

In my environment, I set up my lvm volume groups like this:

Data1 - product datafiles for EBS


Data2 - more product datafiles for EBS


Archivelogs - archivelog destination


Redo - online redo


System - system/sysaux datafiles


RDBMS - Oracle database installation (binaries)


As you can imagine, the files contained in these groups are pretty large; well over a terabyte.  In an ideal situation, I’d take the database down cold and simply rsync the files to the new host in the same location, then crank everything up.  But that would take hours I don’t have, so I went a different route.  It’s worth noting, that on OL5, my disks on oldhost are ext3, and OL6 delivers ext4.  Since I’m moving the disks “as is”, I’m getting ext3 filesystems on newhost, the OL6 server.  It’s compatible, and something I just have to live with.

I feel like I should mention at this point that probably the very first step was to build newhost as an OL6 environment with appropriate cpu and memory.

LVM Prep (oldhost)

• The first step is to shut everything down, obviously, front and back end on oldhost and appshost.
• Next, I unmount all the volumes pertaining to the environment (in this case, dev).  For my environment, each of these corresponds to an LVM volume group
  • umount /u01/appdev
  • umount /u03/appdev
  • umount /u04/appdev
  • umount /u05/appdev
  • umount /u06/appdev
  • umount /usr/local/oracle/archive
• Set all the volume groups to inactive
  • vgchange -an Data2
  • vgchange -an Data1
  • vgchange -an Archivelogs
  • vgchange -an Redo
  • vgchange -an System
  • vgchange -an RDBMS
• Export the volume groups
  • vgexport Data2
  • vgexport Data1
  • vgexport Archivelogs
  • vgexport Redo
  • vgexport System
  • vgexport RDBMS

Disk manipulation

At this point, the volumes can be safely manipulated by the disk admins.  What this entails will vary greatly based on  your datacenter, and is outside the scope of linux so I’m not going to detail it here.  The gist of it is that your admin needs to remove the disks/virtual disks from the old host and install them on the new one by whatever means is appropriate.  I apologize in advance if I’m not saying this part correctly.

Adding the disks (newhost)

The first thing to try (assuming your new host received the disks “hot”) is simply

pvscan

They may just show up.  If not, you can try rescanning the scsi host:

ls /sys/class/scsi_host/ | whileread host ; do echo "- - -"> /sys/class/scsi_host/$host/scan ; done

-or this-


echo "- - -"> /sys/class/scsi_host/(host#, hit tab or guess)/scan

-for example this may turn into:

 echo “- – -” > /sys/class/scsi_host/host0/scan

echo “- – -” > /sys/class/scsi_host/host1/scan

echo “- – -” > /sys/class/scsi_host/host2/scan

Then issue

fdisk -l

and it’s worth noting you can watch /var/log/messages for the system to recognize new disks being added.  When all else fails, a reboot has never failed to get all disks recognized.

At this point, pvscan should show all your volume groups

Final steps (newhost)

It’s kind of the opposite of the prep steps:

• Import the volume groups
  • vgimport Data2
  • vgimport Data1
  • vgimort Archivelogs
  • vgimport Redo
  • vgimport System
  • vgimport RDBMS
• Active the volume groups
  • vgchange -ay Data2
  • vgchange -ay Data1
  • vgchange -ay Archivelogs
  • vgchange -ay Redo
  • vgchange -ay System
  • vgchange -ay RDBMS
• Mount the disks (and put them in /etc/fstab so they survive a reboot!)
  • mount /dev/mapper/RDBMS-Dev /u01/appdev
  • mount /dev/mapper/System-Dev /u03/appdev
  • mount /dev/mapper/Redo-Dev /u04/appdev
  • mount /dev/mapper/Data1-Dev /u05/appdev
  • mount /dev/mapper/Data2-Dev /u06/appdev
  • mount /dev/mapper/Archivelogs-Logs /usr/local/oracle/archive

Final Steps

In my case, I had to do 3 final steps to comply with Oracle’s document and make everything work:

• change /etc/hosts on appshost to explicitly refer to the newhost as oldhost… just in case
• change listener.ora on newhost to be the new hostname on the database ($TNS_ADMIN)
• add DNS record to change calls to old host to new one
• Start up the database (newhost)

Parting Shots

This method took me about 30 minutes.  If the database were smaller, I’d simply rsync the ext3 filesytems to new disks built as ext4 on newhost.

 

 

When ADOP breaks because of the magic of checkfile

As usual, it’s been some time since I’ve thrown anything out here.  What’s been keeping me busy has been largely to do with ADOP.  I have a real love/hate relationship with this new addition to the EBS family, and lately it’s been more hate.

Oracle defines nocheckfile as an unsupported option which isn’t to be used unless explicitly stated in the readme for a patch.  Intriguing, wonder what that’s all about, why would it be there at all?  Let’s start from the beginning, defining default adop behavior and what checkfile does.  In theory, during the adop apply phase, the default option “checkfile” says “go see if I already know I have a more advanced or equal version of this code before I apply it”.  If it thinks it already has ingested this particular bit of code, don’t process it in an effort to save time during the patch.  Couple of key things here to note.  First, it’s trying to save me time.  Second, is it ever wrong about whether it’s already ingested an advanced version of the code?

First, let’s talk about saving time.  That’s a good thing.  If I only have to ingest 30% of a large patch, even using adop with the EBS available, that’s a good thing.  Now the second part.  It’s only good if it’s accurate.  100% accurate.  Because if it skips code I actually need, it turns out you wind up with a big, stinking mess to sort through.  There’s a little foreshadowing there.

Take this scenario, which you’ve probably guessed by now happened to me.  Some bug surfaced as a result of some aborted patch cycles which made my EBS adop cycle think it could safely skip pieces of a patch which it had thought were applied.  Only they weren’t, that patch cycle was aborted.

Here’s your first hint that you may be experiencing this: you get a warning during the apply phase that the patch already exists in your system, but you know that not to be the case.  At this point, Oracle thinks the patch is in, but you know it’s not.  So you innocently go on and use forceapply, per the adop “HINT”.  Now you also think the patch is in your system.  But it’s not.  Because of the magic of checkfile, and the fact that the patch was previously aborted and adop thinks it’s in there, you’re in a world of hurt now.

It’s a really insidious state to find yourself in, and the fix is to reapply the patches, using nocheckfile and forceapply.  This basically says “apply the patch despite the fact that you already think it’s installed, and force every bit of code into the system regardless of whether you think you already have it”.

I can tell you this: never run an adop abort without a full cleanup, as well as an fsclone to reset your patch environment.  That should keep you in good shape.  Officially Oracle now advertises this in Oracle E-Business Suite Maintenance Guide, Release 12.2 (Part Number E22954-20).  I strongly recommend reading that from cover to cover.  Because early on in 12.2, this wasn’t how adop abort was explained even by Oracle University.

My EBS R12.2 adop patch failed, now what?

adop fails a lot, especially when you’re building a regression test environment.  Remember the old DOS days when you yanked out the floppy disk but still had a: on the screen and DOS said (a)bort, (r)etry, (f)ail?  And abort and fail always seemed like the same thing, but somehow they weren’t?  Same thing with adop, only it calls them abandon and restart.

adop defaults to abandon=no restart=yes if not specified, but to me, that’s still a bit unclear.  For example, what the heck would abandon=yes restart=no do?  Here’s what I came up with

(1) abandon and cleanup (I strongly urge you to do a full cleanup, or you may be asking for trouble)

adop phase=abort,cleanup cleanup_mode=full
adop phase=fs_clone

(2) fix the problem via whatever means (adctrl in the patch environment, drop the index the patch is trying to re-create, etc) and retry from where it failed

adop phase=apply patches=17020683 restart=yes [abandon=no is implied and not required]

(3) fix what’s wrong but retry from the beginning of the patch, or try a new patch (why you would do the latter, I have no idea)

adop phase=apply patches=17020683 abandon=yes [restart=yes is implied and not required]   (same patch)
adop phase=apply patches=17893964,18497540 abandon=yes          (new patch)

 

R12 Take Aways and Warnings Part 1 (Overview)

What could possibly go wrong?

I put my thoughts together, and figured I could cite some specifics for the benefit of anyone who cares to listen.  I’ll be honest, functionally, R12.2 behaves pretty similarly to 11i.  The most significant impact for us was in the Financials with subledger accounting, but our WIP, Sales, HR, Purchasing, Order Management, Inventory, and Quality mostly just worked out of the box.  Sure we had to tweak all our custom reports, but the end-user experience was largely unchanged.

Most of the problems encountered lie in the techstack, and really, that’s why you’re here reading this, I suppose.  So without further ado…

Documentation: Before you Start

Wow, it’s ever-changing under your feet.  In the real world, you pick a baseline and have to dance with that partner all the way through the upgrade process.  In our case, this was 6 months.  Many, many things changed during that span, and it’s not easy to stay on top of.  If you’re like me by the time you get through your first pass against 12.2.3 you’ll have a stacks and piles of documents all over your desk.  This post is by no means meant to be all-encompassing, but the following are sure bets:

Here’s your first stop for information for 12.2.3 (it stands to reason there’s an equivalent on for 12.2.4): Doc ID 1586214.1  If you look at the end of the document, there’s a change log.  During the relevant span of my upgrade (1-May-14 through 1-Dec-14) there were many changes, some really quite critical.  This is really important to note: you’re on the bleeding edge here, I don’t care what Oracle says.  This document is shifting under your feet in critical ways and you need to throw out your printed copy and reprint it every time it changes.

For your database and the new techstack, I followed Doc ID 1594274.1 Again, you’ll notice a massive change log.  This is partially due to the changes as a result of new releases of the startCD.  When I began working the project, I chose the most advanced version I could (47) and I see now there’s a 48 and 49.  I strongly urge you to use the latest version.  With each one you get newer pieces and fixes for your techstack, which reduces your upgrade time in the long run because you don’t have to manually apply so many patches.

Finally, subscribe to and follow the updates from Steven Chan’s blog.  There’s nothing else out there like it that I’m aware of, and there were many relevant, timely updates which helped me through the process of the upgrade.

Timing is Everything

It probably goes without saying, but choose whatever the latest version is you can.  In our case, we started on 12.2.3 in May, and sometime before December 12.2.4 was released.  Of course, 12.2.4 contains fixes and enhancements we’d like to have, but as Don Rumsfield said, “you go to war with the Army you have” so we stuck with 12.2.3.  I have to admit, it was tempting to consider 12.2.4 but it would have meant a project re-baseline, and nobody wanted that.

Education & Training

Go ahead and check out Oracle education, and see how many classes there are on Ebusiness Suite 12.2.  In May, there were 0.  The first run of an online patching virtual (online) class was right around July 4th, and I took it.  Good class, by the way.  My point is this: Oracle is light years behind on providing training on this release.  I don’t know what the hold up is, but you’re pretty much on your own.  Go to conferences, read blogs, attend webinars, do whatever you can, because at this time, Oracle Education isn’t going to help you much.

Support

This was an unexpected surprise.  Perhaps I’d just gotten used to response times for 11i (which was basically on life support) but WOW!  Oracle really responds quickly to R12.2 issues.  It’s refreshing, and to that I say kudos to support.

Tax

If you’re running financials, you probably already know that there are significant changes to the tax engine in R12.  I’m not a functional guy, but I have done monthly Vertex updates, and now that I’ve gone through the upgrade I also had to do the installation of the latest compatible Vertex version.  And that’s not the half of it, there’s something called a Tax Regime which had to be recreated or adjusted, and all the while Oracle is pushing you to use EBTax which is their version of tax.  We skipped EBTax because it didn’t replace Vertex completely, and to my understanding still required us to source tax changes from elsewhere.  Regardless, know that you need to address tax, and from a functional standpoint there may not be a lot of knowledge about R12.2 tax.  In our case, we engaged Perficient to assist us and had a good experience with it.

Consulting Partner

Choose your consulting partner wisely, if you’re outsourcing some of the work.  We are a small shop, and essentially had no choice.  It’s frustrating to know that the consultants are learning the product on your dime.  In the end, as a DBA I strongly recommend you do your own heavy lifting, or you’re unlikely to like the final result, much less understand it.  I also recommend you approach the upgrade to 12.2 as a technical upgrade, and ensure your consulting partner’s strength lies in core database and applications DBA tasks.  If not, you might as well do it yourself.

Up next time: AD changes and your new techstack

R12.2, Oracle Java, Microsoft, Google and Security

The Playing Field

It’s no secret Google hates Microsoft (M$) hates Oracle hates Google hates Oracle hates Microsoft.  But EBS users are caught in the crossfire this time, and it’s not pretty.

M$ recently announced here they were going to start blocking Java launches out of IE.  You know, because they’re looking out for us.

Java started squalking about unsigned Jar files awhile back, but if your version was old enough, you wouldn’t see this message because it was too dumb to know it.  It also pesters you incessantly if you’re not running the latest version.  Always.

Google started stopping Java from running in Chrome which I noted here.  It was technically never certified with Oracle EBS, so I hold them least culpable at the moment.

Security

I appreciate that all of these players are protecting their own interests and implementing these changes in the name of security.  People complain about Java security?  Great, now Oracle will issue an update every week, and if you’re more than 2 updates behind it will nag you forever.  Launching EBS which doesn’t have signed JAR files?  Big scary window pops up warning the user.  Want to run old Java on Windows 7?  Microsoft says nope.

I’m in the middle of an R12.2 upgrade.  I have the luxury of taking the time to go get a certificate from Thawte and sign my jar files, and test it.  The truth is that it’s pretty easy.  But so is the process of getting a certificate.  Meaning, nothing against Thawte, but by signing my jar files all I’ve done is satisfy the JRE plug in.  I haven’t really enhanced my security, in my opinion.  Same thing with upgrading to the latest version of Java.

Meanwhile, here in the real world

I work for a manufacturer.  The floor workers who interact with Oracle EBS don’t know/care/understand what Java is, what a signed jar file is, what version of Java they’re running, etc.  They just want to get into Oracle EBS.  We’re not internet-exposed.  Sure there’s risk of someone exploiting a security hole, but it would probably have to come from the inside.  This is a known risk we’re probably more willing to take than the risk of having to deal with hours and hours of confusion every time M$ makes a unilateral decision about Java, or Oracle delivers yet another Java update.   We’ve been running EBS 11i on Java 1.6_12 for over 5 years now.  Never caused a problem.  It’s ancient, but it works.  Much like, I suspect, many companies, we have a lot of legacy Java running.  The effort required to push a newer version of Java to these machines is not insignificant, since they’re all locked down on the manufacturing floor.

Battle of the sexes

I’m sick of it.  Java consumers (and I’m not talking about coffee here) are stuck in the crossfire between M$ and Oracle.  Here’s what I see happening:

1. We regression test and deliver R12.2 with Java 1.7.0 update 67 in 6 months

2. Oracle issues 25 more versions of Java 7 thus every user gets the “Update (Recommended), Block or Later” dialogue box

3. Some users block, some update, some say later.  66.66% of these scenarios cause IT problems

4. Microsoft, not to be outdone, declares Java 1.7 update 67 obsoleted, and thus adds it to a block list

Managing the Mess

Sure, we can manage Win7 updates through policy, and the same for Java updates.  But are we actually improving security, in any way?  Or just creating busy work for ourselves, where the version which runs on our isolated network is largely immaterial?

 

 

Before flashback DB was cool

There was RMAN to flash your database back in time:

run
{
set until time “to_date(’07/04/13 04:00:00′,’mm/dd/yy hh24:mi:ss’)”;
allocate auxiliary channel d1 type disk;
allocate auxiliary channel d2 type disk;
restore database;
recover database;
alter database open resetlogs;
}

Agent Autostart process on Linux

Ever wonder how EM agents automagically start up during host restarts on Linux?

I recently had cause to dig into this, and figured I’d share the architecture for general knowledge sake.

For purposes of this entry, I’m making the rather obvious assumption that the agent has been successfully installed on the Linux host.

Relevant scripts:

These scripts should be landed by the Agent install.

/etc/rc.d/init.d/gcstartup

• call the $AGENT_HOME/install/unix/scripts/agentstup script with an argument of start or stop

$AGENT_HOME/install/unix/scripts/agentstup

• start or stop the agent (based on argument $1) via $AGENT_HOME/bin/emctl start|stop agent

$AGENT_HOME/install/unix/scripts/lockgcstartup (/etc/rc.d/init.d/lockgcstartup)

• touch /var/lock/subsys/gcstartup

• touch /var/lock/subsys/unlockgcstartup

$AGENT_HOME/install/unix/scripts/unlockgcstartup (/etc/rc.d/init.d/unlockgcstartup)

• rm -f /var/lock/subsys/gcstartup

• rm -rf /var/lock/subsys/unlockgcstartup

Relevant Symbolic Links by Run Level:

These sym links should be created during the Agent install.

Run Level 2

/etc/rc2.d/S98gcstartup → /etc/rc.d/init.d/gcstartup

/etc/rc2.d/K98gcstartup → /etc/rc.d/init.d/gcstartup

/etc/rc2.d/S99lockgcstartup -> /etc/rc.d/init.d/lockgcstartup

/etc/rc2.d/K99unlockgcstartup -> /etc/rc.d/init.d/unlockgcstartup

Run Level 3

/etc/rc3.d/S98gcstartup → /etc/rc.d/init.d/gcstartup

/etc/rc3.d/K98gcstartup → /etc/rc.d/init.d/gcstartup

/etc/rc3.d/S99lockgcstartup -> /etc/rc.d/init.d/lockgcstartup

/etc/rc3.d/K99unlockgcstartup -> /etc/rc.d/init.d/unlockgcstartup

Run Level 5

/etc/rc5.d/S98gcstartup → /etc/rc.d/init.d/gcstartup

/etc/rc5.d/K98gcstartup → /etc/rc.d/init.d/gcstartup

/etc/rc5.d/S99lockgcstartup -> /etc/rc.d/init.d/lockgcstartup

/etc/rc5.d/K99unlockgcstartup -> /etc/rc.d/init.d/unlockgcstartup

Walking through the bootup sequence for Runlevel 5:

• /etc/rc calls the kill scripts (for i in /etc/rc$runlevel.d/K* ; do ) with the command “stop”
  • /etc/rc5.d/K98gcstartup stop {awesome oracle syntax there!}
  • /etc/rc5.d/ K99unlockgcstartup stop {takes no arguments though one is passed}
• /etc/rc calls the startup scripts (for i in /etc/rc$runlevel.d/S* ; do ) with the command “start”
  • /etc/rc5.d/S98gcstartup start {startup, start!}
  • /etc/rc5.d/S99lockgcstartup start {takes no arguments though one is passed}

Incidentally:

• Note ID 374068.1 covers some of this information.
• I’m testing EM 12c and the agent install hooks are exactly the same (now gcstartup calls $AGENT_HOME/core/12.1.0.1.0/install/unix/scripts/agentstup)

Another Oddity

• For some reason, a Linux x86_64 agent of mine was failing to start on boot
• The console error was <AgentHome>/install/unix/scripts/agentstup: line 19: [: =: unary operator expected
• The “fix” from support was to use quotes to protect shell variables in their startup script
  • vi $AGENT_HOME/install/unix/scripts/agentstup
  • change both occurrences of $executingUser to “$executingUser”
  • change both occurrences of $installUser to “$installUser”
• This is detailed in 789363.1
• I’m floored that the fix, from support, is to manually edit an Oracle provided script, but at least I got a solution!

 

How many DBA’s are 12?

It’s been a long time since I posted, and I know this isn’t technical.  But I couldn’t resist.  I stopped reading comic books when I was about 12.

Awhile back, Oracle started rolling out their Database Machine or Exadata with Iron Man as their spokesman.  I’m not into Iron Man, but whatever.  I know the geek culture often permeates to the gaming/comic book world.  Then today I got the new Oracle Magazine.  We’re talking Hulk, Captain America, Iron Man and Thor.  Probably some I’ve missed or don’t know.

What I’m trying to figure out, is why the Hulk is hawking their proprietary hardware, and what his endorsement means to me.  Since a bunch of fictitious comic book heroes support an overpriced proprietary hardware platform, that’s supposed to influence me to buy it?  How many 12 year olds are purchasing authorities in your company?

If you can figure it out, let me know.  You can find them here