Kevin Kempf's Blog

November 18, 2016

EBS R12.2 Security

Filed under: Oracle, R12.2, Security — kkempf @ 10:27 am


That light at the end of the tunnel might be a freight train

If you’re running 12.2 and considering 12.2.6 any time in the near future, you need to be aware of significant changes coming to the security requirements of EBS.  At the East Coast Oracle conference in Raleigh a few weeks back, I sat in a session by Elke Phelps (Oracle) entitled “Ready or Not: Applying Secure  Configuration to Oracle  E – Business Suite“.   Oracle EBS is getting serious, if not belligerent, about security.

Bottom line: If you don’t comply with their security recommendations, your users will not be able to log into EBS.  That’s right.  This is probably the most concerning thing about the changes: they’re not suggested.  Don’t believe me?  Check out Doc 2174164.1 section 3.1:

The Secure Configuration Console automates the security configuration process by consolidating the security configuration process onto one user interface and creating a single checkpoint entry into the system. It checks your system against 16 high-priority security configuration guidelines and makes recommendations to the system administrator to either fix or suppress the failure. Until the system administrator acknowledges these checks, users will be denied entry into the system.

Do I have you attention now?

Here’s the thing: most of the things it appears to be concerned with are either simple fixes or common sense.  While I haven’t seen it myself, I’ve been assured there is a page somewhere (in OAM?) where the apps DBA can go in and check boxes saying “I understand I’ve broken security rules, now please let me use my ERP”. Think: DEV environments.

One problem with all of this is it’s implemented in Oracle’s typical unwieldy and disparate manner.  Start by grinding your way through the 468 page Oracle Ebusiness Suite R12.2 security guide.  Let me just say it’s not a real page turner and I’ve had less reading in graduate level courses.


So we move on to 2069190.1 where you can find a nifty set of scripts called  This is run against your database, and comes back with somewhat actionable results.  I will admit, it found some things I wasn’t aware of, but it’s not perfect.  By the numbers:

  • Check: Security Profiles: Configuration ERRORS
    • This checks POVs to see if you’re doing anything wrong.  In my case, it didn’t like a site level POV called Framework Validation Level (value was set to none).  The problem with this assessment, is that this isn’t a POV I can find in any way.  As in, I can’t query it up in forms, and when I try to use to change it, it errors.  First check and I have to open an SR.  Great start!
  • Check: Security Profiles: Configuration WARNINGS
    • This appears to check and see who has the ability to run diagnostics in EBS, as well as who can attach files and personalize self service bits.  The problem is, it doesn’t bother to check and see if the EBS user with these privs is end dated.  So I wound up with a list of end-dated consultants from our R12.2 upgrade, none of which can log in, and a list of IT super users.
  • Check: Security Profiles: Configuration MISSING
    • No idea, I “passed” this one
  • Check: Application Users with Default Passwords
    • I’ll give Oracle credit on this one, it found some interesting things here.  I had to change my guest password, and disable autoinstall.  Hint: try to find that user.. when you get tired of that game, run FND_USER_PKG.DISABLEUSER(‘AUTOINSTALL’);
    • It also found some schemas which had the default password.  As in, all the new schemas presumably introduced by the 12.2 upgrade, so kudos to Oracle there.  Mine were ddr, dna, dpp, gmo, ibw, inl, ipm, jmp, mth, qpr and rrs.  Obviously I hadn’t run FNDCPASS against “ALL” since the upgrade, and so these were just hanging around.
  • Check: DB Users With Default Passwords
    • Again, a handful out there, but if the script would exclude “EXPIRED & LOCKED” that’d be great.  Most significant was applsyspub.  I swear it used to be you were told not to change this, but it’s easy enough to fix with FNDCPASS
  • Check: For excessive privs in APPLSYSPUB
    • Passed this one
  • Check: Oracle Applications User Passwords Migrated to Non-Reversible Hash Password
    • Yeah, I hadn’t done this.  It’s an easy fix see 457166.1
  • Check: Server Security Status
    • Passed this one
  • Check: SSL Status
    • You need to be using TLS 1.2.  See 2143101.1.  If you’re not familiar with the process, welcome to a lot of work.  And now you get to keep up with an external signing authority!  Yay work!
  • Check: Credit Card Encryption Status
    • Beats me, I failed this, but we don’t store this in the database, so perhaps that would be helpful to note.
  • Check Status of 12.2 Security Features
    • I had a recommendations here, because I allowed unrestricted JSP access,  unrestricted redirects and a setting it didn’t like for Cookie Domain.  This is probably legitimate, but it’s going to cause havoc with our Oracle APEX implementation and will likely take time to fix.  There’s a script called in $FND_TOP/patch/115/bin which may help you (“Oracle E-Business Suite Release 12.2.6 delivers a configuration script which can assist in configuring the products in your Allowed JSP lists. Products will be turned off in the Allowed JSP family configuration files based on whether recent transactions are detected for the product. Customers are strongly recommended to configure the Allowed JSPs using this script.”).  Check out the Oracle EBS Security Guide
  • Check: Users with Access to Sensitive Pages
    • Disabled by default.  I guess I pass?

Where this falls apart for me

They’re going to hold your EBS hostage until you address security gaps?  Unacceptable.  While it’s true, you’re not going to wake up one morning to users unable to log into production, it’s still a rather arrogant shift.  I happen to know my company paid a fortune for their software, but now I’m not so confident we own it.  This feels like some back room legal department at Oracle decided to get a record that “so and so” at XYZ company deliberately agreed to ignore our security recommendations, and we have it on the record because they checked boxes acknowledging they assume the risk.  Except the reciprocal is not true.  If I get hacked despite having complied with Oracle recommendations, I’m pretty sure Oracle isn’t going to write me a check to help me fix it.

Oracle is making assumptions about my environment and punishing me if I don’t comply.  What if I have no internet exposure?  What if this is a DEV environment where I don’t care or can’t afford to keep up with TLS 1.2?

Instead, a warning message (old school nag screen) at login to any user stating something like “Your corporate EBS may be at risk to security vulnerabilities, please contact your system administrator” would have been equally effective.  I’m not going to let that screen be up there for the world to see, and I will get phone calls.  But if something happens beyond my control, my users can still get into EBS.

April 28, 2015

Security Patch Releases with EBS: Mission Impossible

Filed under: R12.2, Security — kkempf @ 12:33 pm

Regression Test Time

We’re about to enter our first R12.2 regression test.  High on my list is to get to Delta 6 on the AD/TXK side, and since the security patches just came out I figured I’d get up to date there.  For pretty good reason, we’re still running base, and it took me some time to get the ETCC (patch 17537119) happy that I’ve got all the required patches.

Per 1967243.1 I figured I’d start with the core database.  Now we have combo PSU’s for database, OJVM, and GI, in addition to SPU’s and stand-alone PSU’s for each component.  It’s gotten complicated since I looked last!  The patches are all opatch installed; ideally I’d get the OJVM, SPU and PSU applied to the database home.

OJVM (20406239)

This is a documented problem on MOS.  Apparently, it’s not really cumulative, because I have to be on Oct 2014 or better security set to apply this.  Next.

The following make actions have failed :

Re-link fails on target “jox_refresh_knlopt ioracle”.

Do you want to proceed? [y|n]
User Responded with: N


PSU (20299013)

This one was even worse!

There are no patches that can be applied now.

Following patches have conflicts. Please contact Oracle Support and get the merged patch of the patches :
20488666, 20299013, 19791273, 19730032, 18260550, 17420796

Following patches are not required, as they are subset of the patches in Oracle Home or subset of the patches in the given list :
17811789, 19393542, 18828868, 18614015, 17892268, 17600719, 17468141, 16992075, 16929165

Following patches will be rolled back from Oracle Home on application of the patches in the given list :
20488666, 17811789, 19791273, 19730032, 19393542, 18260550, 18828868, 17420796, 18614015, 17892268, 17600719, 17468141, 16992075, 16929165

Conflicts/Supersets for each patch are:

Patch : 20299013

Bug Conflict with 20488666
Conflicting bugs are:
17912217 ETCC R12.2 requirement per 1594274.1

Bug Superset of 17811789
Super set bugs are:

Conflict with 19791273 ETCC R12.2 requirement per 1594274.1
Conflict details:

Bug Conflict with 19730032 ETCC R12.2 requirement per 1594274.1
Conflicting bugs are:
17174582,  18282562,  18244962,  17614134,  18674024,  17050888,  17478145,  18331850,  18964939,  17883081,  18436307

Bug Superset of 19393542
Super set bugs are:

Conflict with 18260550 ETCC R12.2 requirement per 1594274.1
Conflict details:

Bug Superset of 18828868
Super set bugs are:

Conflict with 17420796 ETCC R12.2 requirement per 1594274.1
Conflict details:

Bug Superset of 18614015
Super set bugs are:

Bug Superset of 17892268
Super set bugs are:

Bug Superset of 17600719
Super set bugs are:

Bug Superset of 17468141
Super set bugs are:

Bug Superset of 16992075
Super set bugs are:

Bug Superset of 16929165
Super set bugs are:

Security Patch Update (20299015)


Conflicts/Supersets for each patch are:

Patch : 18203837

Conflict with 18260550 ETCC R12.2 requirement per 1594274.1
Conflict details:

Patch : 19972566

Conflict with 18614015 ETCC R12.2 requirement per 1594274.1
Conflict details:

Patch : 20506715

Conflict with 19730032 ETCC R12.2 requirement per 1594274.1
Conflict details:

Patch : 20631274

Bug Superset of 17600719 ETCC R12.2 requirement per 1594274.1
Super set bugs are:

Following patches have conflicts: [   18260550   18203837   18614015   19972566   19730032   20506715 ]
Refer to My Oracle Support Note 1299688.1 for instructions on resolving patch conflicts.

Security or Compatibility: Why are these two at odds?

I’m trying to do the right thing, but Oracle is making it so hard to do that I’ve lost interest.  I don’t have a month to figure out what merged patches I need, what patch can go and what can stay. Why can’t there be a single note for EBS by version, with all patches for all components?  A database patch set which already deconflicted all the oddball ETCC requirements and delivered something which you could actually install out of the gate would make life much easier.


November 10, 2011

Security Patching EM 11g : I don’t have all day!

Filed under: Enterprise Manager, Security — kkempf @ 3:35 pm

What’s in a Name?

I should begin by saying Enterprise manager is now Enterprise Manager Base Platform.  See ID 1361443.1 if you’re curious as to why they would take a good name and turn it into a terrible one.  If they thought they’d do this to avoid confusion, they failed.  I’ll continue to refer to it as EM 11g, except when necessary for greater clarity.

Oracle Critical Patch Update October 2011

Okay, so I didn’t get around to looking at the October security patch update until a few weeks back.  I still figure that’s better than those who don’t look at it at all.  I decided to start with a non-critical system, my EM 11g setup.  In the olden days, I seem to recall Enterprise Manager had its own category from the main page; maybe I’m mistaken.  Regardless, now you have to click through “Oracle Fusion Middleware 11g Release 1, versions,,” to find EM.  From there, scroll down for days until you get to section 3.3 and there’s Enterprise Manager.  Since I’m running 11g, I proceed to section 3.3.3 “Patch Availability for Oracle Enterprise Manager Base Platform“.

This consists of 5 distinct pieces, by Oracle classification

  • Database home (CPU, DB PSU, GI PSU, or Exadata BP12)
    • CPU = Critical Patch Update, incremental security patch
    • PSU = Patch Set Update, cumulative patch which includes recommended + security
    • GI PSU = Grid Infrastructure Patch Set Update, cumulative patch which includes recommended + security for rich people (Grid/Rac users)
    • Exadata BP12 = Oracle Exadata Database Recommended Patch, cumulative patch which appears to include recommended + security for super rich people (Exadata/Rac users)
  • Enterprise manager Base Platform – OMS home: (OMS)
  • Enterprise manager Base Platform – OMS Fusion Middleware home (Weblogic Home)
  • Enterprise manager Base Platform – OMS Fusion Middleware Oracle HTTP Server home (OMS, I think)
  • Enterprise manager Base Platform – Agent home (Agent Home)

Getting on my soapbox

Dear Oracle,

I know you have lots of products, you buy new companies every week, and you are the 800 pound gorilla of the business software world.  Could you please simplify patching?  It’s gotten worse, not better, in the past couple of years.  Why isn’t there one place I can go within each application stack/entity, to see what patches have been applied?  Why are there 5 different methods of security patching (SQL Plus, opatch, adpatch, shell scripts and the wacky Weblogic GUI or CLI) for Oracle apps and Oracle EM11g (sorry, Enterprise Manager Base Platform)?  Also, I don’t have spare weeks to apply quarterly patches to all my systems.  Believe it or not, there’s other things I’m responsible for.  Thanks.

PS: If you want to see a good patch management architecture, check out the RedHat Network.  Systems, once configured, check in every couple of hours and see if there’s something to apply.  If there is, the patch can be released from the website, and either downloaded or applied.  It quite literally runs circles around Oracle’s Configuration Manager (OCM).

I can’t get that day back

There’s basically 4 environments in an EM 11g home: The RDBMS, the OMS, the Agent, and the WLS home.

    • Thankfully, patching the database hasn’t changed in years (Linux, non-RAC)
      1. Pull the database PSU to your desktop (in my case, 12827726 PSU for
      2. sftp/scp the file to the RDBMS server/staging area
      3. Shut down the database and listener
      4. opatch apply
      5. sqlplus / as sysdba and run catbundle.sql psu apply
      6. Start the database and listener
  • OMS (Enterprise Manager Base Platform – OMS home for those who prefer maximum verbosity)
      1. opatch apply (12833678)
      2. I think I hit a weird java exception applying this; you may need to apply patch 12620174 first.  I don’t mean to sound vague; I simply don’t remember.
  • Agent Home
      • opatch apply (9345921)
  • WLS (Enterprise Manager Base Platform – OMS Fusion Middleware home for those who prefer maximum verbosity or want to impress their friends)
    • I gotta tell you, this is where it got wacky: Oracle Smart Update (aka  I never patched a WLS home before.  Shame on me.  Apparently, there are two choices: run their GUI or run their CLI
    • I chose the GUI
      • You might reference ID 1072763.1 regarding how to patch WLS… I thought I had a better example but that one will suffice.  It also covers command line patching.
      • cd $ORACLE_OMS_HOME/../utils/bsu
      • Land the following patches to my desktop.  sftp/scp them to the server under $ORACLE_OMS_HOME/../utils/bsu/cache_dir
        • 12875001
        • 12875006
        • 12874981
        • 10625613
        • 10625676
      • unzip the 5 patches above.  remove the .zip file, and the README file included with them all.
      • ./
        • Using the GUI, the patches appear at the bottom.
        • Ensure you have the right Middleware home selected on the left (in my case, WLS runs on this server as the Discoverer server in a separate Oracle Home)
        • Hit the arrow or some such nonsense to make them go to the top
        • Here’s some screenshots to show you the general flow

Obviously, how you'd launch a patching utility

BSU Main Screen

Select from the list of patches in your cache directory, and hit the green up arrow

After clicking the green arrow, the patch is validated against... something

End state. Everything is installed. I think.

To Summarize

Just to patch EM 11g, the discrete steps involved for me were

  • Read the CPU to determine applicability
  • Determine which patches need to be applied
  • Pull the patches from the world’s slowest support site
  • Stage the patches to the EM server
  • Apply the patches to the database using opatch and sqlplus
  • Apply the patches to the OMS using opatch
  • Apply the patches to the Agent Home using opatch
  • Apply the patches to WLS using the wacky GUI

While this is somewhat of a detailed overview of how to apply the CPU to EM11g last month, I wanted to make two points.

  • First, there are too many disparate ways of patching Oracle, in my opinion.  They range from the simplicity of a GUI for WebLogic patching to literally issuing unzip and cp commands on a Linux host to apply a patch to the 11i techstack home (don’t believe me?  check out patch 10410398).
    • As a result of the above, patching (especially security patching) takes too long
    • As a result of it taking too long, it’s very easy to see how one would choose to ignore security patching
  • I wanted to show the “new” WLS patching method on the blog, as I hadn’t seen it before.  It’s surprisingly simple, yet it felt like Oracle took the ball to the opponents 4 yard line and fumbled.  Why not just automatically pull patches to the cache directory based on a checkin like RedHat (RHN)?  Apply them and report back in a web GUI somewhere?

January 20, 2011

January, 2011 Critical Patch Update is out

Filed under: Security — kkempf @ 11:30 am


The January 2011 CPU came out.  For those of us on 11i who diligently apply these things, it means burning a day to dissect what is available, needed, and worthwhile.

First, the good news

On the Apps side, the 11i/ work is finally down to 1 (one!) patch to apply: 10258309 Ironic, that it takes the product going into extended support, before getting down to 1 patch.  Would have loved to have seen this for the past 5 years, but hey, it’s a step in the right direction.

Now comes the bad news… you do the math

Ah, the RDBMS.  After my most recent debacle with overlays, recommended patches, performance patches, and advanced compression patches, I’ve done the math.  For, I can either

a) Apply the CPU only (10249534)

b) Blindly trust Oracle to decide what’s best for me and apply the PSU (10248531), 5 overlay patches (see Note 1147107.1), and then double check and see what patches were knocked out as a result.

Hmm… let me think about it.  11i is in extended support.  My database is running fine.  I’m not looking for anything but a stable ship right now, while we try to figure out how or why to upgrade to R12.  I’ll choose door #1, thanks for playing!

A Rant About Patching EM 11g

So I pulled all the patches I could for EM 11g; this included the obvious (RDBMS patch), the EM Agent (OPatch), the OMS Home (OPatch).  No problem, these applied without incident.  OPatch is a pretty well oiled tool at this point.

Next: a lot of conFusion about my WLS 10.3.2 home.  According to the CPU note (1263333.1), I’m supposed to apply patch MOS: 9893328, or MOS: 9893736.  Not sure what the or is all about, but I clicked through.  The patch is labeled for WLS release 10.3.3.    In fairness, the master note says “The WebLogic plug-ins include all cumulative bug fixes and thus include fixes for all previously released advisories. These plug-ins are compatible with all versions of WebLogic Server”.   In that case, why list versions on the master note? It’s just confusing!

No problem, I’ll pull the patch and see what’s up.  Here’s the entire readme for 9893328:

The Web Server plug-ins built at change number CL1338089

Supported configurations
The WebLogic webserver plugins are common to all versions of WebLogic servers. 
For specific supported configurations, refer to the Weblogic server documentation.

Upgrade instructions.
- Save a back-up copy of your existing plug-in module.
- replace the plug-in module with the one found in this zip-file
- restart your web server.

WTF does that mean?  Replace what plug-in module?  I’m completely unfamiliar with patching Weblogic Server, a little help maybe?  Let me check the other side of this “or” patch equation.  Here’s the readme for 9893736:

 Oracle WebLogic Server Web Server Plugins 1.1 README

Zip file contents:

This zip distribution contains Oracle WebLogic Server Web Server Plugins 1.1 zip files for the platforms and web servers supported. For more information, please extract the appropriate zip file which has the README.txt file explaining the installation and configuration details.

For complete documentation, refer to the documentation steps listed below:

- Go to the Oracle Fusion Middleware 11g Release 1 (11.1.1) Documentation below:

- From here, navigate to the Oracle Fusion Middleware 11g Release 1 Patchset 2 ( documentation for "WebLogic Server" and then to "Using Web Server Plug-ins".

Seriously?  There’s no coherent, consistent or detailed message here, Oracle!  First I’m looking to patch WLS 10.3.2, and your patch says it’s for 10.3.3.  Then it says I can install one or the other (fielder’s choice?).  Then I pull the readme’s for a product I’m barely familiar with and they don’t tell me squat.  In fact, they reference FMW 11.1.1 and FMW and I have no idea if your crazy versioning scheme means that’s WLS 10.3.2.

I can’t be certain if this is just growing pains (i.e., Oracle hasn’t ironed out a simple CPU application method to FMW/WLS), integration pains (Oracle hasn’t incorporated WLS into the “fold”) or what, but it can’t get worse, nor can it remain this bad.  I’ll skip applying this CPU to WLS, hoping it looks better next quarter.  Note 1263374.1 promises PSU’s “soon”: “Starting with Oracle WebLogic Server 10.3.4, Oracle will release WebLogic Server Patch Set Updates (PSUs). Details on the WebLogic Server PSU program will soon be available.”

October 23, 2009

New Patch Set Updates (PSU’s) Available

Filed under: 10g, 11g, Oracle, Security — kkempf @ 9:52 am

Tuesday when Oracle released its (delayed due to Open World) Fall 2009 Critical Patch Update, it also released an update to Patch Set Updates (PSU’s).  This information is obtained from Document ID 854428.1, dated 20 Oct 2009.

As mentioned earlier, I’m a big fan of the PSU concept and am pleased to see that it has been expanded with this Critical Patch Update.  The arrival of RDBMS 11.1 PSU’s, clusterware, as well as EM is great news for me, as it makes life simpler.  I figure I’m already going to have to apply the CPU, might as well get the benefits of the PSU and a version change (internally) to prove it.

I’ve already installed on 4 production and 8 development/test/training 10g databases without incident, and plan to get moving on testing this release.  I’d really like it if Oracle would actually show a PSU’d database as, as it would be a visual queue that the PSU had been applied to the database, but at least on the internal company Wiki I can just tag the RDBMS version as such and not have to separately track CPU’s anymore.

After a little digging, I found that Oracle has mad some “oddball” platforms “By request only”, while most mainstream platforms (Linux, Solaris, HP-UX, AIX) are available as PSU’s.  Again, curiously, Windows has been classified “Not applicable” as an OS since “Windows fixes are released in periodic bundle patches which contain all the CPU and PSU content”.  Whatever that means.

RDBMS PSU’s available (patch number in parenthesis):

  • (8833297)
  • for CRS (Cluster Services) (8287931)
  • (8833280)
  • for CRS (Cluster Services) (8705958)

Oracle Enterprise Manager Grid Control (patch number in parenthesis):

  • for OMS (8864918)

edit:  I ran the PSU against my Enterprise Manager OMS ( and ouch it broke it.  Badly.  The application was 2 parts; opatch apply and an RDBMS script.  The opatch piece was fine.  The RDBMS script ended with many errors, and EM wouldn’t start.  I rolled back the RDBMS script per the readme… and EM wouldn’t start.  For the record, I am running RDBMS 11.2 (uncertified for EM) and not the 10.2 version..

edit 2: I ran the PSU against my regression test ERP database ( and it applied (seemingly) without incident.

edit 3: At this point, I’ve run this PSU successfully against the following environments:

  • Enterprise Manager OMS (  8874205
    • I installed without the RDBMS script
    • You must install the Linux x86 version even if you’re running x86-64, as the default 64-bit install of EM runs 32-bit binaries.  Go figure.
    • Not sure what the problem with this patch was the first time I tried it
  • Discoverer 10g (AS 8874205
  • RDBMS (15 instances) 8833280
  • RDBMS (1 instance) 8833297

Blog at