Kevin Kempf's Blog

November 10, 2011

Security Patching EM 11g : I don’t have all day!

Filed under: Enterprise Manager, Security — kkempf @ 3:35 pm

What’s in a Name?

I should begin by saying Enterprise manager is now Enterprise Manager Base Platform.  See ID 1361443.1 if you’re curious as to why they would take a good name and turn it into a terrible one.  If they thought they’d do this to avoid confusion, they failed.  I’ll continue to refer to it as EM 11g, except when necessary for greater clarity.

Oracle Critical Patch Update October 2011

Okay, so I didn’t get around to looking at the October security patch update until a few weeks back.  I still figure that’s better than those who don’t look at it at all.  I decided to start with a non-critical system, my EM 11g setup.  In the olden days, I seem to recall Enterprise Manager had its own category from the main page; maybe I’m mistaken.  Regardless, now you have to click through “Oracle Fusion Middleware 11g Release 1, versions,,” to find EM.  From there, scroll down for days until you get to section 3.3 and there’s Enterprise Manager.  Since I’m running 11g, I proceed to section 3.3.3 “Patch Availability for Oracle Enterprise Manager Base Platform“.

This consists of 5 distinct pieces, by Oracle classification

  • Database home (CPU, DB PSU, GI PSU, or Exadata BP12)
    • CPU = Critical Patch Update, incremental security patch
    • PSU = Patch Set Update, cumulative patch which includes recommended + security
    • GI PSU = Grid Infrastructure Patch Set Update, cumulative patch which includes recommended + security for rich people (Grid/Rac users)
    • Exadata BP12 = Oracle Exadata Database Recommended Patch, cumulative patch which appears to include recommended + security for super rich people (Exadata/Rac users)
  • Enterprise manager Base Platform – OMS home: (OMS)
  • Enterprise manager Base Platform – OMS Fusion Middleware home (Weblogic Home)
  • Enterprise manager Base Platform – OMS Fusion Middleware Oracle HTTP Server home (OMS, I think)
  • Enterprise manager Base Platform – Agent home (Agent Home)

Getting on my soapbox

Dear Oracle,

I know you have lots of products, you buy new companies every week, and you are the 800 pound gorilla of the business software world.  Could you please simplify patching?  It’s gotten worse, not better, in the past couple of years.  Why isn’t there one place I can go within each application stack/entity, to see what patches have been applied?  Why are there 5 different methods of security patching (SQL Plus, opatch, adpatch, shell scripts and the wacky Weblogic GUI or CLI) for Oracle apps and Oracle EM11g (sorry, Enterprise Manager Base Platform)?  Also, I don’t have spare weeks to apply quarterly patches to all my systems.  Believe it or not, there’s other things I’m responsible for.  Thanks.

PS: If you want to see a good patch management architecture, check out the RedHat Network.  Systems, once configured, check in every couple of hours and see if there’s something to apply.  If there is, the patch can be released from the website, and either downloaded or applied.  It quite literally runs circles around Oracle’s Configuration Manager (OCM).

I can’t get that day back

There’s basically 4 environments in an EM 11g home: The RDBMS, the OMS, the Agent, and the WLS home.

    • Thankfully, patching the database hasn’t changed in years (Linux, non-RAC)
      1. Pull the database PSU to your desktop (in my case, 12827726 PSU for
      2. sftp/scp the file to the RDBMS server/staging area
      3. Shut down the database and listener
      4. opatch apply
      5. sqlplus / as sysdba and run catbundle.sql psu apply
      6. Start the database and listener
  • OMS (Enterprise Manager Base Platform – OMS home for those who prefer maximum verbosity)
      1. opatch apply (12833678)
      2. I think I hit a weird java exception applying this; you may need to apply patch 12620174 first.  I don’t mean to sound vague; I simply don’t remember.
  • Agent Home
      • opatch apply (9345921)
  • WLS (Enterprise Manager Base Platform – OMS Fusion Middleware home for those who prefer maximum verbosity or want to impress their friends)
    • I gotta tell you, this is where it got wacky: Oracle Smart Update (aka  I never patched a WLS home before.  Shame on me.  Apparently, there are two choices: run their GUI or run their CLI
    • I chose the GUI
      • You might reference ID 1072763.1 regarding how to patch WLS… I thought I had a better example but that one will suffice.  It also covers command line patching.
      • cd $ORACLE_OMS_HOME/../utils/bsu
      • Land the following patches to my desktop.  sftp/scp them to the server under $ORACLE_OMS_HOME/../utils/bsu/cache_dir
        • 12875001
        • 12875006
        • 12874981
        • 10625613
        • 10625676
      • unzip the 5 patches above.  remove the .zip file, and the README file included with them all.
      • ./
        • Using the GUI, the patches appear at the bottom.
        • Ensure you have the right Middleware home selected on the left (in my case, WLS runs on this server as the Discoverer server in a separate Oracle Home)
        • Hit the arrow or some such nonsense to make them go to the top
        • Here’s some screenshots to show you the general flow

Obviously, how you'd launch a patching utility

BSU Main Screen

Select from the list of patches in your cache directory, and hit the green up arrow

After clicking the green arrow, the patch is validated against... something

End state. Everything is installed. I think.

To Summarize

Just to patch EM 11g, the discrete steps involved for me were

  • Read the CPU to determine applicability
  • Determine which patches need to be applied
  • Pull the patches from the world’s slowest support site
  • Stage the patches to the EM server
  • Apply the patches to the database using opatch and sqlplus
  • Apply the patches to the OMS using opatch
  • Apply the patches to the Agent Home using opatch
  • Apply the patches to WLS using the wacky GUI

While this is somewhat of a detailed overview of how to apply the CPU to EM11g last month, I wanted to make two points.

  • First, there are too many disparate ways of patching Oracle, in my opinion.  They range from the simplicity of a GUI for WebLogic patching to literally issuing unzip and cp commands on a Linux host to apply a patch to the 11i techstack home (don’t believe me?  check out patch 10410398).
    • As a result of the above, patching (especially security patching) takes too long
    • As a result of it taking too long, it’s very easy to see how one would choose to ignore security patching
  • I wanted to show the “new” WLS patching method on the blog, as I hadn’t seen it before.  It’s surprisingly simple, yet it felt like Oracle took the ball to the opponents 4 yard line and fumbled.  Why not just automatically pull patches to the cache directory based on a checkin like RedHat (RHN)?  Apply them and report back in a web GUI somewhere?

November 1, 2011

RDBMS is out

Filed under: 11g, Support — kkempf @ 9:47 am


So I’m reading the components to update for CPU 1011 which came out a few weeks ago, and I noticed that if I upgrade my RDBMS to I don’t have to apply the RDBMS CPU patch.  Hmm, didn’t even know was out, but there it is, 23-Sep-2011.


Then I remembered Oracle’s awesome new plan about one-off patches.  They’re not patches anymore.  In fact, I don’t even understand why they call it patch set for oracle database server.  It’s 5.1gb, a complete base install:

Ladies and Gentlemen, the world's largest patch, brought to you by the world's slowest support site!

HTTP transfers

Perhaps the issue is our corporate internet, but in my experience, http is an awful way to download files on a massively shared pipe.  I think it took me 4 weeks to download R12.  Basically, without error correction or the ability to resume, the http downloads just “die” silently in the browser. Oracle, if you’re going to make me download 5.1gb every time you come out with a one-off patch, how about an sftp site, java downloader, something more professional than http?  Do we have to go back to the days when I contact support and have them send me a bunch of physical media?

Blog at