Kevin Kempf's Blog

January 30, 2015

Determine Weblogic Server Version in EBS R12.2

Filed under: R12.2, Weblogic — kkempf @ 10:08 am

A diversion from my topic in progress

Stumbled across this via an analyst on an SR I was working, thought it might be useful for others.

$ cd $FMW_HOME/wlserver_10.3/server/lib

$ java -cp weblogic.jar weblogic.version

WebLogic Server 10.3.6.0 Tue Nov 15 08:52:36 PST 2011 1441050

Use ‘weblogic.version -verbose’ to get subsystem information

Use ‘weblogic.utils.Versions’ to get version information for all modules

Advertisements

January 13, 2015

R12 Take Aways and Warnings, Part 2 (AD Changes and your new Techstack)

Filed under: R12.2, Techstack — kkempf @ 9:53 am

warning

ADOP and your new Techstack

As promised, I’m going to continue the discussion about about my R12.2 upgrade and the lessons learned as a part of it by digging into the techstack.  There’s a lot of ground to cover here!

General Upgrade Flow and Layout

For a general understanding, I thought I’d start with a basic explanation of the upgrade process.  Let’s assume a simple installation, where you have a database tier and an applications tier.  As usual, I’m tossing out my disclaimer that this list is NOT complete.  It’s a high level look at the basic tasks.  Follow the official doc!

  • 11i Pre-Patching/Prep Work

You will need to first prep your old environment (in my case 11i, though I suspect this will hold true if you’re on 12.0/12.1) for the upgrade.  This means under 11i, you need to apply database patches, run checkers, clean up your customizations, etc.  Since R12.2 allows for a 64-bit front end, we chose to create a new front end host and leave the database where it was (since it ran fine).  Compared to 11i, these new certifications allow you all kinds of nice things, such as running Linux 6, and allocating huge sums of memory (with a 64-bit OS) to the host without running a PAE (physical address extension for memory) kernel.  This in turn allows you to easily accommodate many, large JVMs on the front end.  Regardless, you need to be at the appropriate baseline before you begin the upgrade.

  • 11i Database Patching

Pretty self explanatory; there’s a bunch of mandatory patches to the RDBMS tier.

  • 12.2 Rapid Install

You first lay down the file system(s) on whatever will be your new R12 application server.  Note: You can’t do this ahead of time in production.  At this point you’re touching the database and affecting 11i

  • Application DBA Tasks

In the middle of your upgrade, you need to perform a bunch of AD tasks.  Such as getting on the right version of the techstack, extending tablespaces, fixing customizations, etc.

  • Adpatch

This is the last time you’ll run adpatch, and it’s a monster: the largest, longest bit of work involved.  Lots of worker fails with errors which you have to go research.

  • Another 12.2 Rapid Install pass

Short pass here, against your new R12 filesystem.  Congratulations, after this you’re on 12.2 (not a production release, but you can login after this).

  • Enabling 12.2 Online Patching

This is why you decided to install 12.2 right?  This is a long, database intensive task.

  • Applying Techstack Patches

You can mitigate this by using the latest install CD for 12.2.X, but you’ll be installing patches against Weblogic, forms and reports, http server, FMW etc

  • Applying 12.2.3 (or 12.2.4, presumably!)

This is it’s own patch, applied with adop

  • Post-install Tasks

The devil is in these details, which are specific to your installation.  An example is configuring Vertex (tax) for your financials

JVM Sizing

If you’re serious about this upgrade, please Google up Oclay Sariouglu and Dimas Chbane’s (both from Oracle) presentation from East Coast Oracle Users Group conference titled “E-Business Suite 12.2: Architecture and New Features”.  Specifically, a PowerPoint called Chbane 12.2 Architecture.pdf is worth reviewing; I’m sorry I don’t know with certainty that you can get it easily if you didn’t attend the conference, you can probably get it through official channels.  Here’s the bottom line: Oracle has a scaling recommendation (and a revised scaling recommendation!) for adding oacore and forms JVM’s based on the number of users.  The default installation for R12.2 gives you one (1) JVM for forms (forms_server1) and one (1) JVM for oacore (oacore_server1), and each take maybe 256MB or 512MB (I forget which).  This is ludicrously small, for any real deployment, and I’m glad I went to this conference and attended this session to at least get the references on how to size them and how to add them.  These JVMs live inside of Weblogic, and you need to size them correctly.  Because if you go live with the defaults and more than a few dozen users, your JVMs will crash.  Check out 1905593.1.  We went live with 4 of each (forms, oacore) at 2GB per for an installation footprint of about 1000 users (about a third of whom are in professional forms) and it’s served us well.  The truth is our forms footprint is probably too big, but we haven’t cut it back yet.

Adding JVMs is a somewhat manual, command line process, or at least was when we did it, and for being rather essential to a go-live, it stands out as an after-thought to the whole process.  This task also includes changes to the context files, which also has hooks to your start/stop scripts (adstrtal.sh/adstpall.sh).  Consequently, this has the side effect of making an application server startup (or a maintenance adop cutover) even slower (if it will go down cleanly at all, but I digress).  Another bit of fallout is that when you clone, you may need to cut back these JVMs, depending on the memory resources you have available on your non-production host(s).  If you find yourself in this situation, I’ll offer this tidbit: this can be accomplished by changing the s_oacorename and s_formsname parms in the context file to eliminate, for example, oacore_server2, oacore_server3, etc. until you get it sized appropriately.

AD/TXK Patching

As you probably know, officially, AD is the product abbreviation for Applications DBA, and TXK corresponds to Teckstack.  The two are tightly interwoven, to the point that in R12.2 (perhaps in earlier editions, unsure) they must always be on the exact same version, and patched together.  It raises an interesting aside, in that to me, as the DBA, as they’re now indistinguishable why bother to make any distinction and not just roll them up under one product… but I digress.

In the middle of your upgrade, you apply AD.C.  In my original plan, I think I had the choice of applying AD/TXK delta 3 or delta 4.  Since I had nothing to lose, I went with delta 4, and during the course of our upgrade testing delta 5 was released.  Delta 5 fixed so many critical bugs in the AD stack that I had to beg the project manager to let me put it in late in the testing phase.  For example, we use mobile supply chain (MSCA) and under delta 4 the telnet ports would change during an adop cutover phase (imagine if you will telling hundreds of manufacturing operators to change their telnet ports on PC’s and hand-held devices after every maintenance weekend).  All of this gets rather confusing, because officially I’m on AD/TXK delta 5, but that’s not all.  Now there’s essential updates on top of AD/TXK which get released, seemingly at random.  For example, patches 19445058/19581770 (now superseded by 20034256: additional fixes that were not in the first bundle/November bundle patches for AD Delta 5!) are called “BUNDLE FIXES FOR R12.TXK.C.DELTA.5” and “BUNDLE FIXES FOR R12.AD.C.DELTA.5”.  This is a mess.  To me, it would be much simpler to consolidate the AD/TXK patches and put some meaning to the versions.  Why can’t AD/TXK delta 5 with bundle fix 1 be called, simply, AD.C.5.1?  No TXK, I don’t care about the distinction since I always have to apply them together, why are they distinct products to the customer?  It’s like selling a cup and the water separately at a restaurant.

Regardless, my point about the AD/TXK C, Delta 3, 4 and 5, additional fixes, essential bundles, mandatory mess ups, fantastic fixes, or whatever else they call it: they change a lot.  Research what the latest, greatest version is and install it.  Now welcome to another new job task: you need to keep up with this in 12.2 (subscribe to Steven Chan’s blog, at least you’ll get an email when these are released, among other things).

I’ve rambled enough for today, I have plenty more to cover.  Up next time: more AD changes and your new techstack!

January 9, 2015

Online Patching

Filed under: Online Patching, R12.2 — kkempf @ 11:11 am

ADOP my friend (today)

So I took the plunge and did the main thing we chose R12.2 for, as opposed to R12.1.  I applied a few patches prior to a downtime/maintenance cycle this weekend in my Production system.  Kind of scary, but everything went OK.  Here’s what’s interesting:

online patching load

Guess when I ran my first adop phase=apply (…) ?   <spoiler: 11:25am>

It was a curiously intense bit of work for the database to apply this patch (19523116 for the record) in online mode.  Oracle never advertised that online patching did its work with 0 impact, but it was a bit of a surprise to see the database system CPU maxed out as a result of it.

I should add that I did this yesterday, and didn’t get any calls or complaints so this in no way affected Production that I ‘m aware of.  My point is only that you should be aware that online patching has the potential to impact your server in significant ways.  Maybe don’t run it during month end close.

January 7, 2015

R12 Take Aways and Warnings Part 1 (Overview)

Filed under: Oracle, R12.2 — kkempf @ 11:30 am

warning

What could possibly go wrong?

I put my thoughts together, and figured I could cite some specifics for the benefit of anyone who cares to listen.  I’ll be honest, functionally, R12.2 behaves pretty similarly to 11i.  The most significant impact for us was in the Financials with subledger accounting, but our WIP, Sales, HR, Purchasing, Order Management, Inventory, and Quality mostly just worked out of the box.  Sure we had to tweak all our custom reports, but the end-user experience was largely unchanged.

Most of the problems encountered lie in the techstack, and really, that’s why you’re here reading this, I suppose.  So without further ado…

Documentation: Before you Start

Wow, it’s ever-changing under your feet.  In the real world, you pick a baseline and have to dance with that partner all the way through the upgrade process.  In our case, this was 6 months.  Many, many things changed during that span, and it’s not easy to stay on top of.  If you’re like me by the time you get through your first pass against 12.2.3 you’ll have a stacks and piles of documents all over your desk.  This post is by no means meant to be all-encompassing, but the following are sure bets:

Here’s your first stop for information for 12.2.3 (it stands to reason there’s an equivalent on for 12.2.4): Doc ID 1586214.1  If you look at the end of the document, there’s a change log.  During the relevant span of my upgrade (1-May-14 through 1-Dec-14) there were many changes, some really quite critical.  This is really important to note: you’re on the bleeding edge here, I don’t care what Oracle says.  This document is shifting under your feet in critical ways and you need to throw out your printed copy and reprint it every time it changes.

For your database and the new techstack, I followed Doc ID 1594274.1 Again, you’ll notice a massive change log.  This is partially due to the changes as a result of new releases of the startCD.  When I began working the project, I chose the most advanced version I could (47) and I see now there’s a 48 and 49.  I strongly urge you to use the latest version.  With each one you get newer pieces and fixes for your techstack, which reduces your upgrade time in the long run because you don’t have to manually apply so many patches.

Finally, subscribe to and follow the updates from Steven Chan’s blog.  There’s nothing else out there like it that I’m aware of, and there were many relevant, timely updates which helped me through the process of the upgrade.

Timing is Everything

It probably goes without saying, but choose whatever the latest version is you can.  In our case, we started on 12.2.3 in May, and sometime before December 12.2.4 was released.  Of course, 12.2.4 contains fixes and enhancements we’d like to have, but as Don Rumsfield said, “you go to war with the Army you have” so we stuck with 12.2.3.  I have to admit, it was tempting to consider 12.2.4 but it would have meant a project re-baseline, and nobody wanted that.

Education & Training

Go ahead and check out Oracle education, and see how many classes there are on Ebusiness Suite 12.2.  In May, there were 0.  The first run of an online patching virtual (online) class was right around July 4th, and I took it.  Good class, by the way.  My point is this: Oracle is light years behind on providing training on this release.  I don’t know what the hold up is, but you’re pretty much on your own.  Go to conferences, read blogs, attend webinars, do whatever you can, because at this time, Oracle Education isn’t going to help you much.

Support

This was an unexpected surprise.  Perhaps I’d just gotten used to response times for 11i (which was basically on life support) but WOW!  Oracle really responds quickly to R12.2 issues.  It’s refreshing, and to that I say kudos to support.

Tax

If you’re running financials, you probably already know that there are significant changes to the tax engine in R12.  I’m not a functional guy, but I have done monthly Vertex updates, and now that I’ve gone through the upgrade I also had to do the installation of the latest compatible Vertex version.  And that’s not the half of it, there’s something called a Tax Regime which had to be recreated or adjusted, and all the while Oracle is pushing you to use EBTax which is their version of tax.  We skipped EBTax because it didn’t replace Vertex completely, and to my understanding still required us to source tax changes from elsewhere.  Regardless, know that you need to address tax, and from a functional standpoint there may not be a lot of knowledge about R12.2 tax.  In our case, we engaged Perficient to assist us and had a good experience with it.

Consulting Partner

Choose your consulting partner wisely, if you’re outsourcing some of the work.  We are a small shop, and essentially had no choice.  It’s frustrating to know that the consultants are learning the product on your dime.  In the end, as a DBA I strongly recommend you do your own heavy lifting, or you’re unlikely to like the final result, much less understand it.  I also recommend you approach the upgrade to 12.2 as a technical upgrade, and ensure your consulting partner’s strength lies in core database and applications DBA tasks.  If not, you might as well do it yourself.

Up next time: AD changes and your new techstack

January 6, 2015

Oracle R12.2 Post-Upgrade Ramblings

Filed under: R12.2 — kkempf @ 4:55 pm

upgrade

Live on 12.2

Over Thanksgiving weekend, we upgraded from 11.5.10.2 to 12.2.3.  It was a monumentally big project, at least for our organization, and I’m very happy it’s in the rear-view mirror.  There was only one “if this doesn’t get fixed we’re rolling back” moment, and for the most part the upgrade and transition has gone smoothly.  After being live for over a month, and surviving month end closes, year end closes, and multiple clones, I feel confident saying R12.2 behaves better than 11.5.10.2 once properly tweaked.

What the Upgrade Looks Like

To be honest, the upgrade is a ton of work.  There’s 11i pre-patching and prep-work (some of which cannot be done prior to down time for go-live), followed by a rapid install of 12.2, application dba tasks, running good old adpatch against a gigantic merged driver, another rapid install pass on 12.2, enabling online patching, apply lots of techstack patches, applying 12.2.3 with adop (your new best friend/worst enemy), and enough post-install tasks to make any DBA happy.  In real terms, this meant cramming about 50 hours worth of wall time into 62 hours of a weekend window.  Not much sleep.

Next Post

While the task is still fresh in my mind, I’m going to express some of the (many) pain points R12.2 brought with it, and how we coped with them.  It’s a whole new ballgame.

Blog at WordPress.com.