Kevin Kempf's Blog

July 22, 2010

A really nasty 11i bug

Filed under: 11i, Bugs — kkempf @ 3:32 pm

When is a user not a user anymore?

In 11i, we tend to end-date users’ responsibilities, and assume that’s the end of their ability to run jobs in the Apps.  Not always the case, it turns out.

So there I was, minding my own business

I’d actually seen this bug before, but at the time, when I opened an SR, the analyst “stumbled over” the solution (stop concurrent managers, run cmclean.sql, kill all pending/no manager jobs, start concurrent manager), without identifying the root cause.  So when my 11i front end became near-unresponsive, and I saw this query running at 100% on every CPU, I at least recognized it:

.
SELECT  R.Conc_Login_Id, R.Request_Id, R.Phase_Code, R.Status_Code, P.Application_ID, P.Concurrent_Program_ID, P.Concurrent_Program_Name, R.Enable_Trace, R.Restart, DECODE(R.Increment_Dates, ‘Y’, ‘Y’, ‘N’), R.NLS_Compliant, R.OUTPUT_FILE_TYPE, E.Executable_Name, E.Execution_File_Name, A2.Basepath, DECODE(R.Stale, ‘Y’, ‘C’, P.Execution_Method_Code), P.Print_Flag, P.Execution_Options, DECODE(P.Srs_Flag, ‘Y’, ‘Y’, ‘Q’, ‘Y’, ‘N’), P.Argument_Method_Code, R.Print_Style, R.Argument_Input_Method_Code, R.Queue_Method_Code, R.Responsibility_ID, R.Responsibility_Application_ID, R.Requested_By, R.Number_Of_Copies, R.Save_Output_Flag, R.Printer, R.Print_Group, R.Priority, U.User_Name, O.Oracle_Username, O.Encrypted_Oracle_Password, R.Cd_Id, A.Basepath, A.Application_Short_Name, TO_CHAR(R.Requested_Start_Date,’YYYY/MM/DD HH24:MI:SS’), R.Nls_Language, R.Nls_Territory, DECODE(R.Parent_Request_ID, NULL, 0, R.Parent_Request_ID), R.Priority_Request_ID, R.Single_Thread_Flag, R.Has_Sub_Request, R.Is_Sub_Request, R.Req_Information, R.Description, R.Resubmit_Time, TO_CHAR(R.Resubmit_Interval), R.Resubmit_Interval_Type_Code, R.Resubmit_Interval_Unit_Code, TO_CHAR(R.Resubmit_End_Date,’YYYY/MM/DD HH24:MI:SS’), Decode(E.Execution_File_Name, NULL, ‘N’, Decode(E.Subroutine_Name, NULL, Decode(E.Execution_Method_Code, ‘I’, ‘Y’, ‘J’, ‘Y’, ‘N’), ‘Y’)), R.Argument1, R.Argument2, R.Argument3, R.Argument4, R.Argument5, R.Argument6, R.Argument7, R.Argument8, R.Argument9, R.Argument10, R.Argument11, R.Argument12, R.Argument13, R.Argument14, R.Argument15, R.Argument16, R.Argument17, R.Argument18, R.Argument19, R.Argument20, R.Argument21, R.Argument22, R.Argument23, R.Argument24, R.Argument25, X.Argument26, X.Argument27, X.Argument28, X.Argument29, X.Argument30, X.Argument31, X.Argument32, X.Argument33, X.Argument34, X.Argument35, X.Argument36, X.Argument37, X.Argument38, X.Argument39, X.Argument40, X.Argument41, X.Argument42, X.Argument43, X.Argument44, X.Argument45, X.Argument46, X.Argument47, X.Argument48, X.Argument49, X.Argument50, X.Argument51, X.Argument52, X.Argument53, X.Argument54, X.Argument55, X.Argument56, X.Argument57, X.Argument58, X.Argument59, X.Argument60, X.Argument61, X.Argument62, X.Argument63, X.Argument64, X.Argument65, X.Argument66, X.Argument67, X.Argument68, X.Argument69, X.Argument70, X.Argument71, X.Argument72, X.Argument73, X.Argument74, X.Argument75, X.Argument76, X.Argument77, X.Argument78, X.Argument79, X.Argument80, X.Argument81, X.Argument82, X.Argument83, X.Argument84, X.Argument85, X.Argument86, X.Argument87, X.Argument88, X.Argument89, X.Argument90, X.Argument91, X.Argument92, X.Argument93, X.Argument94, X.Argument95, X.Argument96, X.Argument97, X.Argument98, X.Argument99, X.Argument100, R.number_of_arguments, C.CD_Name, NVL(R.Security_Group_ID, 0)
FROM fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C, fnd_concurrent_queues Q, fnd_application A2, fnd_executables E, fnd_conc_request_arguments X
WHERE R.Status_code = ‘I’ And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or (R.OPS_INSTANCE = decode(:dcp_on,1,FND_CONC_GLOBAL.OPS_INST_NUM,R.OPS_INSTANCE))) And R.Request_ID = X.Request_ID(+) And R.Program_Application_Id = P.Application_Id(+) And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+) And R.Program_Application_Id = A.Application_Id(+) And P.Executable_Application_Id = E.Application_Id(+) And P.Executable_Id = E.Executable_Id(+) And P.Executable_Application_Id = A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id = C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id = :q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag is NULL OR P.Enabled_Flag = ‘Y’) And R.Hold_Flag = ‘N’ And R.Requested_Start_Date <= Sysdate And ( R.Enforce_Seriality_Flag = ‘N’ OR ( C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag = ‘N’ OR Not Exists
(Select Null
From Fnd_Concurrent_Requests Sr
Where Sr.Status_Code In (‘R’, ‘T’) And Sr.Enforce_Seriality_Flag = ‘Y’ And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <= Q.Max_Processes And R.Rowid = :reqname And ((P.Execution_Method_Code != ‘S’ OR (R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN ((0,98),(0,100),(0,31721),(0,31722),(0,31757))) AND ((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN ((510,40112),(510,40113),(510,40910),(510,40911),(530,42260),(530,42261),(535,40893),(535,40894),(535,40895),(660,44069),(660,44100),(660,44808),(660,44849),(660,46421),(660,46422),(702,33152),(20004,44868),(20009,45000)))) FOR UPDATE OF R.status_code NoWait
SELECT  R.Conc_Login_Id, R.Request_Id, R.Phase_Code, R.Status_Code, P.Application_ID, P.Concurrent_Program_ID, P.Concurrent_Program_Name, R.Enable_Trace, R.Restart, DECODE(R.Increment_Dates, ‘Y’, ‘Y’, ‘N’), R.NLS_Compliant, R.OUTPUT_FILE_TYPE, E.Executable_Name, E.Execution_File_Name, A2.Basepath, DECODE(R.Stale, ‘Y’, ‘C’, P.Execution_Method_Code), P.Print_Flag, P.Execution_Options, DECODE(P.Srs_Flag, ‘Y’, ‘Y’, ‘Q’, ‘Y’, ‘N’), P.Argument_Method_Code, R.Print_Style, R.Argument_Input_Method_Code, R.Queue_Method_Code, R.Responsibility_ID, R.Responsibility_Application_ID, R.Requested_By, R.Number_Of_Copies, R.Save_Output_Flag, R.Printer, R.Print_Group, R.Priority, U.User_Name, O.Oracle_Username, O.Encrypted_Oracle_Password, R.Cd_Id, A.Basepath, A.Application_Short_Name, TO_CHAR(R.Requested_Start_Date,’YYYY/MM/DD HH24:MI:SS’), R.Nls_Language, R.Nls_Territory, DECODE(R.Parent_Request_ID, NULL, 0, R.Parent_Request_ID), R.Priority_Request_ID, R.Single_Thread_Flag, R.Has_Sub_Request, R.Is_Sub_Request, R.Req_Information, R.Description, R.Resubmit_Time, TO_CHAR(R.Resubmit_Interval), R.Resubmit_Interval_Type_Code, R.Resubmit_Interval_Unit_Code, TO_CHAR(R.Resubmit_End_Date,’YYYY/MM/DD HH24:MI:SS’), Decode(E.Execution_File_Name, NULL, ‘N’, Decode(E.Subroutine_Name, NULL, Decode(E.Execution_Method_Code, ‘I’, ‘Y’, ‘J’, ‘Y’, ‘N’), ‘Y’)), R.Argument1, R.Argument2, R.Argument3, R.Argument4, R.Argument5, R.Argument6, R.Argument7, R.Argument8, R.Argument9, R.Argument10, R.Argument11, R.Argument12, R.Argument13, R.Argument14, R.Argument15, R.Argument16, R.Argument17, R.Argument18, R.Argument19, R.Argument20, R.Argument21, R.Argument22, R.Argument23, R.Argument24, R.Argument25, X.Argument26, X.Argument27, X.Argument28, X.Argument29, X.Argument30, X.Argument31, X.Argument32, X.Argument33, X.Argument34, X.Argument35, X.Argument36, X.Argument37, X.Argument38, X.Argument39, X.Argument40, X.Argument41, X.Argument42, X.Argument43, X.Argument44, X.Argument45, X.Argument46, X.Argument47, X.Argument48, X.Argument49, X.Argument50, X.Argument51, X.Argument52, X.Argument53, X.Argument54, X.Argument55, X.Argument56, X.Argument57, X.Argument58, X.Argument59, X.Argument60, X.Argument61, X.Argument62, X.Argument63, X.Argument64, X.Argument65, X.Argument66, X.Argument67, X.Argument68, X.Argument69, X.Argument70, X.Argument71, X.Argument72, X.Argument73, X.Argument74, X.Argument75, X.Argument76, X.Argument77, X.Argument78, X.Argument79, X.Argument80, X.Argument81, X.Argument82, X.Argument83, X.Argument84, X.Argument85, X.Argument86, X.Argument87, X.Argument88, X.Argument89, X.Argument90, X.Argument91, X.Argument92, X.Argument93, X.Argument94, X.Argument95, X.Argument96, X.Argument97, X.Argument98, X.Argument99, X.Argument100, R.number_of_arguments, C.CD_Name, NVL(R.Security_Group_ID, 0)FROM fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C, fnd_concurrent_queues Q, fnd_application A2, fnd_executables E, fnd_conc_request_arguments XWHERE R.Status_code = ‘I’ And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or (R.OPS_INSTANCE = decode(:dcp_on,1,FND_CONC_GLOBAL.OPS_INST_NUM,R.OPS_INSTANCE))) And R.Request_ID = X.Request_ID(+) And R.Program_Application_Id = P.Application_Id(+) And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+) And R.Program_Application_Id = A.Application_Id(+) And P.Executable_Application_Id = E.Application_Id(+) And P.Executable_Id = E.Executable_Id(+) And P.Executable_Application_Id = A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id = C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id = :q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag is NULL OR P.Enabled_Flag = ‘Y’) And R.Hold_Flag = ‘N’ And R.Requested_Start_Date <= Sysdate And ( R.Enforce_Seriality_Flag = ‘N’ OR ( C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag = ‘N’ OR Not Exists(Select NullFrom Fnd_Concurrent_Requests SrWhere Sr.Status_Code In (‘R’, ‘T’) And Sr.Enforce_Seriality_Flag = ‘Y’ And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <= Q.Max_Processes And R.Rowid = :reqname And ((P.Execution_Method_Code != ‘S’ OR (R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) IN ((0,98),(0,100),(0,31721),(0,31722),(0,31757))) AND ((R.PROGRAM_APPLICATION_ID,R.CONCURRENT_PROGRAM_ID) NOT IN ((510,40112),(510,40113),(510,40910),(510,40911),(530,42260),(530,42261),(535,40893),(535,40894),(535,40895),(660,44069),(660,44100),(660,44808),(660,44849),(660,46421),(660,46422),(702,33152),(20004,44868),(20009,45000)))) FOR UPDATE OF R.status_code NoWait

What Enterprise Manager was telling me

The database itself was reasonably happy.  I mean it was running a stupid query over and over, but it was doing its job.  It was the application server that was getting killed

As you can see, the application server got slammed about 11:15am.  It was running out of memory, the load was in the 20’s (8 CPUs available) and Linux was starting to kill processes in order to preserver itself.

Root Cause and Fix

If you ever see this query, or possibly this one:

update  fnd_concurrent_requests set phase_code = :phase ,status_code = :status, completion_text = :text where request_id = :reqid

you are likely encountering bug 9109247: SCHEDULED REQUESTS STILL RUN FOR END_DATED USERS 100% CPU.

The immediate fix is to shut down concurrent managers (adcmctl.sh) and look in the concurrent request queue to see who is trying to run what (start with the lowest request ID which is trying to run).  In my case, one user (HFAULLING) had submitted about 20 similar requests.  I cancelled these requests (in fact, all of this users’ requests), can cmclean.sql and restarted the managers.

Turns out, this user was end dated the day this happened.  In other words, they weren’t even a valid Oracle apps user anymore.  But this bug allowed their scheduled concurrent requests to run, get stuck in some crazy endless loop, and kill my system (since he had submitted more requests than I had standard managers and they all queued up at the same time).

In closing

I decided to post this information for two reasons.  One, I think it’s rather random and impossible to predict when you may hit this bug.  It’s my opinion that this should have received a lot more press from Oracle as a recommended patch!  Wouldn’t it be rather embarrassing if it got out that Oracle 11i not only allowed terminated employees to run concurrent requests, but by doing so they could effectively sabotage the system?  Second, it’s a bug which can have catastrophic consequences to the whole ERP.  The first time I hit this bug I extracted the (long) query (above) and submitted it to the analyst asking them what the h*ll it was.  At the time, they had no idea.  Seems like it should be in the various notes related to the bug as well.


Advertisements

February 15, 2010

Advanced Compression with 11i, Take 2, Day 1

Filed under: 11g, 11i, advanced compression, Bugs — kkempf @ 10:56 am

So far, so good

After waiting about a year since my first attempt, the timing was finally right for me to patch the core 11.1.0.7 RDBMS with Patch 8277580: ORA-07445: [__INTEL_NEW_MEMCPY()+44].  This was Oracle’s fix for the data corruption I was seeing on my dataguard physical standby after I compressed BOM.CST_ITEM_COST_DETAILS (no other tables seemed affected in this way).

I had my doubts, but nearly 24 hours later, and several cost rollup runs have elapsed without incident. The raw stats look great:

OWNER TABLE_NAME                     VERSION         PRE_COMPRESSION_MB POST_COMPRESSION_MB COMPRESSION_RATIO
----- ------------------------------ --------------- ------------------ ------------------- -----------------
BOM   CST_ITEM_COST_DETAILS          11.1.0.7.0                 1717.38              429.88               .75

By the way, it’s amazing how much “stuff” touches that particular table in a discrete manufacturing 11i environment.  Can you tell when I recreated the indexes?

Rebuild your indexes!

November 24, 2009

Opatch: This patch is not suitable for this operating system

Filed under: 10g, Bugs, Linux, Oracle — kkempf @ 9:17 am

An imposter

Last night I was applying the October 2009 CPU/PSU to a bunch of development databases and application servers when I hit a bug which I hadn’t seen in a while.  The system was a Red Hat 5 32-bit install, nothing special there:
uname -a
Linux localhost 2.6.18-164.6.1.el5PAE #1 SMP Tue Oct 27 11:46:58 EDT 2009 i686 i686 i386 GNU/Linux
Still, during the opatch napply step of patch 8874205 for AS 10.1.2.3, I received this error:
OPatch detects your platform as 23 while this patch 8447875 supports platforms:
46 (Linux Intel)
226 (Linux x86-64)
OPatch detects your platform as 23 while this patch 8447875 supports platforms:   46 (Linux Intel)   226 (Linux x86-64)
This patch is not suitable for this operating system.
Please contact support for the correct patch.
ERROR: OPatch failed during prerequisite check.
What’s sad is that I remembered seeing this before, and just needed a refresh on the syntax to fix it.  As I recall, you can also go into the patch and change some script, but the simplest fix if you hit this is as follows:
$ export OPATCH_PLATFORM_ID=46
$ opatch napply
Worked great after this, opatch completed and the patchset went in fine.  A word of caution.  I’m sure this is a Linux specific error, and My Oracle Support confirmed this.  I’m also sure it’s there for good reason: an idiot-proofing check to make sure you’re not applying the HP-UX patch to RedHat Linux, or the like.  Before you do what I’ve shown, be extra certain that your OS is what you think it is (including whether it’s 32 or 64 bit) and also that the patch you pulled is for this same corresponding platform!

July 27, 2009

Pulling the plug on 11i Advanced Data Compression

Filed under: 11g, advanced compression, Bugs, Dataguard — kkempf @ 10:49 am

This past weekend I uncompressed all my tables, rebuilt the associated indexes, and said adieu to Advanced Compression. Based on my (still open) Dataguard block corruption SR, the admin at work likes to call it Advanced Data Corruption. Regardless, I may have lived with the first issue as a beta tester, but after a second ORA-0600 related to the product cropped up last week, I gave up. When support admits that they have no documentation of the error internally, and begins suggesting the most inane course of action you can come up with, it’s time to give it the boot. It is my opinion that while this product works in the simplest environment, on the whole it is half-baked, not well tested, and not worth being a “guinea pig” or “production beta tester” for it.

Incidentally, the syntax to uncompress (and move, recreating the object completely uncompressed) is:

alter table BOM.MTL_MATERIAL_TRANSACTIONS move nocompress;

I have to say I really wanted this product to work. In principle, it’s a great idea, and in practice, it seems close to achieving what I would say was success:

  • Disk sat idle most of the day because everything was in the buffer cache
  • Most objects compressed extremely well, and total disk consumption fell significantly
  • With the exception of Dataguard (and the unknown OLTP ora-0600), it operated seamlessly with the apps

In the end, however, I’d tell anyone considering it to wait.  It’s not there yet.

*edit* Dataguard has been up and running just fine, for over 2 weeks now since I uncompressed all the database objects. Oracle development must be very close to what they think is a fix, based on my SR/bug. But I’m not putting compression back in anytime soon.

July 8, 2009

When EM Flips Out

Filed under: Bugs, Enterprise Manager, Oracle — Tags: , , , — kkempf @ 7:35 am

For no apparent reason, yesterday afternoon about 2pm Enterprise Manager went haywire. It was using 100% of 1 CPU, and causing considerable I/O (log writer). The truth is, it doesn’t take much to kill a 1-CPU virtual machine, but this was without precedent.
EM1EM2

After some digging, the offending SQL was:
SELECT execution_id
FROM MGMT_JOB_EXEC_SUMMARY e, MGMT_JOB j
WHERE e.job_id = j.job_id AND j.is_corrective_action = 0
AND status IN (5,4,3,18,8)
AND (CAST(SYS_EXTRACT_UTC(SYSTIMESTAMP) AS DATE) – e.start_time) > (:tf)
AND ROWNUM < 500 ORDER BY start_time desc

A few searches on Metalink, and I find DOC 839080.1, which tells me to apply patch 8517252. This patch applies against the EM Application Home, not the database. So I shut down the application, opatch in 8517252, but the post script @post_install_script.sql is hanging. Had to hard kill (kill -9) the 100% cpu process at the OS level and this cleared the lock contention. Finished installing the patch, fired up the EM application again, and all was good (this is in the graphs above, about 5pm).

June 18, 2009

11g Dataguard/Advanced Compression Bug

Filed under: 11g, Bugs, Oracle — Tags: , , — kkempf @ 3:50 pm

Ah it was inevitable. I spoke too kindly of RDBMS 11g. Now I’m stuck waiting on Oracle Development to fix a major problem. The gist of it is that after I began compressing tables with 11g Advanced Compression, the dataguard instance would crash.

First, I saw odd ORA-07445 [__INTEL_NEW_MEMCPY()+44] SIGSEGV and ORA-0600 errors in my standby alert log. Eventually the instance crashed. After much research, I enabled the init parm db_block_checking (highly recommended!) and the error became much clearer; because of db_block_checking it was no longer writing garbage it was failing the check:

Errors in file /u01/appprod/oracle/proddb/11.1.0/log/diag/rdbms/proddg/PROD/trace/PROD_pr0e_226
66.trc:
ORA-10562: Error occurred while applying redo to data block (file# 19, block# 445469)
ORA-10564: tablespace APPS_TS_TX_DATA
ORA-01110: data file 19: ‘/u05/appprod/proddata/apps_ts_tx_data07.dbf’
ORA-10561: block type ‘TRANSACTION MANAGED DATA BLOCK’, data object# 1186389
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kddummy_blkchk], [19], [445469], [6110], [], [], [], [], [],
[], [], []

Datafile 19, block 445469 tied back to the APPS_TS_TX_DATA tablespace and the object was BOM.CST_ITEM_COST_DETAILS. Yep, it was one of the handful of tables which I’d compressed so far.

The analyst tied it to bug 8277580 and tells me there’s unpublished parts of the bug which mention compression
Bug 8277580 ORA-7445: [__INTEL_NEW_MEMCPY()+44]
RDBMS Ver: 11.1.0.7
O/S: 226 Linux x86-64

What scares me is that the bug was opened 4 months ago. If they drag their feet too long, I guess I uncompress everything, and they send me a refund for Advanced Compression and the year of support I already paid, right?

One note about db_block_checking. It defaults to “FALSE” and the docs say it incurs a 1-10% overhead. With the alternative prospect of silently corrupting my standby (Gah!) I can’t help but think this is a no-brainer to activate. With db_block_checking enabled, the behavior was what I would consider to be appropriate: The standby database managed recovery process stops, and the RDBMS stays up. If you restart the managed recovery process, it dies on the exact same log and with the exact same error.

One follow up thought: At this point, I have to ask myself, do I have a recoverable database? In other words, my assumption so far is that dataguard and the log ship process is somehow corrupting logs due to advanced compression. Interesting trivia note: if you MD5 checksum the same archivelog on the primary and the standby, it will NOT match! I only thought to do this under 11g.. anyone know if they match under 10g? Is this my core problem?

I will have to prove this by restoring a backup of my database and recovering until cancel to roll through a few hundred logs. If, on the other hand, the local archivelogs are being written “wrong”… I’m fooked.

*edit* I’m not getting far with Oracle on this bug. It turns out, when I look at the bug status codes for this issue, it’s essentially unchanged since it was open. I sent this email to my sales rep in hopes of “lighting a fire” as we used to say in the Army:

After further working with support, I’m extremely unhappy with this situation. After we spoke last week, I’d mentioned that this bug had existed since late February 2009. Since then, there have been no significant updates to the bug, and when I looked up the bug status code, I was even more dismayed. According to Doc 16660.1, this bug has been in 1 of 2 statuses since inception:

10: Use to file a bug without all the needed information (for instance, trace files from the customer), to inform Development that a bug will soon become Status 11.

16: Used by BDE to pre-sort well-formed bugs before development start fixing.

This isn’t exactly encouraging! The way I read this is that for 4 months, development has either been trying to gather needed information or pre-sorting the bug before actually working on it. This means that our (certified) configuration has been broken since we bought it.

The only option the analyst gave me was to uncompress my tables, and I’m afraid that I may have to go this route since I have absolutely no feel from support whether this bug is even being worked or when it may be fixed. While I agree this is likely to fix the Dataguard bug, does it come with a refund for Advanced Compression?

*edit* It’s been one month since support positively identified my bug was existing bug 8277580. While there is still no resolution, I am being asked for a bunch of trace files and logs this week, which hopefully at least means someone is looking at it. While they don’t always tie out to the same object or block, the data guard apply always fails with an ORA-0600 and a block reference to a compressed table. Meanwhile, our admin is keeping a live snapshot at our standby site via the SAN; meaning at worst, if our primary failed we could start up the database on the remote end without losing much (if any) information. It requires a huge amount of storage and bandwidth, however, which Dataguard does not.

*edit* Now support is asking for the possibility of uploading/providing all the components necessary to reproduce the bug on their end.  Odd, since they acknowledge the bug, but I’ll be curious to see what comes of this.  I can’t very well upload the entire database to them (at least not easily), and even to send them one offending datafile with logs to reproduce the issue seems difficult.   I break my “data” and “index” datafiles into 8GB’s per, and currently have about 9 of them (last I looked), so at a minimum we’re probably talking about 8-10GB to reproduce this at their end.

Blog at WordPress.com.