Kevin Kempf's Blog

May 30, 2010

Edisto Island 2010

Filed under: Vacation — kkempf @ 6:25 am

We’re heading back to Edisto this summer, we picked a place on the Wyndham this year.  Obviously we’re really happy with Edisto; it hits the quiet spot between isolated nature preserve and over-populated (Hilton Head!)

We’ve stayed at the Wyndham before (perfect little cabin!), on a long weekend, and this year it fits our needs perfectly.

Last summer, the extended family stayed across from the beach at this rental.

One drawback with the rental this summer is that it’s not oceanfront, but our small kids prefer the pool (there’s several within a short walk), plus there’s a playground, miniature golf, and a shuttle to the beach.

767 Summerwind Circle map


Emergency Numbers

Dr. Rebecca Borders, Island Family Medicine 843-869-3111
Dr. Karen Yeargin, Dentist 843-869-1294
Edisto Dental Associates 843-869-3368
All other emergencies: 911

General Sites


Piggly Wiggly 102 Jungle Rd 843-869-0010 (HOURS: 7:00AM-9:00PM, Sun.-Thur.; 7:00AM-10:00PM, Fri.-Sat.)  (Best grocery store in the area, has little selection but everything you need)

Sea Spirits Liquor 104 Jungle Rd 843-869-1822 145 Jungle Rd 843-869-3222  (We had a great lunch here our first time out)
Po Pigs BBQ 2410 Highway 174 843-869-9003 (About a 20 min drive from the island, good BBQ but no better or worse than what’s available elsewhere in SC) 114 Jungle Road 843-869-5000
Island Video and Ice Cream  114 Jungle Rd 843-869-3134 101 Palmetto Blvd 843-869-4474 (We had lunch here last year, it was fine but the service wasn’t great) 1442 Highway 174 843-869-2339 (This is on our list, we keep hearing good things) 108 Jungle Rd 136 Jungle Rd 843-869-1400 (Had a great breakfast here last year)
Plantation Grille 21 Fairway Drive 843-869-2561
Bell Buoy Seafood 3706 Docksite Rd 843-869-2222       (Appears that this is gone as of 2009, which is sad)
Bailey’s Sweet Shop 1467 Highway 174 843-869-9008
King’s Farm Market 2559 Highway 174 (843) 869-3600 (On your way onto the Island – Wonderful pies!)

Rental Agencies




Edisto Realty

Things to Do 140 Jungle Rd  (The family rented several bikes from here last year, no problems at all) 3731 Docksite Road 436 Hwy 174 (We’ve visited here before, nice beach, nice amenities, but I wouldn’t pay to go here if I was already staying on the island) 1374 Highway 174   (Our 3 year old can’t stop talking about how he wants to go back here – surprisingly excellent!) 1589 Highway 174 (golf) 19 Fairway Dr

May 25, 2010

It turns out, Windows 7 wasn’t my idea

Filed under: Ubuntu, Windows 7 — kkempf @ 8:48 pm

Up front warning: I’ve abandoned Windows 7 as my primary OS at work.  I dumped it for Ubuntu 10.04 LTS “Lucid Lynx” with Windows 7 running in a Sun/Oracle Virtual Box.  Instead of “nitpicking” about all of the things it didn’t do to my satisfaction, I decided to look at what I really need my work PC to be able to do.  Through the course of this, I also realized that my work PC does a completely different set of tasks than my home PC, were Windows 7 is more suited.

The following table chronicles how I “work” on a PC in the workplace.  While some features are available on OS’s I didn’t checkbox, it’s because their implementation is inferior to the other one which is.  Details below the table.

Feature Ubuntu Windows 7 Cent OS
Email Client
ssh client
Flash plugin for browser
Java plugin for browser
VNC/RDP client
SQL Developer
sftp client
Screen Manager
Discoverer 10g Admin Client
Tether Blackberry modem (bluetooth) with VPN
Dual Monitor Support
Open/edit modern Word, Excel, Powerpoint
SQLplus Client
Nice to Have
Fast Bootup
IM Client

Email Client/Calendar: Of course, Ubuntu and CentOS offer an email client with Calendar, called Evolution. The problem is, it’s an awful product. It crashes frequently. It doesn’t sync well with a Blackberry, doesn’t import my calendar/appointments right, and you must install an unincluded Exchange MAPI protocol to get anywhere close to usable with Exchange.

SSH Client: You can find many clients for Windows. None I found are worth the electrons it takes to download them.

Java/Flash plugins: Take note, none of these are officially supported under any flavor of Linux.

VPN: It’s point and click under Windows 7 and (surprisingly!) Ubuntu.  I’m referring specifically to PPTP.

VNC/RDP: Windows has this native, which is easier.  My compromise was to run Ultra VNC under Wine in Ubuntu.

Screen Manager: Virtual screens, on your taskbar, which track what applications you are running there.  On a dual monitor setup, this means you can easily have 8 applications running full screen, and simply click the appropriate screen in the taskbar to get to the ones you need.  For example, I keep the Windows 7 Virtual Box (Exchange) and Oracle Enterprise Manager Grid Control (performance monitor) running on one screen.  Those are pretty much always needed.  The other 3 screens are for whatever I’m working on “at the moment” and provide a clean desktop to do so, instead of tabbing through 5-10 applications.

Tethering a Blackberry: Again, surprisingly simple under Windows 7 with Verizon access manager, but also simple (all GUI fied) under Ubuntu.   The information online and in Ubuntu help is not so great; it’s good fodder for a future post.

Fast Bootup: Ubuntu is about a 1 minute boot to a functional OS.  Windows 7 is at least three times this.

Silverlight is nifty for streaming Netflix movies during lunch.

In the end, the only thing Windows was giving me that I needed every day was a good email & calendar client.  Since my forays into Discoverer Admin are occasional at best, I’m happy to relegate Windows  to a VM in (free) Sun/Oracle Virtual Box.

Finally, to give a visual, here’s what the Ubuntu desktop looks like (at least screen 1 of 4):

Note that Windows is happily running inside Ubuntu (double task bars there).

May 11, 2010

Advanced Compression and Interface Tables

Filed under: 11g, 11i, advanced compression — kkempf @ 2:57 pm

Advanced Compression – More Stories from the Front Lines

We’re several months into advanced compression, after the Dataguard bug/debacle, and we’re really quite happy with the product.  Practically speaking, the ERP runs from the buffer cache, and thus our I/O is down to a trickle.  As a matter of course, I began to compress tables, in mass, during the evening in an effort to just “get everything compressed”.  Not that the gain is so good (most of the big tables are done) but in the interest of standardization.  I don’t want to have to go look and see “is it compressed?”.  Recently, however, I arrived one morning to a standard email report I have delivered which told of dozens and dozens of lines stuck in the INV.MTL_TRANSACTIONS_INTERFACE table.  For those not familiar with discrete manufacturing, this is the table where material moves and completions are processed among other things (due at least in part to Highjump, a bolt on application which allows workers to use wireless hand held terminals to do their job).  In other words, in a manufacturing environment, it’s a busy place and it needs to work correctly.

Hello, ORA-00600

The alert log doesn’t lie.  This entry corresponded to the failure times of the material transactions:

Errors in file /u01/appprod/oracle/proddb/diag/rdbms/prod/PROD/trace/PROD_ora_8556.trc  (incident=9989):
ORA-00600: internal error code, arguments: [kdsgrp1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/appprod/oracle/proddb/diag/rdbms/prod/PROD/incident/incdir_9989/PROD_ora_8556_i9989.trc

The trace file revealed the RDBMS file# and block#; running that to ground confirmed that it was the INV.MTL_TRANSACTIONS_INTERFACE table.   Searching Metalink/MOS on the term KDSGRP1-1 (and changing the default search so it included bugs) , I found bug 9072813.  This was for an HP environment, but was such a similar hit that I pursued the “base bug” 8771916 which, although you couldn’t see the bug, had a patch corresponding to it.  A very, very recent patch, entitled “8771916: ORA-00600 [KDSGRP1] WHEN DOING AN UPDATE”.

Summing it all up

The first thing I did was to uncompress the MTL_TRANSACTIONS_INTERFACE table.  It went from 16 blocks back to 800, and .13 Mb back to 6.25 Mb.  That fixed the issue, but there’s more to the story.  The bug in question appears to pop when there’s massive write contention to a table (read: block).  I had just reduced the number of blocks in this table 50 fold; that’s bound to cause more contention.  In the end my fix was to also apply patch 8771916 to my production environment, knowing there could be other tables where this could occur.  If you’re running Advanced Compression, I’d urge you to at least look into the bug/patches.  If you’re running 11i and compression, I’d definitely avoid compressing interface tables.

Am I compatible with Windows 7 in the Workplace?

Filed under: Windows 7 — kkempf @ 2:14 pm

Resistance is futile


I use 64-bit Windows 7 at home, on my primary PC.  It has a lot of memory, 4 CPUs and in general does everything I could ever ask of it, and it does it well.  So I can say from the onset, I don’t have any dislike for Windows 7 (Vista was iffier).  Because of an IT initiative, all new PCs at work are being purchased with Windows 7, and I figured it was time to try it on the desktop.  I did the unfathomable, I wiped CentOS and installed 64-bit Windows 7 on my Dell D820.

There’s something really nice about having a PC which just “works”, in so many ways.   Most notably, this is in regards to hardware.  I was happily running installs of all my old favorite software… but the honeymoon is over.  Now I’m really debating, what have I done and which operating system is truly more suited for the way I work?

I have to warn in advance, this will likely be a living entry, as I find things that amaze or annoy me with Win 7.

The Terminal

I spend most of my (working) day in a terminal session.  There’s almost never not at least one open on my PC somewhere.  It’s critically important that I like it.

  • I tried Cygwin, but when I couldn’t figure out how to get an ssh session going after 5 minutes, I gave up.
  • I tried Powershell, but I was stuck in a crummy Windows command prompt.
  • I grudgingly use Putty, but it’s so GUIfied that it annoys me.
  • I’ve installed Sun Virtual Box, running Ubuntu in seamless mode on my Windows 7 desktop.  Yes, I effectively have a whole VM and kernel dedicated to running a terminal session at the moment.

The Matrix

I decided I’d just set up a matrix here, of the things each OS did right, and wrong.  Empirical evidence, if you will.

Windows 7 – The Good

  • Wireless
  • VPN
  • Office 2007 docx’s finally open!  Open Office could do this, but not on the version CentOS delivered
  • Outlook is about a million times better than Evolution, in every way from stability to features
  • Windows Clipping tool for screenshots
  • I can run my favorite editor ever, Codewright again!
  • Chrome for Windows is great.  It’s superior to IE in every way. Don’t tell Oracle, but unofficially and unsupported it runs 11i, with no tweaks on AD.I.6/ATG PF H RUP 6.  It looks like the Linux version might be out now..
  • Verizon Access Manager software for using my Blackberry as a Bluetooth tethered modem.  Wow that’s easy.

Windows 7 – The Bad

  • No native screen or page manager (WinPager works OK)
  • Always nagging me about updates

Windows 7 – The Ugly

  • Java must run 4x slower under Windows than under Linux.  I think SQL Developer takes 4 days to launch.
  • There is not a single terminal program I’ve found that I like
  • I have to run an sftp GUI to move files I download and land onto servers – it’s called Filezilla and it’s awful
  • IE 8 is the clumsiest,  slowest-launching, most bloated browser I’ve ever seen.  I believe if I had to use it to get to the maddeningly slow “My Oracle Support”, it might never come back.   And it has the nerve to default my searchbar to Bing??

CentOS – The Good

  • Oh, the simplicity of right clicking on your desktop and having a feature rich terminal session open!
  • Highlighting text and having it automatically on your clipboard, then right clicking to past.  No ctrl-c, ctrl-v or right click copy, right click paste.  This is perhaps the most frustrating loss.
  • X server automatically!
  • Security.  I can lock my terminal and pretty much nobody in the company can get into my PC.  Including IT folks unless they’re REALLY determined.  No viruses.  No 100mb McAfee antivirus updates.

CentOS – The Bad

  • Using GIMP for cropping screenshots
  • Wine is a hit and miss proposition; so, too, is Crossover Office, no matter what they claim.

CentOS – The Ugly

  • Wireless is a nightmare

May 8, 2010

EM 11g Grid Control Install

Filed under: Enterprise Manager, Oracle — kkempf @ 7:46 pm


For no reason other than I could, I recently decided to undertake installing the new Oracle Enterprise Manager 11g Grid Control (technically, upgrade my environment).   For those not in the know, it was released about 2 weeks ago, and one major change is that it utilizes Weblogic server instead of the old iAS (this is the Fusion model).   In fact, there is no application server included in the EM 11g download; the installer assumes you have a Middleware home with the appropriate version of Weblogic server already installed.   The first thing I did in preparation for the upgrade  was to contact our Oracle techstack sales guy and ensure that I was allowed to install Weblogic server.  While I’m fully licensed to run EM with tuning and diagnostics packs, I didn’t want to make any assumptions about Oracle giving up a new application server for free.  After confirming my legality from a license perspective, I began to update our internal wiki with all of the pieces of EM which would be hard to remember/replace in the event that they were lost.  This included a list of current EM users and their access, scheduled jobs, and user-defined metrics.  Finally, I shut everything down and made a VMWare snapshot of the disk, so that I could revert painlessly, if necessary.

Weblogic Server

The next step was to install Weblogic server, which is a pre-requisite to running the EM 11g installer.  In reality, you could install this web server anytime, whether EM 10g was running or not.  I went to pull it from OTN, and noticed a few things.  First, it appears that Fusion Middleware 11gR1 (10.3.3 – in typical Oracle versioning tradition) has been release very recently.  I was only interested in what I knew was certified, Fusion Middleware 11g (10.3.2 – couldn’t Oracle put one person in charge of making sure that the technical versions match the product names, and that the versions were generally aligned?).  You can find the download page here.  I was looking for linux x86_64, which isn’t on their list, so I pulled the generic platform version.  There’s a small net installer (stub file which goes out and grabs the install later) or a package installer (traditional, everything in there version). Grab whatever floats your boat.   When I grab the generic version, it requires I have my own JDK installed, so to invoke the installer it looks something like this:

java -jar wls1032_generic.jar

The familiar Oracle looking installer starts; I’ll use screenshots from here on as it will probably be easier to follow, just go left to right:

1. Initial splash screen

2. Give us the information we already have!

My personal favorite after leaving email address blank:

Are you sure you want to remain clueless??

You didn't give us your email! Are you really really sure we can't spam you?

OK, now back to the installation… I missed the screenshot, but somewhere you are asked for 2 passwords.  Make sure you

jot them down, as you’ll need them later to administer the web server and also to do the actual EM install.

3. Where is your JDK?

4. Where do you want to put the Weblogic App Server?

5. Default components

6. Installer at Work


7. Complete (Uncheck Quickstart)

8. Quickstart - If you get to this screen... cancel it and exit!

Enterprise Manager 11g Installation

This installation impressed me; it was straightforward and worked.  I do have to ding it on a few things.  First, Oracle decided that EM 11g would run by default as https using an SSL cert it generates during the install.  So many problems there.  First, our proxy server (where all web traffic goes, by default, even if  redirected back internally) has a security mechanism set up to not allow SSL on non-standard (443, 8443) ports.  Oracle chose something logical: port 1159.  Second, we have an internal certificate I could use, which would be far more legitimate.  How about asking me if I have something for it to chew on, instead of having it generate some garbage cert which I have to figure out how to replace?

I ran into some RDBMS parameter issues.  EM 11g requires you to have the repository running 11.1 or 11.2, and there’s even some patches they want in place before you do the upgrade.   Well worth reviewing the prerequisites and documentation.  Before you run the installer, issue an

alter system set job_queue_processes=0;

and revert it upon completion to its original value.

Finally, I ran into a problem early on when the installer detected the Oracle Applications Management Pack in my EM 10g inventory.  At some point in the past, I’d installed it with the intention of seeing what it brought to the table.  That’s as far as I ever got, never even configured it, but the installer basically says you can’t do an upgrade with that pack in place.  You have to do a full install.  Ick.  So I uninstalled it via OUI from the OMS home and it happily proceeded next time I ran the installer.  I guess if you’re using that APM you’re out of luck.

You can grab EM 11g from OTN here; once extracted & staged you just use the old standby:


1. Welcome to our latest product! Can we have your email please?

2. We really need to know more about you. Can we have your username and password to the world's slowest support site?

3. I chose to upgrade

4. Hey! I found your old stuff!


5. I see your listener - whats the sys password?

6. Hey Bozo! Didn't you read the upgrade guide? There's some mandatory parameters you need to fix!

7. I found your Middleware home. Where do you want to install EM 11g?

8. What were your Weblogic passwords?



Did you notice in step 8, it’s adding a tablespace?

9. Do you want to change ports that you don't even understand the use of?

10. Ready to launch!



11. Chugging along for about an hour

12. The we all know and love, now available in!


13. Read the fine print; it has pertinent URLs and information!

14. More fine print





15. Do you agree to tell the truth, the whole truth, and nothing but the truth?







Success & First Impressions

First of all, it works, the installer didn’t error out too much, and after getting past the ssl port/cert issues (use IE it has the best error messages!) I have to say my first impression was that it did everything right.  My users were all still there, all my targets were there, my user-defined metrics survived, and my jobs were all running.  It even emailed me an alert I was subscribed to when I shut down a scrub database.  No tweaks.  The graphics look a little cleaner.  I am guessing that the java pages were compiled on the fly, as it was painfully slow to do anything the first time.  After that, it was noticeably faster than EM 10g.  Must be all the Fusion going on.

Here’s what the screen looks like:

A new look, but familiar in most ways.








Starting and Stopping

This has changed a little bit, but I thought I’d just copy my start/stop scripts in here wholesale and you can tweak them for your environment:


$ORACLE_HOME/bin/emctl start oms
$AGENT_HOME/bin/emctl start agent
$ORACLE_HOME/bin/emctl status oms -detail


$ORACLE_HOME/bin/emctl stop oms -all
$AGENT_HOME/bin/emctl stop agent

Final Notes

What’s up with all the periods (.) all over this post? I fought WordPress the whole time to keep my screenshots from overlapping section headers and each other. Eventually I gave up and used periods to occupy white space as a placeholder.
A special note of acknowledgment to sysadmin guy who made me understand why no browser could find the login page (the non-standard SSL port issue noted above).   I literally did the install twice, thinking I’d screwed something up or missed some step.  In the end, I probably had it right the first time, but didn’t see any way to reach the login page so reverted and reinstalled all pieces.
The Weblogic server admin screens have about a million tabs, checkboxes, radio buttons and text-entry areas for parameters.  There is a learning curve there:

There's tabs on the tabs!


I feel compelled to add an addendum to this posting because of a few things I’ve noticed after running 11g a few days.  First, it consumes more memory; we had to throw another gig at it (database footprint unchanged).  Second, after rebooting the host, we noticed it trying to start processes; it slipped a gcstartup file in the init.d directory.  This script basically checks your oratab file and tries to start whatever homes it finds in there.  Fine and good (I guess)  but really of no value for me.  Having the Weblogic server and the OMS “running” before the database isn’t really useful to me.

Useful URLs and other eratta

main EM login screens:

Weblogic console

It appears that some of the logs aren’t automatically rotated in $ORACLE_INSTANCE/WebtierIH1/diagnostics/logs/OHS/ohs1; I’d double check it if I were you.  My mod_wl_ohs.log was 500Mb.  I fixed this by gzipping the old mod_wl_ohs.log at the end of my stop enterprise manager script; every time I shut down EM, it will gzip it and eventually age 2 weeks and be caught by:

In addition, you probably want to consider a cron job to delete old access/error/em_upload_https logs; something like this is pretty vanilla in the crontab (delete logs older than about 2 weeks):

/usr/bin/find /u01/oem/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1 -mtime +14 -exec rm -rf {} \;

May 6, 2010

Fun with adpatch

Filed under: 11i, Utilities — kkempf @ 10:21 am

Maintenance Time

This weekend I have a maintenance window (of course, on Mother’s Day) and hadn’t actually applied the approved patches in about 6 months.  Since I had the luxury of time to perform a “dry run” in an 11i sandbox environment, I went ahead and started with the patching.  The environment was a recent clone of production, so I didn’t anticipate any problems, but yet they found me.  I thought the nature of the error and the fix might be of interest.

Could not create the Java virtual machine

Yeah, what fun.  Shortly into the adpatch run, I hit the following error:

Running adjcopy.class:


adjava -mx512m -nojit @/u02/appmis/misappl/admin/MIS/out/apps.cmd


AutoPatch error:

ERROR  Running adjopy.class


AutoPatch error:

Error updating master archive

The Fix

What a mess.  The log pointed to the solution:

Unrecognized option: -nojit
Could not create the Java virtual machine.

A little (somewhat creative) digging on Metalink and I stumbled across ID 301542.1 which was a similar issue.
My $APPL_TOP/admin/adovars.env file had a bad environment variable:

# Set java program used by AD
ADJVAPRG=/opt/jdk/bin/jre (change jre to java)

To complete the fix, simply reinstantiate your environment (typically $APPL_TOP/APPSORA.env)

Minimum Baselines for 11i Extended Support

Incidentally, the patching I’m doing at this point is in response to new minimum baselines (wouldn’t have been a lot easier?) required for extended support for 11i beyond November 2010.

Blog at