Switching from Redhat Linux to Oracle Linux in about 5,000 easy steps

Wayback When

I remember being at Oracle Open World when Larry Ellison unveiled Oracle Enterprise Linux (OEL, which is now just Oracle Linux, or OL).    I think I even have a foam Oracle penguin and maybe even a t-shirt somewhere.  I was trying to understand why, as a loyal RedHat customer, I’d ever consider switching over to the “dark side”.  I even remember laughing to a colleague “Who in their right mind would want Oracle to support their Linux environment, they can’t even support their database?”.

Warming up to the idea

I thought it might be useful to note, for a moment, my experience level with Linux.  I’ve been using some flavor of Unix/Linux in various workplaces since 1999.  I hold an RHCSA (Red Hat Certified System Administrator) cert and I hope to pursue the next level, RHCE, within a few months.  Why am I mentioning this?  To demonstrate that I’m not an Oracle fan boy and if anything I have more of an inclination to run Red Hat than any other flavor of Linux at the moment.

As I mentioned in another post, we recently did a fairly major hardware refresh in our datacenter.  My 11i production database is currently a physical machine, and the box was available to me to tinker with prior to migration.  We’d even gotten to the point where we’d installed Red Hat Enterprise Linux (RHEL) 5.7 before I got the crazy idea to take a 2nd look at Oracle Linux.  I have no idea what the adoption rate is for OL, but they claim 8,000 customers on their information page.  I don’t know if that’s a lot or not.

What initially drove me to even consider Oracle Linux was not cost, but rather a series of really bad support tickets I had with Red Hat.  Unrelated, system service requests where Red Hat support went 0 for 3.  Why did my system lock up and the kernel panic?  Redhat: No idea. Twice.  The second time with really good logging enabled.   Then I had an issue where the system CPU (as opposed to user CPU) time was crazy high – in some sar reports as much as 20% of the total CPU usage.  Why is it so high?  Redhat: No idea.  At this point the little light bulb went off in my head.  I can pay less than half as much for bad support from Oracle.  And that’s really a pessimistic view.  In truth, there are some actual advantages to running Oracle on Oracle Linux, especially when you consider the Oracle Unbreakable Enterprise Kernel (UEK).

The Flavors of Oracle Linux

You can go, right now and pull Oracle Linux and install it on you machine.  You will, of course not be able to open a ticket (through my Oracle Support).  Basically, there’s 3 flavors of support with different cost levels:

• Network Support: patches and updates only (this flavor was not offered by sales, not sure about this level)
• Basic Support: add “complete Linux server lifecycle management, cluster software” (this is what we have)
• Premier Support: add ksplice

I think I should take a minute to define a few things more clearly, as I wish my sales team would have:

• Unbreakable Enterprise Kernel (UEK): Oracle’s home-grown kernel, available at any level of support.  From what I can tell, this kernel especially optimizes what Oracle perceives as deficiencies in the Red Hat kernel’s ability to handle big multi-processor (SMP) machines.
• ksplice: a zero-downtime kernel patcher.  This is available only if you buy Premier Support and only if you are running the Oracle Unbreakable Enterprise Kernel (UEK)
• Red Hat compatible kernel: the kernel Oracle creates based upon the same open-source feed Red Hat gets.  You can run Oracle Linux using either the UEK or the RH Compatible kernel, selected at boot time via grub
• The Physical Address Extension (PAE) 32-bit kernel is available for Oracle Linux 5 (x86).  This was important to me because I run the 11i front end (has to be 32-bit!) on the PAE kernel which gets around the 4gb RAM limit imposed by 32-bit architecture.

Things I wish I’d known (or researched better) in retrospect

Before I dig into the “how” part of how to convert your Red Hat 5 machine to Oracle Linux, I thought I’d tell you more about the areas in which I have buyer’s remorse.

• UEK.  This was one of the big upsells for me to Oracle Linux.  I planned to cut over to the Red Hat compatible kernel, then begin regression testing the “super” UEK kernel.  Except for one thing.  We’re a VMWare shop.  It’s not certified to run as a VMWare guest.  Until that changes, UEK is DOA in our datacenter.  If UEK is DOA, then so is ksplice.  If ksplice is DOA, then that means I definitely don’t need premier support.
• Unbreakable Linux Network (ULN) pales in comparison to the Redhat Network.  You cannot release patches for update to the servers like you can with the Red Hat network.  Basically you can see what systems are registered, if they need updates, and what your CSI is.  Beyond that, you can see what versions of what packages are available in what release.
• Managing servers is rather manual.  There is some promise that Enterprise Manager may make server management easier, but I can’t verify this.

Switching to Oracle Linux

I’m assuming at this point, that you’re running RHEL 5.7.  While certainly you could be running something else, you’d have to be smart enough to make some changes in the steps I will outline.   RH6 isn’t certified with much, as far as I can tell.

This outline is derived from a lot of disparate sources.  In fact, I wish Oracle would have one, good, thorough document to walk me through all this, but they don’t appear to.

• Go get the appropriate up2date packages from Oracle, based on the RH release you’re running, and land the RPMs somewhere on your server.

• Install the up2date RPMs
  • rpm -Uvh up2date*rpm

• Import the GPG Key for RPMs
  • rpm –import /usr/share/rhn/RPM-GPG-KEY
• Save yourself a headache, update the RHN uuid of your server
  • # /usr/bin/uuidgen -r
    • e949dadc-c182-3ec2-9de7-44ag8a0d2bea
    • vi /etc/sysconfig/rhn/up2date-uuid
      • confirm the line rhnuuid matches the key you just generated
      • if not, pound it out and replace it:
        • rhnuuid=e949dadc-c182-3ec2-9de7-44ag8a0d2bea
• Run the Oracle Registration TUI (Text User Interface)
  • up2date-nox – – register     (this actually launches the TUI) ** for OL6, it’s bundled now just run uln_register
  • If you have a proxy server between you and the internet
    • export http_proxy=http://username:password@yourproxy.domain.com:<your port>
    • edit /etc/yum.conf for future ease
      • under [main] add this line
      • proxy=http://username:password@yourproxy.domain.com:<your port>
      • Import the GPG Key for RPMs
  • Here’s screenshots from the TUI
  

  It always seems to say this...

  

  Hit next to waive all your rights

  

  Not sure I like hard coding my ULN login

  

  Hardware gathered information

  

  RPMs

• Update your Repos and packages
  • cd /etc/yum.repos.d
  • rename your existing .repo files
    • mv rhel-source.repo rhel-source.repo.old
    • mv rhel-debuginfo.repo rhel-debuginfo.repo.old
    • note that the .repo suffix is what signifies that a repo file is “active”
  • use wget to fetch the oracle repos
    • wget http://public-yum.oracle.com/public-yum-el5.repo
  • edit the public-yum-el5.repo to activate (enable) the appropriate subscription channels
    • [ol5_u7_base]
    • …
    • enabled=1
  • update your packages
    • # up2date -i yum-rhn-plugin
    • # yum update
  • #yum install kernel
  • #yum install kernel-uek (optional, install the unbreakable kernel)
  • #yum install oracle-linux (optional Oracle packages, pair with UEK I believe)
• reboot to new kernel
  • check /etc/grub.conf to ensure it is to your satisfaction
  • #reboot -i to restart the host, select the appropriate kernel when the grub menu option appears

First Impressions

So what I failed to mention in my original post were impressions about the migration.  We’ve been running Oracle Linux 5.7 for 3 weeks now.   Aside from the branding/logo changes  (A penguin in Oracle armor instead of the Red Hat shadow man) I don’t see much difference at all.  Not that I’d really have any reason to expect to.

Since this OS upgrade coincided with a  hardware upgrade, I feel it would be unfair to speculate on performance improvement.  Meaning, I suspect anything works better on the latest CPU and hardware architecture.  Suffice it to say, after a short period of fretting about the new OS, I just don’t monitor it anymore.  It works fine.

My concern about how much more manual the Oracle Linux experience hasn’t changed much.  I now understand that I could create a local RPM mirror, which would be updated daily through Oracle Enterprise Manger 11g, and thus through Enterprise Manager, I could push or release updates to my Oracle Linux servers.  That’s all fine and good, but that’s just one more layer of complication I’d rather not have to deal with.  I may go that route eventually, but since I’m comparing apples to apples, I simply say:  “I don’t have to do that with Red Hat”.

One impression I got throughout the whole conversion process was a general disjointedness from Oracle.  One of the main reasons I posted this blog entry was because the whole process, from information (sales) to install is all over the place.  I mean literally, all over the place.  The steps above are provided in a complete manner nowhere that I am aware of.  There’s a document here about how to wget the repo’s, a document there about how to deal with duplicate RHN UUID’s, another page to download the up2date RPMs, another page telling you how to register with the Oracle Linux network.  I suppose I can’t expect Oracle to advertise that the UEK kernel isn’t certified for VMWare, but it’s important, and I’d rather have learned that up front than on my own trying to boot a DEV VM into UEK.   Support told me to use up2date to update my packages, but when I put that in my (original) blog posting here, I was corrected in the comments and told of yum-rhn-plugin (noted in the steps above).   I looked back to see what Open World I was at when Oracle Linux was announced:  it was Fall of 2006.  I would think that after 5 years, this would be a bit more refined.

All that said, my final word on this is that while frustrating and non-intuitive, the migration to Oracle Linux has been fine.  It’s too early to say I recommend it, but I will say at this point, that I’m not regretting moving to it, and I think it’s worth a look.  Even if you’re from the old-school Sun/RedHat/AIX/HPUX environments like me.

The future face of My Oracle Support?

Reminds me of the good old days

Good News

It seems I’m going to have time to finish up on several half-baked posts today, but I had to put this quick one in.  Anyone who knows my blog knows that since its launch in November of 2009, I have thoroughly despised the “New” My Oracle Support (support.oracle.com).  It’s wholly unusable, running under a buggy flash interface.  In fact, I’ve often referred to it as the world’s slowest support site.  Well good news, folks, the future is bright.  I tried supporthtml.oracle.com after their update, and it’s awesome!  I can hardly believe I wrote that.  It’s using Oracle’s own application development framework, and it’s FAST, CLEAN and the drop down selections and text boxes are bug-free!  Try it, you’ll never go back to that other thing.  Support tells me (rumor?) that this is where support.oracle.com is going, no timeline.

Annoying Agent Problems

New PROD RDBMS host

We run the main ERP database on a physical machine; I’d love to virtualize, and probably will oneday soon, but we couldn’t get to vSphere 5 (required because of CPU count) before the hardware refresh.  So we migrated Oracle to a brand new spiffy Dell R610 and it’s smokin’ fast.  The process was what is known as physical to physical (P to P) server migration, and it went as well as can be expected.  There was a bit of LVM manipulation required at the OS level, but for the most part we managed to bumble our way through it.

In the process of migrating to the new physical machine (from a rather reliable but ancient IBM blade server, incidentally), I took the plunge and cut over our production database hosts from RedHat 5.7 to Oracle Linux 5.7.  I say take the plunge, but in truth the risk was a known entity: it’s a RedHat compatible kernel.  What sparked this decision was 2 miserable, unresponsive tickets with RedHat support about high system CPU on my application server.  Not to be funny about it, but if I can pay about half as much to get bad support, perhaps better, from Oracle, why wouldn’t I?  Incidentally, the process of migrating from RH5 to OL5 (formerly OEL5, know they just call it OL5) is something which I will put in a detailed post shortly.

Angry Agents

After bringing up the database on new hardware, the agent would not communicate with the OMS:

The Oracle Management Server (OMS) has blocked this agent because it has either been reinstalled or restored from a filesystem backup.  Please click on the Agent Resynchronization button to resync the agent.

Your agent is hopelessly confused

When I “clicked on the agent resynchronization button to resync the agent” if failed with an error.   You can bet your last dime, however, the first thing Oracle Support asked in my ticket?  “Did you try clicking the agent resynchronization button?”.    This is the subsequent message (see below as well):

Agent Operation completed with errors.  For those targets that could not be saved, please go to the target’s monitoring configuration page to save them.  All other targets have been saved successfully.  Agent has not been unblocked.

Error communicating with agent.  Exception message – oracle.sysman.emSDK.emd.comm.CommException: IOException in reading Response :: Connection reset

Blocked Agents

If there’s one thing I hate, it’s blocked agents.  You bet I tried to unblock it, then resync it.  I tried command line updates like emctl status agent, emctl upload agent, emctl unsecure agent, emctl secure agent.  You name it.  Nada.

The Fix

I stumbled across Document ID 1307816.1 while my analyst was busy asking me things like “can you upload your log files”.  In the end, as the sysman user, I ran this against your EM database:

exec mgmt_admin.cleanup_agent(‘problemhost.domainname.com:3872’);

After that my agent was happy, could talk to the OMS, and life was good.