Kevin Kempf's Blog

August 9, 2016

How to Migrate an Oracle database host in R12.2 using LVM

Filed under: Oracle, Oracle Linux — kkempf @ 9:40 am

Painting the backdrop

I realize this post is somewhat specific to your setup, but I believe it may hold value to some folks out there so I thought I’d formalize it.  In my case, I have a rather large (1TB+) database residing on Oracle Linux (RHEL) 5 which serves as the back end for my EBS 12.2 environment.  We do not use ASM nor RAC, but do use Linux LVM (logical volume manager) to make growing disks easier.  The disk itself is on a SAN in the data center, so in my case this process involves some assistance from the systems/network folks.

I’m moving from OL5 to OL6 because support for OL5 is running down.  Let’s face it: the database works great on OL5, and I have no compelling reason to migrate it, but this is part of our lives: upgrades for the sake of upgrading.

For the purposes of this post, I’ll simply use oldhost as the OL5 hostname, and newhost as the OL6 hostname for the database, and appshost as the hostname for the applications tier.

In the olden days

Oracle used to have a bonafide methodology for migrating the database tier, in 11i it’s spelled out under DOC 338003.1.  It made sense; it said to use the tech stack on the front end to tell the application tier that there’s a new host for the database.

The R12.2 solution

I opened an SR to get the equivalent document for 12.2 and the analyst basically said “Follow Doc ID 1968231.1 to use logical hostnames, you can sort of use autoconfig but we don’t support it”.  I don’t know if that’s really the best answer, but it was all I had to go with.  I had been planning to use DNS as a safety net, not the primary vehicle of changing the database hostname.  But apparently it’s the only vehicle now, despite the fact that I enter hostname on the front end in the context file.  While I think this is a terrible answer, that’s not the point of my post so I’ll let it go.

LVM Setup (oldhost)

In my environment, I set up my lvm volume groups like this:

Data1 - product datafiles for EBS
Data2 - more product datafiles for EBS
Archivelogs - archivelog destination
Redo - online redo
System - system/sysaux datafiles
RDBMS - Oracle database installation (binaries)
As you can imagine, the files contained in these groups are pretty large; well over a terabyte.  In an ideal situation, I’d take the database down cold and simply rsync the files to the new host in the same location, then crank everything up.  But that would take hours I don’t have, so I went a different route.  It’s worth noting, that on OL5, my disks on oldhost are ext3, and OL6 delivers ext4.  Since I’m moving the disks “as is”, I’m getting ext3 filesystems on newhost, the OL6 server.  It’s compatible, and something I just have to live with.
I feel like I should mention at this point that probably the very first step was to build newhost as an OL6 environment with appropriate cpu and memory.

LVM Prep (oldhost)

  • The first step is to shut everything down, obviously, front and back end on oldhost and appshost.
  • Next, I unmount all the volumes pertaining to the environment (in this case, dev).  For my environment, each of these corresponds to an LVM volume group
    • umount /u01/appdev
    • umount /u03/appdev
    • umount /u04/appdev
    • umount /u05/appdev
    • umount /u06/appdev
    • umount /usr/local/oracle/archive
  • Set all the volume groups to inactive
    • vgchange -an Data2
    • vgchange -an Data1
    • vgchange -an Archivelogs
    • vgchange -an Redo
    • vgchange -an System
    • vgchange -an RDBMS
  • Export the volume groups
    • vgexport Data2
    • vgexport Data1
    • vgexport Archivelogs
    • vgexport Redo
    • vgexport System
    • vgexport RDBMS

Disk manipulation

At this point, the volumes can be safely manipulated by the disk admins.  What this entails will vary greatly based on  your datacenter, and is outside the scope of linux so I’m not going to detail it here.  The gist of it is that your admin needs to remove the disks/virtual disks from the old host and install them on the new one by whatever means is appropriate.  I apologize in advance if I’m not saying this part correctly.

Adding the disks (newhost)

The first thing to try (assuming your new host received the disks “hot”) is simply


They may just show up.  If not, you can try rescanning the scsi host:

ls /sys/class/scsi_host/ | whileread host ; do echo "- - -"> /sys/class/scsi_host/$host/scan ; done
-or this-
echo "- - -"> /sys/class/scsi_host/(host#, hit tab or guess)/scan
-for example this may turn into:

 echo “- – -” > /sys/class/scsi_host/host0/scan

echo “- – -” > /sys/class/scsi_host/host1/scan

echo “- – -” > /sys/class/scsi_host/host2/scan

Then issue
fdisk -l
and it’s worth noting you can watch /var/log/messages for the system to recognize new disks being added.  When all else fails, a reboot has never failed to get all disks recognized.
At this point, pvscan should show all your volume groups

Final steps (newhost)

It’s kind of the opposite of the prep steps:

  • Import the volume groups
    • vgimport Data2
    • vgimport Data1
    • vgimort Archivelogs
    • vgimport Redo
    • vgimport System
    • vgimport RDBMS
  • Active the volume groups
    • vgchange -ay Data2
    • vgchange -ay Data1
    • vgchange -ay Archivelogs
    • vgchange -ay Redo
    • vgchange -ay System
    • vgchange -ay RDBMS
  • Mount the disks (and put them in /etc/fstab so they survive a reboot!)
    • mount /dev/mapper/RDBMS-Dev /u01/appdev
    • mount /dev/mapper/System-Dev /u03/appdev
    • mount /dev/mapper/Redo-Dev /u04/appdev
    • mount /dev/mapper/Data1-Dev /u05/appdev
    • mount /dev/mapper/Data2-Dev /u06/appdev
    • mount /dev/mapper/Archivelogs-Logs /usr/local/oracle/archive

Final Steps

In my case, I had to do 3 final steps to comply with Oracle’s document and make everything work:

  • change /etc/hosts on appshost to explicitly refer to the newhost as oldhost… just in case
  • change listener.ora on newhost to be the new hostname on the database ($TNS_ADMIN)
  • add DNS record to change calls to old host to new one
  • Start up the database (newhost)

Parting Shots

This method took me about 30 minutes.  If the database were smaller, I’d simply rsync the ext3 filesytems to new disks built as ext4 on newhost.



March 20, 2015

Installing Oracle Linux on your PC

Filed under: Oracle Linux — kkempf @ 9:16 am


A break from the usual

I’m not a big fan of Windows as a PC operating system in the workplace, at least for what I do every day.  For the past 4-5 years I’ve been running Ubuntu and I recently upgraded to their version 14.  It was fine, but being a Debian release it was still clumsy and some things were very awkward/required hacks to make work (for example, the Oracle client).  So I decided to try something new.  The only requirement was it had to pass some arbitrary usability threshold in my head, and be installable from a USB key because I don’t have a DVD drive.

Fedora 21

The screenshots looked promising, but in the end this was a debacle.  I followed the instructions on their website, which included how to create a bootable USB drive.  The download took forever for the smallish install, though when complete I had a bootable USB drive (after using the creation utility they recommended).  I booted to it, but once in the GUI it was really acting up.  I have two monitors, and the GUI kept going on and off between the two.  It made installing it really hard, and by the time I got to the disk partitioning part and it said I didn’t have enough room, I was done with this release.  To be fair, the PC had existing partitions on it, and they were encrypted, this may have been the issue.  I can’t believe this is a show stopper for an installer, but something wasn’t right.

Oracle Linux 6.6

Actually figuring out how to get the installer going is a bit tricky, and the essence of this post, but once installed I’m feeling like it’s where I’ll be for awhile.  I’m assuming you’re doing this from Windows; if that’s not the case, you will have to read up on how to create the USB drive from your operating system.  Here’s the path to happiness:

  • Get an 8GB or larger USB drive
  • Download the install
    • Go to Oracle E-Delivery
    • Sign away your life, and select Oracle Linux 6 Update 6 Media Pack for x86_64 (64 bit) or 32-bit I suppose if you’re into that
    • Here you’ll see 5 downloads, but we’re only concerned with 2
      • Oracle Linux Release 6 Update 6 for x86_64 (64 Bit)
      • Oracle Linux Release 6 Update 6 Boot iso image for x86_64 (64 bit)
    • OLdownloads
    • Now head over to Linux Live USB creator, and install it as a Windows application
      • Point this application at the smaller Boot ISO image (226MB one), select your USB drive as the target, hit go, and now your USB key is bootable
    • This is the part which tripped me up and caused some grief: after creating your boot key, copy the big raw ISO file (3.7GB) to the USB key also
      • After you configure the basics in the installer GUI, it needs this ISO to actually do the install
      • If you don’t do this, you get an error like: Missing ISO image The installer has tried to mount image #1, but cannot find it on the hard drive.
        • It seems possible that you could have this ISO on another USB drive and point it to it, but I had no success with it
        • It appears that the installer will not recognize a dynamic mount of a USB key in the middle of the process, so you can’t switch out keys or add a 2nd one
          • I even tried to put two USB keys in, booting from the small image and then pointing to the big ISO (on /dev/sdc) when the installer got confused but it didn’t work either
    • At this point, you just put the USB key into your PC, ensure USB is in the boot sequence at the bios level, and start the PC
    • Early in the installer, the TUI (text user interface: a big red box) asks where something is located.  I’m sorry I can’t be more specific, nor do I remember what it was looking for.  But I have the answer in general:
      • My PC is simple: one hard drive, and the USB drive
        • Linux sees these as sda1, sda2, sda3 (my hard drive, with 3 partitions) and sdb1 (my usb drive)
      • What it needs to be told is sdb1 (your USB drive) and then it chugs along happily


I feel obligated to say that you have a strong chance of wiping out whatever is on your PC if you don’t know what you’re doing.  In my case, that was the whole idea, so the risk was low.  If your goal is to create a live USB operating system, or dual boot OS, that’s beyond the scope of what I’m talking about, and requires more/different steps.


Let’s assume you’ve installed the OS, and you’re booted to the GUI now.  You need to add the public yum repos to get updates

  • $ su –
  • # export http_proxy=  (skip if you don’t have a proxy, obviously requires some tweaking)
  • # wget -P /etc/yum.repos.d/
  • # yum update

Customizing Your Environment

It takes time to get all the pieces working; your requirements will be different from mine.  I’ll give you a few obvious ones

  • Java (JDK)
    • yum remove java-1.6.0-openjdk
    • yum remove java-1.7.0-openjdk
    • go download Oracle java
    • rpm -ivh *.rpm (it puts it in /usr/java/jdk…)
  • Java plugin for Firefox so you can run Forms in the Ebusiness Suite
    • $ cd ~/.mozilla/plugins
    • $ ln -s /usr/java/jdk1.7.0_67/jre/lib/amd64/   (substitute your version of java where I’ve listed jdk1.7.0_67)
  • SQL Developer
    • Go download the most recent version
    • rpm -Uvh sqldeveloper*rpm (it installs in /usr/local/bin)
    • $ cd /usr/local/bin/
    • ./sqldeveloper
    • when prompted, point it to your JDK installation
      • Type the full pathname of a JDK installation (or Ctrl-C to quit), the path will be stored in /home/kkempf/.sqldeveloper/4.0.0/product.conf
      • enter: /usr/java/jdk1.7.0_67 (obviously change your version to match)

You’re on your feet

The rest is up to you


March 9, 2015

R12.2 Apache won’t start up

Filed under: Linux, Oracle Linux, R12.2, Weblogic — kkempf @ 9:45 am



I had to bounce my R12.2 non-production front end this weekend to switch kernels around.  I restarted services, but admit I didn’t pay much attention to them as it wasn’t PROD.  We run multiple instances of R12.2 on different port pools but on one host (well 2, one for the databases, one for the application servers). We do this not because I like it, but because we don’t have infinite budgets.  Regardless, I get a complaint that our audit environment isn’t working.  Sure enough, the login is unavailable, and Weblogic says the web tier process EBS_web_AUDIT is down.

Identify the problem: To the command line!

I try to simply start it in Weblogic but it errors out, and there’s far too many log links to click for this admin to stay in a GUI.   I run start and get:

You are running version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance … exiting with status 204

Follow the bouncing ball tells me to check my log file: $INST_TOP/logs/appl/admin/log/adapcctl.txt, and here’s what it tells me:

ias-instance id=EBS_web_AUDIT_OHS1

--> Process (index=1,uid=1810789619,pid=11169)
  failed to start a managed process after the maximum retry limit

03/09/15-09:52:40 :: exiting with status 204
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address [::]:10059
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:32] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] (98)Address already in use: make_sock: could not bind to address
[2015-03-09T09:38:34.0590-04:00] [OHS] [INCIDENT_ERROR:20] [OHS-9999] [worker.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] no listening sockets available, shutting down
[2015-03-09T09:38:34.0591-04:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [host_id: (myhostname).com] [host_addr: (myipaddr)] [pid: 10527] 
[tid: 140534409107264] [user: oraaudit] [VirtualHost: main] Unable to open logs

Closing in

So it appears that my user (oraaudit) can’t get a port it wants (10059), and after a little hunting on Google I find the right syntax. I wanted to use netstat, but I couldn’t figure out the PID and lsof made it really easy:

# lsof -i:10059
java    13815 oratrain  305u  IPv6  82113      0t0  TCP (myhostname).com:10059->(myhostname).com:rds2 (ESTABLISHED)

Take down the suspect

It’s some random java process from another instance (Train). See ya.

# kill -9 13815
# lsof -i:10059

Restart Apache start

You are running version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ... exiting with status 0

Nothing to see here

This is something most apps DBAs have seen at some time or in some form. The path to follow changes, but in the end the fix is usually similar to this. Knowing with certainty that it’s OK to kill the blocking PID is always a delicate task. There’s absolutely no way you could have fixed this from within the Weblogic admin console that I can think of, short of going into the other environment and shutting it down completely.

February 14, 2012

Switching from Redhat Linux to Oracle Linux in about 5,000 easy steps

Filed under: 11g, Oracle Linux — kkempf @ 4:15 pm

Wayback When

I remember being at Oracle Open World when Larry Ellison unveiled Oracle Enterprise Linux (OEL, which is now just Oracle Linux, or OL).    I think I even have a foam Oracle penguin and maybe even a t-shirt somewhere.  I was trying to understand why, as a loyal RedHat customer, I’d ever consider switching over to the “dark side”.  I even remember laughing to a colleague “Who in their right mind would want Oracle to support their Linux environment, they can’t even support their database?”.

Warming up to the idea

I thought it might be useful to note, for a moment, my experience level with Linux.  I’ve been using some flavor of Unix/Linux in various workplaces since 1999.  I hold an RHCSA (Red Hat Certified System Administrator) cert and I hope to pursue the next level, RHCE, within a few months.  Why am I mentioning this?  To demonstrate that I’m not an Oracle fan boy and if anything I have more of an inclination to run Red Hat than any other flavor of Linux at the moment.

As I mentioned in another post, we recently did a fairly major hardware refresh in our datacenter.  My 11i production database is currently a physical machine, and the box was available to me to tinker with prior to migration.  We’d even gotten to the point where we’d installed Red Hat Enterprise Linux (RHEL) 5.7 before I got the crazy idea to take a 2nd look at Oracle Linux.  I have no idea what the adoption rate is for OL, but they claim 8,000 customers on their information page.  I don’t know if that’s a lot or not.

What initially drove me to even consider Oracle Linux was not cost, but rather a series of really bad support tickets I had with Red Hat.  Unrelated, system service requests where Red Hat support went 0 for 3.  Why did my system lock up and the kernel panic?  Redhat: No idea. Twice.  The second time with really good logging enabled.   Then I had an issue where the system CPU (as opposed to user CPU) time was crazy high – in some sar reports as much as 20% of the total CPU usage.  Why is it so high?  Redhat: No idea.  At this point the little light bulb went off in my head.  I can pay less than half as much for bad support from Oracle.  And that’s really a pessimistic view.  In truth, there are some actual advantages to running Oracle on Oracle Linux, especially when you consider the Oracle Unbreakable Enterprise Kernel (UEK).

The Flavors of Oracle Linux

You can go, right now and pull Oracle Linux and install it on you machine.  You will, of course not be able to open a ticket (through my Oracle Support).  Basically, there’s 3 flavors of support with different cost levels:

  • Network Support: patches and updates only (this flavor was not offered by sales, not sure about this level)
  • Basic Support: add “complete Linux server lifecycle management, cluster software” (this is what we have)
  • Premier Support: add ksplice

I think I should take a minute to define a few things more clearly, as I wish my sales team would have:

  • Unbreakable Enterprise Kernel (UEK): Oracle’s home-grown kernel, available at any level of support.  From what I can tell, this kernel especially optimizes what Oracle perceives as deficiencies in the Red Hat kernel’s ability to handle big multi-processor (SMP) machines.
  • ksplice: a zero-downtime kernel patcher.  This is available only if you buy Premier Support and only if you are running the Oracle Unbreakable Enterprise Kernel (UEK)
  • Red Hat compatible kernel: the kernel Oracle creates based upon the same open-source feed Red Hat gets.  You can run Oracle Linux using either the UEK or the RH Compatible kernel, selected at boot time via grub
  • The Physical Address Extension (PAE) 32-bit kernel is available for Oracle Linux 5 (x86).  This was important to me because I run the 11i front end (has to be 32-bit!) on the PAE kernel which gets around the 4gb RAM limit imposed by 32-bit architecture.

Things I wish I’d known (or researched better) in retrospect

Before I dig into the “how” part of how to convert your Red Hat 5 machine to Oracle Linux, I thought I’d tell you more about the areas in which I have buyer’s remorse.

  • UEK.  This was one of the big upsells for me to Oracle Linux.  I planned to cut over to the Red Hat compatible kernel, then begin regression testing the “super” UEK kernel.  Except for one thing.  We’re a VMWare shop.  It’s not certified to run as a VMWare guest.  Until that changes, UEK is DOA in our datacenter.  If UEK is DOA, then so is ksplice.  If ksplice is DOA, then that means I definitely don’t need premier support.
  • Unbreakable Linux Network (ULN) pales in comparison to the Redhat Network.  You cannot release patches for update to the servers like you can with the Red Hat network.  Basically you can see what systems are registered, if they need updates, and what your CSI is.  Beyond that, you can see what versions of what packages are available in what release.
  • Managing servers is rather manual.  There is some promise that Enterprise Manager may make server management easier, but I can’t verify this.

Switching to Oracle Linux

I’m assuming at this point, that you’re running RHEL 5.7.  While certainly you could be running something else, you’d have to be smart enough to make some changes in the steps I will outline.   RH6 isn’t certified with much, as far as I can tell.

This outline is derived from a lot of disparate sources.  In fact, I wish Oracle would have one, good, thorough document to walk me through all this, but they don’t appear to.

  • Go get the appropriate up2date packages from Oracle, based on the RH release you’re running, and land the RPMs somewhere on your server.
  • Install the up2date RPMs
    • rpm -Uvh up2date*rpm
    • Import the GPG Key for RPMs
      • rpm –import /usr/share/rhn/RPM-GPG-KEY
    • Save yourself a headache, update the RHN uuid of your server
      • # /usr/bin/uuidgen -r
        • e949dadc-c182-3ec2-9de7-44ag8a0d2bea
        • vi /etc/sysconfig/rhn/up2date-uuid
          • confirm the line rhnuuid matches the key you just generated
          • if not, pound it out and replace it:
            • rhnuuid=e949dadc-c182-3ec2-9de7-44ag8a0d2bea
    • Run the Oracle Registration TUI (Text User Interface)
      • up2date-nox – – register     (this actually launches the TUI) ** for OL6, it’s bundled now just run uln_register
      • If you have a proxy server between you and the internet
        • export http_proxy=<your port>
        • edit /etc/yum.conf for future ease
          • under [main] add this line
          • proxy=<your port>
          • Import the GPG Key for RPMs
        • Here’s screenshots from the TUI

      It always seems to say this...

      Hit next to waive all your rights

      Not sure I like hard coding my ULN login

      Hardware gathered information


  • Update your Repos and packages
    • cd /etc/yum.repos.d
    • rename your existing .repo files
      • mv rhel-source.repo rhel-source.repo.old
      • mv rhel-debuginfo.repo rhel-debuginfo.repo.old
      • note that the .repo suffix is what signifies that a repo file is “active”
    • use wget to fetch the oracle repos
    • edit the public-yum-el5.repo to activate (enable) the appropriate subscription channels
      • [ol5_u7_base]
      • enabled=1
    • update your packages
      • # up2date -i yum-rhn-plugin
      • # yum update
    • #yum install kernel
    • #yum install kernel-uek (optional, install the unbreakable kernel)
    • #yum install oracle-linux (optional Oracle packages, pair with UEK I believe)
  • reboot to new kernel
    • check /etc/grub.conf to ensure it is to your satisfaction
    • #reboot -i to restart the host, select the appropriate kernel when the grub menu option appears

First Impressions

So what I failed to mention in my original post were impressions about the migration.  We’ve been running Oracle Linux 5.7 for 3 weeks now.   Aside from the branding/logo changes  (A penguin in Oracle armor instead of the Red Hat shadow man) I don’t see much difference at all.  Not that I’d really have any reason to expect to.

Since this OS upgrade coincided with a  hardware upgrade, I feel it would be unfair to speculate on performance improvement.  Meaning, I suspect anything works better on the latest CPU and hardware architecture.  Suffice it to say, after a short period of fretting about the new OS, I just don’t monitor it anymore.  It works fine.

My concern about how much more manual the Oracle Linux experience hasn’t changed much.  I now understand that I could create a local RPM mirror, which would be updated daily through Oracle Enterprise Manger 11g, and thus through Enterprise Manager, I could push or release updates to my Oracle Linux servers.  That’s all fine and good, but that’s just one more layer of complication I’d rather not have to deal with.  I may go that route eventually, but since I’m comparing apples to apples, I simply say:  “I don’t have to do that with Red Hat”.

One impression I got throughout the whole conversion process was a general disjointedness from Oracle.  One of the main reasons I posted this blog entry was because the whole process, from information (sales) to install is all over the place.  I mean literally, all over the place.  The steps above are provided in a complete manner nowhere that I am aware of.  There’s a document here about how to wget the repo’s, a document there about how to deal with duplicate RHN UUID’s, another page to download the up2date RPMs, another page telling you how to register with the Oracle Linux network.  I suppose I can’t expect Oracle to advertise that the UEK kernel isn’t certified for VMWare, but it’s important, and I’d rather have learned that up front than on my own trying to boot a DEV VM into UEK.   Support told me to use up2date to update my packages, but when I put that in my (original) blog posting here, I was corrected in the comments and told of yum-rhn-plugin (noted in the steps above).   I looked back to see what Open World I was at when Oracle Linux was announced:  it was Fall of 2006.  I would think that after 5 years, this would be a bit more refined.

All that said, my final word on this is that while frustrating and non-intuitive, the migration to Oracle Linux has been fine.  It’s too early to say I recommend it, but I will say at this point, that I’m not regretting moving to it, and I think it’s worth a look.  Even if you’re from the old-school Sun/RedHat/AIX/HPUX environments like me.

Blog at