Kevin Kempf's Blog

May 8, 2010

EM 11g Grid Control Install

Filed under: Enterprise Manager, Oracle — kkempf @ 7:46 pm

Preparation

For no reason other than I could, I recently decided to undertake installing the new Oracle Enterprise Manager 11g Grid Control (technically, upgrade my 10.2.0.5 environment).   For those not in the know, it was released about 2 weeks ago, and one major change is that it utilizes Weblogic server instead of the old iAS (this is the Fusion model).   In fact, there is no application server included in the EM 11g download; the installer assumes you have a Middleware home with the appropriate version of Weblogic server already installed.   The first thing I did in preparation for the upgrade  was to contact our Oracle techstack sales guy and ensure that I was allowed to install Weblogic server.  While I’m fully licensed to run EM 10.2.0.5 with tuning and diagnostics packs, I didn’t want to make any assumptions about Oracle giving up a new application server for free.  After confirming my legality from a license perspective, I began to update our internal wiki with all of the pieces of EM which would be hard to remember/replace in the event that they were lost.  This included a list of current EM users and their access, scheduled jobs, and user-defined metrics.  Finally, I shut everything down and made a VMWare snapshot of the disk, so that I could revert painlessly, if necessary.

Weblogic Server

The next step was to install Weblogic server, which is a pre-requisite to running the EM 11g installer.  In reality, you could install this web server anytime, whether EM 10g was running or not.  I went to pull it from OTN, and noticed a few things.  First, it appears that Fusion Middleware 11gR1 (10.3.3 – in typical Oracle versioning tradition) has been release very recently.  I was only interested in what I knew was certified, Fusion Middleware 11g (10.3.2 – couldn’t Oracle put one person in charge of making sure that the technical versions match the product names, and that the versions were generally aligned?).  You can find the download page here.  I was looking for linux x86_64, which isn’t on their list, so I pulled the generic platform version.  There’s a small net installer (stub file which goes out and grabs the install later) or a package installer (traditional, everything in there version). Grab whatever floats your boat.   When I grab the generic version, it requires I have my own JDK installed, so to invoke the installer it looks something like this:

java -jar wls1032_generic.jar

The familiar Oracle looking installer starts; I’ll use screenshots from here on as it will probably be easier to follow, just go left to right:

1. Initial splash screen

2. Give us the information we already have!

My personal favorite after leaving email address blank:

Are you sure you want to remain clueless??

You didn't give us your email! Are you really really sure we can't spam you?

OK, now back to the installation… I missed the screenshot, but somewhere you are asked for 2 passwords.  Make sure you

jot them down, as you’ll need them later to administer the web server and also to do the actual EM install.

3. Where is your JDK?

4. Where do you want to put the Weblogic App Server?

5. Default components

6. Installer at Work

.

7. Complete (Uncheck Quickstart)

8. Quickstart - If you get to this screen... cancel it and exit!

Enterprise Manager 11g Installation

This installation impressed me; it was straightforward and worked.  I do have to ding it on a few things.  First, Oracle decided that EM 11g would run by default as https using an SSL cert it generates during the install.  So many problems there.  First, our proxy server (where all web traffic goes, by default, even if  redirected back internally) has a security mechanism set up to not allow SSL on non-standard (443, 8443) ports.  Oracle chose something logical: port 1159.  Second, we have an internal certificate I could use, which would be far more legitimate.  How about asking me if I have something for it to chew on, instead of having it generate some garbage cert which I have to figure out how to replace?

I ran into some RDBMS parameter issues.  EM 11g requires you to have the repository running 11.1 or 11.2, and there’s even some patches they want in place before you do the upgrade.   Well worth reviewing the prerequisites and documentation.  Before you run the installer, issue an

alter system set job_queue_processes=0;

and revert it upon completion to its original value.

Finally, I ran into a problem early on when the installer detected the Oracle Applications Management Pack in my EM 10g inventory.  At some point in the past, I’d installed it with the intention of seeing what it brought to the table.  That’s as far as I ever got, never even configured it, but the installer basically says you can’t do an upgrade with that pack in place.  You have to do a full install.  Ick.  So I uninstalled it via OUI from the OMS home and it happily proceeded next time I ran the installer.  I guess if you’re using that APM you’re out of luck.

You can grab EM 11g from OTN here; once extracted & staged you just use the old standby:

runInstaller

1. Welcome to our latest product! Can we have your email please?

2. We really need to know more about you. Can we have your username and password to the world's slowest support site?

3. I chose to upgrade 10.2.0.5

4. Hey! I found your old stuff!

.

5. I see your listener - whats the sys password?

6. Hey Bozo! Didn't you read the upgrade guide? There's some mandatory parameters you need to fix!

7. I found your Middleware home. Where do you want to install EM 11g?

8. What were your Weblogic passwords?

.

.

Did you notice in step 8, it’s adding a tablespace?

9. Do you want to change ports that you don't even understand the use of?

10. Ready to launch!

.

.

11. Chugging along for about an hour

12. The root.sh we all know and love, now available in allroot.sh!

.

13. Read the fine print; it has pertinent URLs and information!

14. More fine print

.

.

.

.

15. Do you agree to tell the truth, the whole truth, and nothing but the truth?

.

.

.

.

.

.

Success & First Impressions

First of all, it works, the installer didn’t error out too much, and after getting past the ssl port/cert issues (use IE it has the best error messages!) I have to say my first impression was that it did everything right.  My users were all still there, all my targets were there, my user-defined metrics survived, and my jobs were all running.  It even emailed me an alert I was subscribed to when I shut down a scrub database.  No tweaks.  The graphics look a little cleaner.  I am guessing that the java pages were compiled on the fly, as it was painfully slow to do anything the first time.  After that, it was noticeably faster than EM 10g.  Must be all the Fusion going on.

Here’s what the screen looks like:

A new look, but familiar in most ways.

.
.
.
.

.

.

.

.

.

.

Starting and Stopping

This has changed a little bit, but I thought I’d just copy my start/stop scripts in here wholesale and you can tweak them for your environment:

Start:

#!/bin/bash
ORACLE_HOME=/u01/oem/Middleware/oms11g
AGENT_HOME=/u01/oem/Middleware/agent11g
$ORACLE_HOME/bin/emctl start oms
$AGENT_HOME/bin/emctl start agent
$ORACLE_HOME/bin/emctl status oms -detail

Stop:

#!/bin/bash
ORACLE_HOME=/u01/oem/Middleware/oms11g
AGENT_HOME=/u01/oem/Middleware/agent11g
$ORACLE_HOME/bin/emctl stop oms -all
$AGENT_HOME/bin/emctl stop agent

Final Notes

What’s up with all the periods (.) all over this post? I fought WordPress the whole time to keep my screenshots from overlapping section headers and each other. Eventually I gave up and used periods to occupy white space as a placeholder.
A special note of acknowledgment to sysadmin guy who made me understand why no browser could find the login page (the non-standard SSL port issue noted above).   I literally did the install twice, thinking I’d screwed something up or missed some step.  In the end, I probably had it right the first time, but didn’t see any way to reach the login page so reverted and reinstalled all pieces.
The Weblogic server admin screens have about a million tabs, checkboxes, radio buttons and text-entry areas for parameters.  There is a learning curve there:

There's tabs on the tabs!

Addendum

I feel compelled to add an addendum to this posting because of a few things I’ve noticed after running 11g a few days.  First, it consumes more memory; we had to throw another gig at it (database footprint unchanged).  Second, after rebooting the host, we noticed it trying to start processes; it slipped a gcstartup file in the init.d directory.  This script basically checks your oratab file and tries to start whatever homes it finds in there.  Fine and good (I guess)  but really of no value for me.  Having the Weblogic server and the OMS “running” before the database isn’t really useful to me.

Useful URLs and other eratta

main EM login screens:

Weblogic console

It appears that some of the logs aren’t automatically rotated in $ORACLE_INSTANCE/WebtierIH1/diagnostics/logs/OHS/ohs1; I’d double check it if I were you.  My mod_wl_ohs.log was 500Mb.  I fixed this by gzipping the old mod_wl_ohs.log at the end of my stop enterprise manager script; every time I shut down EM, it will gzip it and eventually age 2 weeks and be caught by:

In addition, you probably want to consider a cron job to delete old access/error/em_upload_https logs; something like this is pretty vanilla in the crontab (delete logs older than about 2 weeks):

/usr/bin/find /u01/oem/gc_inst/WebTierIH1/diagnostics/logs/OHS/ohs1 -mtime +14 -exec rm -rf {} \;

Advertisements

30 Comments

  1. As usual, insightful, informative, and entertaining, to boot. Great walkthrough.

    It was good to meet you in person at OAUG last month.

    Regards,
    Steven

    Comment by Steven Chan — May 9, 2010 @ 5:46 pm

  2. Thanks for the kind words, glad you enjoyed the writeup. I’m still working out new bugs (like false alerts in the middle of the night), but I think this product is a keeper.

    Nice to meet you as well.

    Comment by kkempf — May 10, 2010 @ 8:34 am

  3. I am having trouble with the agent uploading to oms. Both the oms and agent on the same host.

    The emctl upload gives the following error
    EMD upload error: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors.

    Comment by Sai — June 11, 2010 @ 3:39 am

    • If I had a quarter for every agent-related issue I’ve had with EM in the past 5 years I’d be off to Vegas. I don’t have an exact solution for your problem; I’d start with My Oracle Support and see what you can find. My most common fixes are clearing the upload directory ($AGENT_HOME/sysman/emd/upload) from the agent host and restarting the agent, or removing and re-discovering the target from with Grid Control (you will lose all history for affected targets).

      Comment by kkempf — June 15, 2010 @ 8:32 am

  4. Nice walkthrough – I also undertook an install of OEM 11g about 2 weeks after it’s initial release at the end of April. I am seeing some questionable behaviour when it comes to memory consumption and I am wondering if you are seeing the same thing. We are running the repository, wls/oms on the same host (single-tier) with Linux x86_64. We have 16gb of real and 16gb of swap for memory. With all defaults for jvm sizing of the managed servers and a 6Gb memory target set on the repository – we are seeing memory utilization go into the swap memory over time. I figure that I only have about 9Gb of memory spoken for – so I am wondering where the rest is coming from. Nothing else is running on the server.

    Comment by Brenna — July 23, 2010 @ 10:23 am

    • I haven’t seen any memory issues, aside from the initial larger footprint I noted in the post. We are running all of this on 1 Red Hat 5.4 x86_64 virtual machine, with 2 cpu’s and 5 gb of RAM. The Oracle (11.2) database gets 1gb for its SGA; the rest is for EM/Weblogic and a few other lightweight applications cohabiting the server. In other words, you have far more system resources than my little install. I’d be curious to hear what you find; sounds like a memory leak somewhere.

      Comment by kkempf — July 23, 2010 @ 10:57 am

  5. Wondering if you can do me a favor? Can you please tell me how much memory is being consumed by the emagent process for your DB repository? I have been using /proc//status and looking at the VMSize and VMPeak, Threads (34) values.
    Another value of interest is to look at the amount of memory being consumed by the java_classpath (jar files loaded into memory). My emagent process is consuming almost 6Gb of memory all by itself, which I find really strange.

    If you would like to take this offline and out of your blog – I can be contacted via e-mail.

    Comment by Brenna — July 23, 2010 @ 5:53 pm

  6. Hello,
    I am trying to installing Enterprise Manager Grid Control 11g on LInux x86_64 bit but i am facing the following problem.and i am using the following softwares

    OEL 5.3 x86_64 bit
    Oracle Grid control 11g R1
    Oracle database 11g R2
    java 1.6 update 18 64 bit
    weblogic 3.2 Generic

    PROBLEM:
    =========
    Jul 27, 2010 4:47:21 PM oracle.sysman.omsca.framework.OMSGenericAdapter postDeployAndReposSetup
    SEVERE: OMSCA-ERR:Securing of OMS failed. Check the trace file:/u01/app/oracle/Middleware/oms11g/cfgtoollogs/omsca/omsca_20100727163923.log
    Jul 27, 2010 4:47:21 PM oracle.sysman.omsca.framework.OMSCAFreshInstall execute
    SEVERE: Securing of OMS failed.
    Jul 27, 2010 4:47:21 PM oracle.sysman.omsca.framework.OMSConfigAssistantDriver main
    FINE: Got resultfalse

    Please help me if any one knows this issue. i face this issue in 32 bit installation but it was a weblogic version problem but there i am using weblogic 3.2 Generic so why they give me the error during installation

    Regards
    Muhammad Aamir

    Comment by Muhammad Aamir — July 29, 2010 @ 12:57 pm

    • Muhammad –

      What does your trace file: /u01/app/oracle/Middleware/oms11g/cfgtoollogs/omsca/omsca_20100727163923.log say?

      I hope that it is not necessary for me to ask this – but anyways… What version of weblogic are you running? Weblogic 10.3.2 (not 3.2 as you indicate above)?

      If you look this error up in ML – it specifically indicates that the error is caused by having Weblogic 10.3.3 installed (Note 1135493.1).

      Comment by Brenna — July 29, 2010 @ 2:42 pm

  7. Wow, this is right on the money. I wish I’d found this before I did my upgrade (10.2.0.4 to 11.1.0.1). I ran into exactly the same issues, right down to the Apps Management Pack. We weren’t using it either for the same reasons, so I did the same thing. Great post.

    Comment by Patrick Melvin — July 30, 2010 @ 11:54 am

    • At least you survived the upgrade.. I think there are some nuances of upgrading vs. re-installing. For instance, I believe the whole port 1159 thing is only applicable to upgrades, but I can’t be sure. Regardless, it’s a rather confusing mess, best seen in the rear view mirror!

      Comment by kkempf — August 3, 2010 @ 8:14 pm

  8. Hi Kevin,

    Do your oraInst.loc files point to the right 11g inventory locations? After upgrade from 10g to 11g mine are still pointing to the old 10 directories that no longer exist. As a result, tools like agentca and oPatch are not working for me on 11g.
    I logged a SR at Oracle Support, but they seem to be puzzled by my problem.

    Comment by Martijn — September 10, 2010 @ 7:38 am

    • I checked on my EM host and every oraInst.loc points to the same/right inventory (there’s a lot of them!). As a matter of clarification, did your inventory location change, and if so, why?

      $ locate oraInst.loc
      /etc/oraInst.loc
      /u01/oem/11.2.0/oraInst.loc
      /u01/oem/Middleware/Oracle_WT/oraInst.loc
      /u01/oem/Middleware/agent11g/oraInst.loc
      /u01/oem/Middleware/oms11g/oraInst.loc
      /u01/oem/Middleware/oracle_common/oraInst.loc
      $ cat /etc/oraInst.loc
      inventory_loc=/opt/oracle/oraInventory
      inst_group=dba
      $ cat /u01/oem/11.2.0/oraInst.loc
      inventory_loc=/opt/oracle/oraInventory
      inst_group=dba
      $ cat /u01/oem/Middleware/Oracle_WT/oraInst.loc
      inventory_loc=/opt/oracle/oraInventory
      inst_group=dba
      $ cat /u01/oem/Middleware/agent11g/oraInst.loc
      inventory_loc=/opt/oracle/oraInventory
      inst_group=dba
      $ cat /u01/oem/Middleware/oms11g/oraInst.loc
      inventory_loc=/opt/oracle/oraInventory
      inst_group=dba
      $ cat /u01/oem/Middleware/oracle_common/oraInst.loc
      inventory_loc=/opt/oracle/oraInventory
      inst_group=dba

      Comment by kkempf — September 10, 2010 @ 9:51 am

      • Hi Kevin,

        Thank you for the information. Maybe I deleted something that I should not have…

        My ‘old’ 10g environment had an oraInventory dir in:
        /oracle/product/oem10g/oraInventory

        My oraInst.loc files are all still pointing to this location. I was under the impression that this was a 10g folder that could be deleted after upgrade.

        Everything in Grid Control seems to be working fine, only commandline tools that use the oraInst.loc files are not working.

        I probably could retreive this ’10g’ oraInventory from backup.

        When I do a ‘find’ on my entire server there is no other oraInventory to be found.

        So, could it be that this oraInventory was upgraded and I should not have deleted it?

        Thanks for any insight.

        Martijn

        Comment by Martijn — September 13, 2010 @ 6:30 am

      • Since it doesn’t appear you don’t have a new (second) inventory from the 11g Grid Control install, I believe you should restore your inventory if possible. I believe you can restore it anywhere you like, as long as every oraInst.loc on the host is corrected to show the new location. Or you could use symbolic links to tidy it up. Did you delete the /oracle/product/oem10g directory after getting to 11g?

        The inventory is not specific to an install; it is where any install/uninstall on the host is recorded. In general, under Linux, the file /etc/oraInst.loc serves as the “master pointer” if you will, to the hosts’ inventory. The first installation lays out this inventory, so, in your case, if it was OEM 10g, it appears that it didn’t go to the default location.

        As a practical tip, I usually tar (tar -cvzf oem10g.tar) up the “old” install directories before I remove them and leave the tar there for a few weeks until I am sure everything is running right. I’ve had to open them up to get tnsnames.ora files, logs, init.ora files, inventories, you name it. It’s a nice insurance policy if you can afford the space.

        Comment by kkempf — September 13, 2010 @ 7:19 am

  9. Hi Kevin,

    Thanks for your explanation, I now understand why I only have one oraInventory.
    Good advice to put the ‘old’ installation in a tar file. Will do that next time. Good thing we make snapshots, so I could restore easily.

    After I did restore the oraInventory my agentca utility still was not working. I edited this file:
    /oracle/product/oem10g/oraInventory/ContentsXML/inventory.xml

    $ cat inventory.xml.bak


    10.2.0.3.0
    2.1.0.6.0

    to this:

    $ cat inventory.xml


    10.2.0.3.0
    2.1.0.6.0

    So now things are pointing to my 11g installation, and agentca is working again.

    I always like to edit files, that Oracle tells you to do not edit by hand…

    Anyway, just to check, are there any other Oracle Home entries in your inventory.xml file? Maybe there are because things are running on Weblogic now?

    Comment by Martijn — September 13, 2010 @ 8:40 am

    • Somehow the contents of inventory.xml did not come through in my posting…
      The Oracle Homes in this file were still pointing to 10g instead of 11g for the agent and oms, so I changed that to 11g.

      Comment by Martijn — September 13, 2010 @ 8:42 am

      • let’s try again:

        old:

        new:

        Comment by Martijn — September 13, 2010 @ 8:45 am

      • I have the following (active, where there’s no “REMOVED = T” flag) entries in my inventory:


        HOME NAME=”OraDb11g_home2″ LOC=”/u01/oem/11.2.0″ TYPE=”O” IDX=”9″
        HOME NAME=”oms11g1″ LOC=”/u01/oem/Middleware/oms11g” TYPE=”O” IDX=”10″
        HOME NAME=”agent11g1″ LOC=”/u01/oem/Middleware/agent11g” TYPE=”O” IDX=”11″
        HOME NAME=”common11g1″ LOC=”/u01/oem/Middleware/oracle_common” TYPE=”O” IDX=”12″
        HOME NAME=”webtier11g1″ LOC=”/u01/oem/Middleware/Oracle_WT” TYPE=”O” IDX=”13″

        Comment by kkempf — September 13, 2010 @ 8:47 am

  10. Aha, so in wordpress you cannot use arrows in your posts…

    Thanks for the info, I added all 4 locations now to the file. I was missing the oracle_common and webtier location.

    All is working now! thanks a million.

    Comment by Martijn — September 13, 2010 @ 9:01 am

  11. Hi.
    when i try to install grid control i got this error.

    ./runInstaller -debug
    Starting Oracle Universal Installer…

    Checking Temp space: must be greater than 150 MB. Actual 200338 MB Passed
    Checking swap space: must be greater than 150 MB. Actual 3982 MB Passed
    Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
    Preparing to launch Oracle Universal Installer from /tmp/OraInstall2010-12-15_07-31-56AM. Please wait …unzip: cannot find or open ../../stage/Components/oracle.jdk/0.0.0.0.0/1/DataFiles/*.jar, ../../stage/Components/oracle.jdk/0.0.0.0.0/1/DataFiles/*.jar.zip or ../../stage/Components/oracle.jdk/0.0.0.0.0/1/DataFiles/*.jar.ZIP.

    No zipfiles found.

    Invalid source path ‘../../stage/Components/oracle.jdk/0.0.0.0.0/1/DataFiles’ specified for unzip. Unzip command failed. Please check oraparam.ini and specify a valid source path.
    : No such file or directory
    Please help me??

    Comment by Decio Brito — December 15, 2010 @ 12:48 pm

    • Looks rather odd; I would be inclined to re-download the install set and, failing that, open an SR with Oracle.

      Comment by kkempf — December 16, 2010 @ 1:35 pm

  12. […] EM 11g Grid Control Install May 2010 22 comments and 1 Like on WordPress.com, 3 […]

    Pingback by 2010 in review « Kevin Kempf's Blog — January 7, 2011 @ 10:41 pm

  13. Hi ,

    Does it mandatory install weblogic server before install Grid 11gR2

    Comment by Sathish — February 24, 2011 @ 3:44 am

    • WLS 10.3.X (aka Weblogic Server) is a mandatory pre-requisite requirement for installing Fusion Middleware 11g (FMW). FMW includes things like Discoverer and Enterprise Manager. When you say Grid 11gR2 it’s not precise enough for me to guess your meaning.

      Comment by kkempf — February 24, 2011 @ 7:30 am

  14. Hi ,

    While installing Grid,successfully installed Weblogic server without any issue,however I am getting following error while installing GRID, please let me know if you have any idea about this.

    ar 4, 2011 10:44:42 AM oracle.sysman.omsca.util.CoreOMSConfigAssistantUtil execCommand
    INFO: error messages of the command :
    weblogic.nodemanager.NMConnectException: Connection refused. Could not connect to NodeManager. Check that it is running at node1.slksoft.com:7,403.
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
    at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
    at java.net.Socket.connect(Socket.java:525)
    at weblogic.nodemanager.client.SSLClient.createSocket(SSLClient.java:38)
    at weblogic.nodemanager.client.NMServerClient.connect(NMServerClient.java:227)
    at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:199)
    at weblogic.nodemanager.client.NMServerClient.checkConnected(NMServerClient.java:205)
    at weblogic.nodemanager.client.NMServerClient.getVersion(NMServerClient.java:52)
    at weblogic.management.scripting.NodeManagerService.verifyConnection(NodeManagerService.java:175)
    at weblogic.management.scripting.NodeManagerService.nmConnect(NodeManagerService.java:168)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.python.core.PyReflectedFunction.__call__(Unknown Source)
    at org.python.core.PyMethod.__call__(Unknown Source)
    at org.python.core.PyObject.__call__(Unknown Source)
    at org.python.core.PyObject.invoke(Unknown Source)

    Comment by satish — March 3, 2011 @ 8:52 pm

    • A quick search on My Oracle Support of “weblogic.nodemanager.NMConnectException: Connection refused” shows Checking the same omsca_.log file one can see that Weblogic Server 10.3.3 has been used and not Weblogic Server 10.3.2 – you can read about it in note 1135493.1

      Comment by kkempf — March 4, 2011 @ 9:08 am

  15. and also i am getting below error

    oracle@node1 grid]$ export ORACLE_SID=prod
    [oracle@node1 grid]$ emctl status agent
    EM Configuration issue. /u01/oracle/product/11.2.0/db_1/node1.soft.com_prod not found.
    [oracle@node1 grid]$

    Comment by satish — March 3, 2011 @ 9:39 pm

    • A quick search on My Oracle Support of “EM Configuration issue” shows Fully Qualified Domain Name (eg. host1.domain.com) was recently added or changed in /etc/hosts file or DNS. I’d suggest you read about it in note 436845.1

      Comment by kkempf — March 4, 2011 @ 9:04 am

  16. Thank you Rob.

    Comment by satish — March 5, 2011 @ 5:14 am


RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Blog at WordPress.com.