A few basic notes on Oracle Database Administration.

Saturday, March 29, 2008

Semafore? Cannot mount in Exclusive.

As Chen says, the challenges always come after 5PM.

I had a Daylight Savings Patch to apply to 2 databases. I think that someone else, was blogging about installing this just recently. The patch is trivial - bring down the database, replace 2 files deep within ORACLE_HOME and bring the database back up. The first install went went off uneventfully, so I figured that I had an idea what I was doing. Then I moved to the second DB and the fun started. The DB wouldn't restart. How can I NOT startup a database? What can be simpler than a STARTUP?

ORA-01102:  cannot mount database in EXCLUSIVE mode

Now that I have a solution it all seems simple.

The problem according to Metalink...

- there is still an "sgadef.dbf" file in the "ORACLE_HOME/dbs"

- the processes for Oracle (pmon, smon, lgwr and dbwr) still exist
no, they are gone..

- shared memory segments and semaphores still exist even though the
database has been shutdown

I got to learn about the unix command ipcs -b, but
nothing owned by oracle,

Shared Memory:
m 1048576 0x7800000a --rw-rw-rw- root system 16777216
m 1048577 0x0d001213 --rw-rw---- root system 1440
m 3 0xffffffff --rw-rw---- root system 4096
s 3145728 0x010000af --ra------- root system 1
s 1 0x6200105e --ra-r--r-- root system 1

- there is a "ORACLE_HOME/dbs/lk" file

and indeed
sculkget: failed to lock /home/oracle/orabase/product/10.2.0/dbs/lkSID exclusive
sculkget: lock held by PID: 299506

though I have yet to figure out what a sculkget is I did find a file that was
skulking around and didn't belong there.

With trepidation I killed the process holding the lock file

kill -9 299506

and removed the lock file

rm /home/oracle/orabase/product/10.2.0/dbs/lkSID

and magic! no more lock, Database starts up normally!

Locks in their place and all is well with the world.

Meanwhile, if anyone else knows what category of animal a

sculkget: message is - I would love to know. There is a singular lack of comment
about it on the web, except as part of this specific problem.


prodlife said...

Challenges always happen after 5pm, and always doing a procedure that was done successfully million times in the past.

Great details of troubleshooting process.

HunterX said...

Great Solution.