Semafore? Cannot mount in Exclusive.

As Chen says, the challenges always come after 5PM.

I had a Daylight Savings Patch to apply to 2 databases. I think that someone else, was blogging about installing this just recently. The patch is trivial - bring down the database, replace 2 files deep within ORACLE_HOME and bring the database back up. The first install went went off uneventfully, so I figured that I had an idea what I was doing. Then I moved to the second DB and the fun started. The DB wouldn't restart. How can I NOT startup a database? What can be simpler than a STARTUP?

ORA-01102:  cannot mount database in EXCLUSIVE mode

Now that I have a solution it all seems simple. 

The problem  according to Metalink...

- there is still an "sgadef.dbf" file in the "ORACLE_HOME/dbs"
 directory
nope

- the processes for Oracle (pmon, smon, lgwr and dbwr) still exist
no, they are gone..

- shared memory segments and semaphores still exist even though the
 database has been shutdown

I got to learn about the unix command ipcs -b, but
nothing owned by oracle, 
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x7800000a --rw-rw-rw-     root   system  16777216
m   1048577 0x0d001213 --rw-rw----     root   system      1440
m         3 0xffffffff --rw-rw----     root   system      4096
T        ID     KEY        MODE       OWNER    GROUP NSEMS
Semaphores:
s   3145728 0x010000af --ra-------     root   system     1
s         1 0x6200105e --ra-r--r--     root   system     1

- there is a "ORACLE_HOME/dbs/lk" file

and indeed
sculkget: failed to lock /home/oracle/orabase/product/10.2.0/dbs/lkSID exclusive
sculkget: lock held by PID: 299506

though I have yet to figure out what a sculkget is I did find a file that was 
skulking around and didn't belong there.  

With trepidation I killed the process holding the lock file

kill -9 299506

and removed the lock file

rm /home/oracle/orabase/product/10.2.0/dbs/lkSID 

and magic!  no more lock, Database starts up normally! 

Locks in their place and all is well with the world.

Meanwhile, if anyone else knows what category of animal a 
sculkget: message is - I would love to know. There is a singular lack of comment
about it on the web, except as part of this specific problem.