Semafore? Cannot mount in Exclusive.

As Chen says, the challenges always come after 5PM.

I had a Daylight Savings Patch to apply to 2 databases. I think that someone else, was blogging about installing this just recently. The patch is trivial - bring down the database, replace 2 files deep within ORACLE_HOME and bring the database back up. The first install went went off uneventfully, so I figured that I had an idea what I was doing. Then I moved to the second DB and the fun started. The DB wouldn't restart. How can I NOT startup a database? What can be simpler than a STARTUP?

ORA-01102:  cannot mount database in EXCLUSIVE mode

Now that I have a solution it all seems simple. 

The problem  according to Metalink...

- there is still an "sgadef.dbf" file in the "ORACLE_HOME/dbs"
 directory
nope

- the processes for Oracle (pmon, smon, lgwr and dbwr) still exist
no, they are gone..

- shared memory segments and semaphores still exist even though the
 database has been shutdown

I got to learn about the unix command ipcs -b, but
nothing owned by oracle, 
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x7800000a --rw-rw-rw-     root   system  16777216
m   1048577 0x0d001213 --rw-rw----     root   system      1440
m         3 0xffffffff --rw-rw----     root   system      4096
T        ID     KEY        MODE       OWNER    GROUP NSEMS
Semaphores:
s   3145728 0x010000af --ra-------     root   system     1
s         1 0x6200105e --ra-r--r--     root   system     1

- there is a "ORACLE_HOME/dbs/lk" file

and indeed
sculkget: failed to lock /home/oracle/orabase/product/10.2.0/dbs/lkSID exclusive
sculkget: lock held by PID: 299506

though I have yet to figure out what a sculkget is I did find a file that was 
skulking around and didn't belong there.  

With trepidation I killed the process holding the lock file

kill -9 299506

and removed the lock file

rm /home/oracle/orabase/product/10.2.0/dbs/lkSID 

and magic!  no more lock, Database starts up normally! 

Locks in their place and all is well with the world.

Meanwhile, if anyone else knows what category of animal a 
sculkget: message is - I would love to know. There is a singular lack of comment
about it on the web, except as part of this specific problem. 

Exposed Password in Data Pump

I was just looking at a production Data Pump today with a text exposed 'system/password'. I was startled.

Then I remembered that in the last shop Data Pumps involved

1. Temporarly change the system user password
2. Run the Data Pump with the temporary password in plain text
3. Changing the password back.
   

Now that I think of it, the one involving all that changing is only slightly better than what I saw today.

Does anyone have a suggestion for running Data Pump with a modicum of security? I had no say in the last shop. Here I may not prevail, but at least I'm in a position to make a suggestion if I have one.

Does the solution involve something with a password protected parameter file? Or? Thanks in advance for any suggestions offered.

When all else fails, follow instructions.

1. Today I tried to open my test database and got:

ORA-16038: log 3 sequence# 729 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 3 thread 1: 'C:\ORACLE\ORA10\ORADATA\DB10\REDO03.LOG'

Researching the problem I found:

• Maybe your Oracle instance has reached the maximum db recovery file destination file size assigned to it? If that is so, probably your oracle instance will stop working and you will get the 3 error messages listed above while trying to start it.
• The relevant parameter is: DB_RECOVERY_FILE_DEST_SIZE - specifies (in bytes) the hard limit on the total space to be used by target database recovery files created in the flash recovery area.
• It works in concert with DB_RECOVERY_FILE_DEST which specifies the default location for the flash recovery area. The flash recovery area contains multiplexed copies of current control files and online redo logs, as well as archived redo logs, flashback logs, and RMAN backups.

1. 
   
2. What is the maximum db recovery file destination file size assigned in my DB?

SQL> show parameter db_recovery_file_dest_size
db_recovery_file_dest_size 2G

At least this wasn't a surprise.

2. 
   

3. I removed excess files

I shutdown the database. I then removed the archivelog files and the flashback files from the file system – even though they didn't amount to the 2gb in total that the system is complaining about. (This is a play database so I can do things like kill these files.)

4. Still had a problem.

I brought the database up to mount state, but immediately there was a trace file message:

ORA-19815: WARNING: db_recovery_file_dest_size of 2147483648 bytes is 99.85% used, and has 3320320 remaining bytes available.

*** 2008-03-03 10:03:19.635

************************************************************

You have following choices to free up space from flash recovery area:

1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard, then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating system command was used to delete files, then use RMAN CROSSCHECK and DELETE EXPIRED commands.

5. The files were still being seen by RMAN! When all else fails, follow instructions, in this case #4 above. “4. Delete unnecessary files using RMAN DELETE command. If an operating system command was used to delete files, then use RMAN CROSSCHECK and DELETE EXPIRED commands.”

RMAN> crosscheck archivelog all;
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=146 devtype=DISK
validation failed for archived log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_733_3WRQ14DY_.ARC recid=70 stamp=648389669

validation failed for archived log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_734_3WRQ4V7R_.ARC recid=71 stamp=648389787

What nice list of all the files that aren't there, and all the locations where they aren't; 68 files that don't exist! [apologies to any reader of English as a second language]

RMAN> delete expired archivelog all;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=146 devtype=DISK
List of Archived Log Copies
Key Thrd Seq S Low Time Name
------- ---- ------- - --------- ----
70 1 733 X 02-MAR-08 C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2
008_03_03\O1_MF_1_733_3WRQ14DY_.ARC

71 1 734 X 03-MAR-08 C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2
008_03_03\O1_MF_1_734_3WRQ4V7R_.ARC

(etc)

Do you really want to delete the above objects (enter YES or NO)? yes

deleted archive log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_733_3WRQ14DY_.ARC recid=70 stamp=648389669

deleted archive log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_734_3WRQ4V7R_.ARC recid=71 stamp=648389787

(etc)

Deleted 68 EXPIRED objects

5. 
   

6. Now all is well. (with the database if not with this blog-entry's numbers). I bounced the database and the alert log shows:
   

db_recovery_file_dest_size of 2048 MB is 0.38% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup.