Moving back to a lost spfile

We had an unexpected outage last saturday morning, and the person who restarted the database inadvertantly did so with an old pfile instead of the spfile - which was lost (not because of the restart). How to get back to the correct set of parameters. EM to the rescue. Looking at all the parameters in EM it shows which have been changed and when. I could then alter the parameters back to where they should be. The dynamic parameters were easy - i.e.
ALTER SYSTEM SET parallel_max_servers = 280;

I then created an spfile from the pfile with only the dynamic parameters corrected.

create pfile from spfile;

But I was looking at an out of date article which put the had an error about the default location of the spfile.

SELECT name, value FROM v$parameter WHERE name = 'spfile';

gave me the actual location of my spfile, which being in the default location was in
$ORACLE_HOME/dbs/spfileSID.ora



Then bounced the database and brought it up with the spfile.

But, of course, the static parameters need a bounce to the database.

I altered the parameters scope = spfile eg:
ALTER SYSTEM SET processes = 300 SCOPE=SPFILE;

and then rebounced the database to apply the dynamic changes also to the current instance.

THEN I BACKED UP THE SPFILE!




(now if only I could control the fonts on this blog, all would be well.)

Semafore? Cannot mount in Exclusive.

As Chen says, the challenges always come after 5PM.

I had a Daylight Savings Patch to apply to 2 databases. I think that someone else, was blogging about installing this just recently. The patch is trivial - bring down the database, replace 2 files deep within ORACLE_HOME and bring the database back up. The first install went went off uneventfully, so I figured that I had an idea what I was doing. Then I moved to the second DB and the fun started. The DB wouldn't restart. How can I NOT startup a database? What can be simpler than a STARTUP?

ORA-01102:  cannot mount database in EXCLUSIVE mode

Now that I have a solution it all seems simple. 

The problem  according to Metalink...

- there is still an "sgadef.dbf" file in the "ORACLE_HOME/dbs"
 directory
nope

- the processes for Oracle (pmon, smon, lgwr and dbwr) still exist
no, they are gone..

- shared memory segments and semaphores still exist even though the
 database has been shutdown

I got to learn about the unix command ipcs -b, but
nothing owned by oracle, 
T        ID     KEY        MODE       OWNER    GROUP     SEGSZ
Shared Memory:
m   1048576 0x7800000a --rw-rw-rw-     root   system  16777216
m   1048577 0x0d001213 --rw-rw----     root   system      1440
m         3 0xffffffff --rw-rw----     root   system      4096
T        ID     KEY        MODE       OWNER    GROUP NSEMS
Semaphores:
s   3145728 0x010000af --ra-------     root   system     1
s         1 0x6200105e --ra-r--r--     root   system     1

- there is a "ORACLE_HOME/dbs/lk" file

and indeed
sculkget: failed to lock /home/oracle/orabase/product/10.2.0/dbs/lkSID exclusive
sculkget: lock held by PID: 299506

though I have yet to figure out what a sculkget is I did find a file that was 
skulking around and didn't belong there.  

With trepidation I killed the process holding the lock file

kill -9 299506

and removed the lock file

rm /home/oracle/orabase/product/10.2.0/dbs/lkSID 

and magic!  no more lock, Database starts up normally! 

Locks in their place and all is well with the world.

Meanwhile, if anyone else knows what category of animal a 
sculkget: message is - I would love to know. There is a singular lack of comment
about it on the web, except as part of this specific problem. 

Exposed Password in Data Pump

I was just looking at a production Data Pump today with a text exposed 'system/password'. I was startled.

Then I remembered that in the last shop Data Pumps involved

1. Temporarly change the system user password
2. Run the Data Pump with the temporary password in plain text
3. Changing the password back.
   

Now that I think of it, the one involving all that changing is only slightly better than what I saw today.

Does anyone have a suggestion for running Data Pump with a modicum of security? I had no say in the last shop. Here I may not prevail, but at least I'm in a position to make a suggestion if I have one.

Does the solution involve something with a password protected parameter file? Or? Thanks in advance for any suggestions offered.

When all else fails, follow instructions.

1. Today I tried to open my test database and got:

ORA-16038: log 3 sequence# 729 cannot be archived
ORA-19809: limit exceeded for recovery files
ORA-00312: online log 3 thread 1: 'C:\ORACLE\ORA10\ORADATA\DB10\REDO03.LOG'

Researching the problem I found:

• Maybe your Oracle instance has reached the maximum db recovery file destination file size assigned to it? If that is so, probably your oracle instance will stop working and you will get the 3 error messages listed above while trying to start it.
• The relevant parameter is: DB_RECOVERY_FILE_DEST_SIZE - specifies (in bytes) the hard limit on the total space to be used by target database recovery files created in the flash recovery area.
• It works in concert with DB_RECOVERY_FILE_DEST which specifies the default location for the flash recovery area. The flash recovery area contains multiplexed copies of current control files and online redo logs, as well as archived redo logs, flashback logs, and RMAN backups.

1. 
   
2. What is the maximum db recovery file destination file size assigned in my DB?

SQL> show parameter db_recovery_file_dest_size
db_recovery_file_dest_size 2G

At least this wasn't a surprise.

2. 
   

3. I removed excess files

I shutdown the database. I then removed the archivelog files and the flashback files from the file system – even though they didn't amount to the 2gb in total that the system is complaining about. (This is a play database so I can do things like kill these files.)

4. Still had a problem.

I brought the database up to mount state, but immediately there was a trace file message:

ORA-19815: WARNING: db_recovery_file_dest_size of 2147483648 bytes is 99.85% used, and has 3320320 remaining bytes available.

*** 2008-03-03 10:03:19.635

************************************************************

You have following choices to free up space from flash recovery area:

1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard, then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating system command was used to delete files, then use RMAN CROSSCHECK and DELETE EXPIRED commands.

5. The files were still being seen by RMAN! When all else fails, follow instructions, in this case #4 above. “4. Delete unnecessary files using RMAN DELETE command. If an operating system command was used to delete files, then use RMAN CROSSCHECK and DELETE EXPIRED commands.”

RMAN> crosscheck archivelog all;
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=146 devtype=DISK
validation failed for archived log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_733_3WRQ14DY_.ARC recid=70 stamp=648389669

validation failed for archived log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_734_3WRQ4V7R_.ARC recid=71 stamp=648389787

What nice list of all the files that aren't there, and all the locations where they aren't; 68 files that don't exist! [apologies to any reader of English as a second language]

RMAN> delete expired archivelog all;

released channel: ORA_DISK_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=146 devtype=DISK
List of Archived Log Copies
Key Thrd Seq S Low Time Name
------- ---- ------- - --------- ----
70 1 733 X 02-MAR-08 C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2
008_03_03\O1_MF_1_733_3WRQ14DY_.ARC

71 1 734 X 03-MAR-08 C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2
008_03_03\O1_MF_1_734_3WRQ4V7R_.ARC

(etc)

Do you really want to delete the above objects (enter YES or NO)? yes

deleted archive log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_733_3WRQ14DY_.ARC recid=70 stamp=648389669

deleted archive log
archive log filename=C:\ORACLE\FLASH_RECOVERY_AREA\DB10\ARCHIVELOG\2008_03_03\O1
_MF_1_734_3WRQ4V7R_.ARC recid=71 stamp=648389787

(etc)

Deleted 68 EXPIRED objects

5. 
   

6. Now all is well. (with the database if not with this blog-entry's numbers). I bounced the database and the alert log shows:
   

db_recovery_file_dest_size of 2048 MB is 0.38% used. This is a user-specified limit on the amount of space that will be used by this database for recovery-related files, and does not reflect the amount of space available in the underlying filesystem or ASM diskgroup.

Trying to lose a datafile.

Chen Shapira wrote me a suggestion regarding losing a data file.

“Try moving a data file while the DB is up and running, and try to make Oracle "notice"
that it is gone.  On Linux, Oracle does not notice anything! You can move a datafile
and all selects, updates, inserts, will work. The OS prevents the DB from seeing
the change. But I noticed that your test system is on windows, so I'm curious
whether it is the same.”

Once my company’s  operator TARed the rest of the database into the system tablespace datafile.
Oracle ‘noticed’!  Are you sure that you really lost the file? Was there some sort of mirroring
of the file?

Anyway, here is what happened when I tried to lose a data file while the DB is up on a Windows based DB…

SQL> select name from v$datafile;

NAME
-------------------------------------------------------------------------------
C:\ORACLE\ORA10\ORADATA\DB10\SYSTEM01.DBF
C:\ORACLE\ORA10\ORADATA\DB10\UNDOTBS01.DBF
C:\ORACLE\ORA10\ORADATA\DB10\SYSAUX01.DBF
C:\ORACLE\ORA10\ORADATA\DB10\O1_MF_USERS_35OQZZ4H_.DBF
C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01.DBF

SQL> host ren C:\ORACLE\ORA10\ORADATA\DB10\O1_MF_USERS_35OQZZ4H_.DBF users.dbf
The process cannot access the file because it is being used by another process.

SQL> host del C:\ORACLE\ORA10\ORADATA\DB10\O1_MF_USERS_35OQZZ4H_.DBF
C:\ORACLE\ORA10\ORADATA\DB10\O1_MF_USERS_35OQZZ4H_.DBF

The process cannot access the file because it is being used by another process.

SQL> host ren C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01.DBF example02.dbf
The process cannot access the file because it is being used by another process.

Also a straight O/S effort:
C:\Windows\System32>del C:\ORACLE\ORA10\ORADATA\DB10\O1_MF_USERS_35OQZZ4H_.DBF

C:\ORACLE\ORA10\ORADATA\DB10\O1_MF_USERS_35OQZZ4H_.DBF
The process cannot access the file because it is being used by another process.

Moving files in Mount State

On Feb 3, I posted “Startup steps review” . Chen Shapira commented

And just for completeness -
You can move data files when in "alter database mount"
Recovery happens on "alter database open"

Which prompts me to try it out. I hadn’t thought about this. Here is how I worked through Chen’s information about moving data files while in the mount stage in my toy database.

I start off with the datafile EXAMPLE01.DBF

SQL> SELECT NAME FROM V$DATAFILE WHERE NAME LIKE '%EXAMPLE%';

NAME
--------------------------------------------------------------------------------
C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01.DBF

I 'lose' the file - by shutting down the database, renaming the file in the OS and
bringing the database back to mount.

SQL> shutdown immediate

Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> host ren c:\Oracle\ora10\oradata\db10\EXAMPLE01.DBF EXAMPLE01_X.DBF

SQL> host dir c:\Oracle\ora10\oradata\db10\E*

Directory of c:\Oracle\ora10\oradata\db10

02/11/2008 11:36 AM 104,865,792 EXAMPLE01_X.DBF

1 File(s) 104,865,792 bytes

SQL> startup mount

ORACLE instance started.

Total System Global Area 201326592 bytes
Fixed Size 1289700 bytes
Variable Size 150995484 bytes
Database Buffers 46137344 bytes
Redo Buffers 2904064 bytes
Database mounted.

I was surprised to see, that in the mount state Oracle takes the file names from the control file, but doesn’t check the operating system to see if they really exist.

SQL> SELECT NAME FROM V$DATAFILE WHERE NAME LIKE '%EXAMPLE%';

NAME
--------------------------------------------------------------------------------
C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01.DBF

It is on the open that Oracle checks reality.

SQL> alter database open;

alter database open
*
ERROR at line 1:
ORA-01157: cannot identify/lock data file 5 - see DBWR trace file
ORA-01110: data file 5: 'C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01.DBF'

And as Chen says – you can move files when in mount phase - so I correct the problem that I made when I 'lost' the file:

SQL> alter database rename file 'C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01.DBF' to 'c:\Oracle\ora10\oradata\db10\EXAMPLE01_X.DBF';

Database altered.

SQL> SELECT NAME FROM V$DATAFILE WHERE NAME LIKE '%EXAMPLE%';

NAME
--------------------------------------------------------------------------------
C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01_X.DBF

And just to look at everything since I am demonstrating:

SQL> alter database backup controlfile to trace;
Database altered.

Now the control file knows where the lost file is. The abbreviated contents of the trace are:

STARTUP NOMOUNT
CREATE CONTROLFILE REUSE DATABASE "DB10" NORESETLOGS NOARCHIVELOG
...
LOGFILE
......

DATAFILE
...
'C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01_X.DBF'

Now I can open the database and all is well:

SQL> alter database open;

Database altered.

SQL> SELECT NAME FROM V$DATAFILE WHERE NAME LIKE '%EXAMPLE%';

NAME
--------------------------------------------------------------------------------
C:\ORACLE\ORA10\ORADATA\DB10\EXAMPLE01_X.DBF

Startup steps review

Do you remember what happens as an instance starts? I didn't. So, to review...

Assuming that you issue the start command in steps:

SQL> STARTUP NOMOUNT;

1 Oracle first opens and reads the initialization parameter file.

2 The memory areas associated with the database instance are allocated. Their size was specified in the parameter file. These allocated memory areas and parameters include the following:

• SGA_TARGET parameter sets the amount of memory available to the SGA.
  
• DB_CACHE_SIZE (default block size) &DB_nK_CACHE_SIZE (non-default block size)
• SHARED_POOL_SIZE (contains the Library Cache & Data Dictionary Cache)
• LOG_BUFFER
  
• LARGE_POOL_SIZE ,
  
• JAVA_POOL_SIZE, & STREAMS_POOL_SIZE
  
• DB_KEEP_CACHE_SIZE & DB_RECYCLE_CACHE_SIZE

3 Oracle background processes are started. In most basic terms this includes the following Oracle background processes:

• DBWR - Database Writer process writes dirty buffers from the database block cache to the data files.
• CKPT - Writes checkpoint information to control files and data file headers.
• LGWR - Log Writer process writes the log buffers out to the redo logs.
• PMON - Process Monitor process recovers failed process resources.
• SMON - System Monitor process recovers after instance failure and monitors temporary segments and extents.
• ARCH - (Optional) Archive process writes filled redo logs to the archive log location(s).
• MMON - The Oracle 10g background process to collect statistics for the Automatic Workload Repository (AWR).

Together, these processes and the associated allocated memory are called the Oracle instance. Once the instance has started successfully, the database is considered to be in the nomount stage.

SQL> alter database mount;

1. When the startup command enters the mount stage, it opens and reads the control file.
2. From the control file Oracle determines the location of the datafiles, but does not yet open them.

SQL> alter database open;

When Oracle opens the database, it

1. accesses all of the datafiles associated with the database. If a file is missing it is reported.

2. makes sure that all of the database datafiles are consistent.

And now the database is ready for use.