Checks found corrupted blocks. What to do now

WARNING: Use this Page only, if previous consistency checks had been performed already!

Overview

Here you can find steps to perform that are either required by SAP Product Support or needed to get rid of blockcorruption!

PLEASE NOTE: General rule for corruption issues is to RESTORE and RECOVER a valid and corruption-free backup or at least parts of it. SAP strongly recommends and informs customers about the importance of backup strategy and highlights that the existence of a valid backup is ESSENTIAL! In case there is no valid backup existing that may be used you as customer have the use one of the below available solutions.

Questions that have to be cleared and answered on customer side!

Is it possible to restore a database backup?
How many corrupted blocks do the database have?
What is the root cause?

Flowchart to solve Oracle Block Corruption issues

This procedure is valid for every SAP customer and clearly states the steps tp be taken.

General Action Plan to follow

Below table shows the steps for the individual parties. Clearly visible, which step has to be performed by who.

Action	Responsible Party
I. Test OS connection to the DB machine

II. Save the status quo
1. Perform a backup of the currently corrupt DB	CUSTOMER
2. Provide SAP the estimated time when backup will be finished after some files have been backed up	CUSTOMER

III. Check if change log between last good backup and now is free of corruptions (other machine(s) required)
1. Restore a backup not containing corruptions to another machine and apply the change log	CUSTOMER
2. If all changes can be applied, the corruption issue can be solved without data loss. If not either a repair of the currently corrupt DB	CUSTOMER
or a reset to an point in time is to be considered.
[Optional, but typical]:
Because the backups have not been checked in the past, the last good backup is not known.
The before mentioned steps need to be done potentially for multiple backups:
the last ASSUMED good backup, the oldest available backup as of which all changes up to now are available
3. Provide SAP the estimated time when restore will be finished after some files have been restored	CUSTOMER
4. Evaluate the change log volume to be applied	CUSTOMER
5. Restore the needed change log (in parallel to restore of backup)	CUSTOMER
6. Provide SAP the estimated time when application of the change log will be finished after some change log files have been applied	CUSTOMER

IV. Evaluate the volume of damage to decide if the better option is to repair the currently corrupt DB or to reset the DB to an older point in time
1. Do a full consistency check on the corrupt productive DB to see the volume of corruptions	CUSTOMER
2. Tell SAP the estimated time when consistency check will be finished after some files have been checked	CUSTOMER
3. Tell SAP new corruptions when logged	CUSTOMER
4. Evaluate possibilities to repair the corrupted tables together with application support	CUSTOMER with SAP
(data needed, cross-dependencies, etc.) (SAP DB Support/Consistency Group/Application Developement)

V. Evaluate the root cause to exclude that further corruptions will be generated
1. Check for special maintenance activity, extraordinary errors on all levels below the DB (OS, IO, storage, HW, etc.) before/while corruptions have been occured	CUSTOMER with HW partner
2. Check the correctness of ALL pieces of HW as careful as possible ("a damaged cable need not to write a log entry")	CUSTOMER with HW partner
3. In case root cause cannot be found until wanted go-live:	CUSTOMER with HW partner
Perform Go-live on completely different HW or at least set up a "Pseudo Standby DB"	CUSTOMER with HW partner
- Copy the currently corrupt DB to another machine with COMPLETELY different HW than production ("Pseudo Standby DB")	CUSTOMER with HW partner
in case only few corruptions occured, they can be repaired but root cause not found until go-live
The "Pseudo Standby DB" recovers permanently the changes of production so that it can be used to switch over in case of new corruptions occur after go-live

# Action Plan for SAP

I. Monitor progress, coordinate actions, double check customer information	SAP
II. Ensure OS access to DB machine (telnet, ssh etc)	SAP
- Setup a OS connection to the DB machine	CUSTOMER