Method and a System for Processing a Log Record

1. Field of the Invention

The present invention relates to a method of processing a log record and, more particularly, to a method of logging a record, which is suitable for a high-traffic on-line data processing system for transaction processing.

2. Description of the Prior Art

In a data processing system for transaction processing in an on-line system according to the prior art, it is necessary to hold the logical consistency of the system even in case of failure such as power interruption or system down. During the system run, therefore, there are logged the records concerning the update of various data by the transactions, the input/output of a message, etc. The log records have to be sequentially logged because their time sequential order of logging is important for their object. As a result, the way of logging log records largely affects the performance of a high-traffic on-line system. As a solution for this problem, there is a method for dispersing the load by logging the records in a plurality of external storages in the order of generation by the round robin method, as disclosed in Japanese Patent Laid-Open No. 60-3036.

This example of the prior art can improve the logging ability in proportion to the number of storages but has a problem that the number of storages becomes excessively large when the traffic increase. Since, moreover, all the log records are written altogether like the method of the prior art, each of the log records is stored more than necessary to raise another problem that the amount of storage of the log records becomes massive.

An object of the present invention is to provide a processing method of logging record, which solves the above-specified problems of the prior art and improves the logging ability without increasing the number of storages while reducing the amount of storage of the log records.

In order to solve those problems, the logging method of the present invention is characterized in that various log records of a data processing system for transaction processing are divided into two, namely records for recovery from failures and other records. The log records for the recovery are logged more frequently than the other records.

For example, the log records for the recovery processing of the system are written in synchronism with the end of the transaction, and the other log records are written altogether for a plurality of transactions.

FIG. 1 is a block diagram showing the summary of one embodiment of the present invention;

FIG. 2 is a block diagram showing the overall construction of a data processing system to which is applied the present invention;

FIG. 3 is a processing flow chart for logging in the embodiment of the present invention;

FIG. 4 is a processing flow chart at a check point; and

FIG. 5 is a processing flow chart of recovery processing of the system.

First of all, the summary of the present invention will be described in the following.

The log records required for the system recovery processing include the update log records of database and the resident tables on-a main storage/virtual storage, the log records concerning input/output messages, and the log records indicating the end of a transaction and so on. These log records are used to recover the transactions and the system into a logically consistent status when the system is stopped by a failure. This makes it necessary to make a logging in a non-volatile storage in synchronism with the end of the transaction. Taking this into consideration, in the present invention, the log records are stored in a high-speed device such as a semiconductor storage, which is made non-volatile by means of a battery, so that frequent input/output operations can be performed at a high traffic rate. The semiconductor storage is usually expensive and does not have a large capacity. However, the log records for the system recovery processing do not require a long period of preservation so that the necessary log records can be written by repeative use of such a device.

On the other hand, the log records to be written for purposes other than the system recovery includes the log records for data base recovery, the log records for recording the business processing, and the log records to be used in a user program for the business processing. These log records need not necessarily be written in synchronism with the end of a transaction, but they need a considerably longer period of preservation than that of the log records for system recovery. In the present invention, therefore, these log records are stored on a disk or magnetic tape which is slower but more inexpensive than the semiconductor storage. In addition, the time period for the mechanical actions, which usually occupies a large portion in the input/output processing of the disk, is minimized by writing the log records altogether for a plurality of transactions, and only the necessary records are written to save storage, whereby improving the logging capability. Incidentally, the disk is advantageous in its large capacity, and the magnetic tape is advantageous in its facility of medium exchange.

The log records to be used in the system recovery processing require a short period of preservation (usually about 1,000 to 10,000 transactions). On the other hand, the log records to be acquired for the purposes other than the system recovery require a long period of preservation but have a less amount of logging than that of the prior art because of the use of necessary records only. As a result, the total amount of the log records is reduced.

The present invention will be described in detail in the following in connection with the embodiment thereof with reference to the accompanying drawings.

FIG. 2 is a diagram showing the overall construction of one embodiment of a data processing system to which is applied the present invention.

In the present data processing system, there are connected with a CPU/memory (i.e., main storage/virtual storage) 10: a data base 20 stored in a magnetic disk or the like; a check point dump file (which will be shortly referred to as "CD file") 30; a log file for system recovery 40; and a log file for archives 50. The CD file 30 is used for periodically writing records from the main storage/virtual storage, which is to be recovered when the system is stopped by failure, against the possible failure while the system is running. The written record will be called the "check point dump record" (which will also be shortly referred to as the "CD information"). The recording medium is exemplified by a non-volatile storage device, usually DASD (i.e., direct access storage device) such as a magnetic disk. The CD information is usually written each time a predetermined number of log records for system recovery are written. The log file for system recovery 40 is a file having the log records necessary for recovering the system which is stopped by a failure. The log records necessary for the system recovery include an update log record of the database, an update log record of the resident table on main storage/virtual storage, a log record concerning input/output messages, and a log record indicating the completion/cancel of a transaction. A medium for the log file for system recovery 40 is a semiconductor disk which is prepared by making a semiconductor storage non-volatile with a battery. The log record necessary for the system recovery is basically a record logged after the writting of final effective CD information is started. If a sufficient file capacity (several times as large as the minimum necessity) for storing the log records within those ranges is retained, the log file for system recovery 40 can be repeatedly used by erasing the old record by the wrap-around method. The log file for archives 50 contains the log record for data base recovery, the log record for recording a business processing contents, and a log record to be used in a user program for a variety of business processings. The medium of archive log file to be used is the DASD such as a magnetic disk and a magnetic tape. The archive log record requires a long period of preservation and has to be finally copied and stored in a magnetic tape even in case it is stored in the DASD.

FIG. 1 is a diagram for explaining the operations of the data processing system according to one embodiment of the present invention. The operations of the present embodiment will be described in brief with reference to FIG. 1. In the present embodiment, a variety of log records accompanying the transaction processing during the system run are classified and .written in accordance with the objects of the log records. Specifically, the classifications are the log records for system recovery used in the system recovery, and the log records for archives used for data base recovery, for recording of business processing contents and for use in various batch application programs. The former records are stored in the log file for system recovery 40 upon completion of each transaction, and the latter records are stored altogehter for several transactions in the log file for archives 50. Incidentally, the log records for system recovery and the log records for archives need not be exclusive but may be partially common. The system operations to be described in the following are divided into the operation at the normal processing, the operation at the check point and the operation at the system failure.

FIG. 3 is a flow chart showing a partial routine of a normal transaction processing but according to the present invention. This routine shows the processing by the CPU/memory 10. If there arise during the transaction processing events such as an update of the data base, an update of the resident table on a main storage/virtual storage, input and output of a message, or completion of the transaction, the log records are written according to the processing flow shown in FIG. 3. The description will be proceeded in the following with reference to FIG. 3.

In case the log record to be written for the system recovery (as judged at the step 101), the processing is as follows:

Normally, a log record is not instantly stored in the file but temporarily in a buffer area on a memory for blocking to improve the performance. For the log record for system recovery, the buffer size is assumed to be about two or three times (e.g., 32 KB) as large as the average amount of the log record for system recovery of one transaction. At this step, it is judged whether or not a enough space to write the log record to be processed is in the log buffer for system recovery.

If it has been judged at the step 102 that the log buffer is full, the contents of the buffer are written in the log file for system recovery 40.

The log record to be written is put into the log buffers for system recovery.

In the case of completion of the transaction (including the end of transaction backout) and a request for buffer purge due to the update of the data base, the log record stored in the buffer has to be written in the log file for system recovery 40. At this step, it is judged whether or not the writing is necessary.

In the case of requirement for synchronous writing into medium at step 102, the log record stored in the buffer is written in the log file for system recovery 40.

In case it is judged (at the step 107) that the log record to be written is that for archives, the routine is as follows:

Like the log record for system recovery, it is judged whether or not the log buffer for archives is full. Incidentally, in the case of the log record for archives, the buffer size is at least several to several tens of times (e.g., 128 KB) as large as the written average amount of the log record for archives of one transaction.

In case the buffer is full in the step 108, the log record stored in the buffer is written in the log file for archives 50.

The log record for archives is put into the log buffer for archives.

There may be some cause for which the content of the log buffer for archives is to be written in the medium. At this step, this necessity is judged.

In case the purge to the medium is necessary at the foregoing step 111, the log records in the buffer are written into the log file for archives 50.

In the routines (a) and (b), it is determined at the initial setting of the system whether or not the various log records are written into either of the log files for system recovery and/or for archives.

In an on-line system or the like, with a view to shortening the time period for system recovery when the system is stopped by a failure, check points are provided at predetermined intervals during the system run so that the contents of the various data including the resident table on the main storage/virtual storage are written at the check points into a non-volatile storage. In the on-line system, moreover, the various summation data and the counters to be referred to or updated for the on-line processing or the data for administering the various kinds of status of terminals or lines are resident on the main storage/virtual storage so that their copies may be written at the check points in the non-volatile storage such as a disk. In the present embodiment, the data thus written are called the "check point dump information (or shortly the "CD information") and are stored in the CD file 30 on the DASD. When the system failure occurs, the data residing in the main storage/virtual storage is lost, they are recovered by the use of the update log records in the log file for system recovery 40. At this time, the log records before the start of CD information writing are basically unnecessary in the system recovery processing by using the CD information stored in the CD file 30 at the latest check point. The flow chart for processing at the check point is shown in FIG. 4. Incidentally, writing of the CD information is timed at the instant when the system is started, when the restarting processing is completed, when a predetermined number (normally, 1,000 to 10,000, as designated by the defining parameters at the initial system setting) of log records for system recovery are written into the log file for system recovery 40, or when the log file for system recovery 40 is reused from the head because the log records are written to the tail of the log file. Moreover, in order to provide against failures during the process of writing CD information, the CD file 30 is formed with a plurality of areas to store CD information, which are sequentially used. The description will be continued with reference to FIG. 4.

The following records (1) are written in the CD information area of the CD file 30.

(1) The record about the position of the first log record for system recovery, which is related to the transaction being executed at the beginning of CD information writing processing.

The resident information on the main storage/virtual storage are written out as the CD information in the CD information area of the CD file 30.

After the end of the CD information writing processing, the followign two processings are executed to validate the CD information written:

1 The content of the log buffer for system recovery is written to the log file for system recovery 40; and

2 Both the positional information of the final log record written by the above processing 1 in the log file for system recovery 40, and the serial number of the CD information writing processing are written to the area for CD information in the CD file 30. By the present processing, the CD information written is made the latest valid CD information.

Incidentally, the area for CD information may be formed in a plurality of mediums so as to provide against the medium failure of the CD file 30.

When the system is stopped by a failure, it is recovered by using both the log records in the log file for system recovery 40 and the CD information in the CD file 30. Of this system recovery processings, the processing relating to the present invention is schematically shown in FIG. 5. The operations will be described with reference to FIG. 5.

The serial number of the CD information writing processing in each CD information area in the CD file 30 is read to determine the latest valid CD information.

The CD information determined at the step 301 is fetched and expanded on the memory.

The log record for archives may partially fail to be written when the system is stopped by failure, because the information of the plurality of transactions is stored in the buffer on the memory. In case the recovery of some kinds of the log records for archives upon failure of logging is designated at the initial setting of the system, this system acquires the designated kinds of log records in the normal operations together with the log record for system recovery in the log record file for system recovery 40. At the present step, moreover, the log record for archives having failed to be written is recovered from the log file for system recovery 40 and written in the log file for archives 50.

The log record in the log file for system recovery 40 is read in the reverse direction from the latest written record up to the record at the beginning of the CD information writing processing (accurately the log record indicated by the positional information recorded at the step 301). The processed (complete/incomplete) status of each transaction is analyzed. As to the incomplete transaction and the transaction cancelled midway of the processing, the update data are undone by the use of the log record for system recovery. More specifically, as to the incomplete transaction and the cancelled transaction running in the CD information writing processing the update data may possibly be reflected upon the CD information, and the resident data on the memory are undone. In case, on the other hand, the data base is updated by the incomplete transactions, the update data are undone to the contents before update.

The log record in the log file for system recovery 40 is read in the forward direction from that at the beginning of the CD information writting processing to the latest written information and the results of update of the committed transactions are reflected upon the data base and resident information on the memory.

Thus, in the present embodiment, it is possible to improve the logging ability and to reduce the amount of storage of the log record while suppressing the increase in the number of storages. Incidentally, in the present embodiment, the logging ability can be better improved if the loads are dispersed like the prior art system among a plurality of systems for the log records for system recovery and for archives, respectively. In the present embodiment, moreover, it is possible to expect a drastic speed-up of the system recovery processing if a semiconductor disk or a high-speed device is used as the medium of the log file for system recovery 40. Incidentally, the embodiment thus far described is directed to an example of the recovery from the system failure. Despite of this fact, however, the present invention should not be limited thereto but may be applied to the recovery from the data base medium trouble, for example, in which the log record for recovery from the data base medium failure is written into one file whereas another log record is written into another file.

The effects of specific examples will be estimated in the following:

In the present embodiment, it is assumed that the log record necessary for data base recovery be written into both the log files for system recovery and for archives. As a result, the total amount of these two kinds of log records is more than that of the log records by the system of the prior art. The specific values will be enumerated in the following:

Total amount of the log record of the prior art system: 12 KB; Log record for system recovery: 8 KB; and Log record for archives: 6 KB.

Average access time: 0.7 ms; and Data transfer rate: 6 MB/sec.

Time for rotation: 16.5 ms; Data transfer rate: 3 MB/sec; and Track capacity: capable of storing ten physical records of 4 KB.

Prior art: logging into the magnetic disk by dividing it into a physical record of 4 KB for each transaction; and Invention: The log record for system recovery is divided like the prior art method and written into the semiconductor disk, but the log record for archives is written as a whole for ten transactions into the magnetic disk.

The throughputs of the logging methods according to the prior art and the present invention are determined. Incidentally, the amount of storage of the log record for each transaction is made, as follows, such that the amount of the present invention is reduced to one half of the prior art.

Method of the prior art: The whole log record of 12 KB is stored; and

Method of the invention: Only the log record for archives of 6 KB is stored.

The log records of one transaction is outputted for three physical blocks of 4 KB. Since each magnetic disk makes a turn of (1+1/10 of one record), the time period required is: 16.5.times.(1+1/10).times.3=54.45 ms, i.e., 18 transactions/sec.

The log records of one transaction is outputted for two physical blocks of 4 KB. Since each transaction requires an access time and a data transfer time of 4 KB, the time period required is: (0.7+4/6).times.2=2.7 ms, i.e., 370 transactions/sec.

For the total output of 60 KB of ten transactions, the magnetic disk requires 1 turn and 60/(4.times.10) turns for the data transfer of 60 KB, and the time period required for each transaction is: 16.5.times.(1+1.5)/10=4.13 ms, i.e., 242 transactions/sec.

From the items (1) and (2), the throughput obtainable according to the method of present invention is similar to that of the prior art method, in which the load is dispersed among systems.

As has been described hereinbefore, according to the present invention, the data processing system is enabled to improve the ability of writing the log records and to reduce the amount of storage of the log records without increasing the number of storages.