Coding scheme for file backup and systems based thereon

[0002] The present invention relates to a novel scheme which allows to protect against limited damage of stored data. In particular, data backup systems, and anti-virus protection systems (including virus disinfection systems) based on the novel scheme are addressed.

[0003] Error control coding (ECC) is a ubiquitous technique. Many electronic media suffer occasional losses or corruption of data; ECC is any data representation which is robust against such loss or corruption. Most ECC techniques are designed for protection of transmitted data, but the focus here is on protection of stored data.

[0004] Typically, coding is to protect against what is called physical errors. Examples are noise on a telephone line, defects on a disk or tape, or even the failure of an entire hard disk (as is protected by a RAID, a Redundant Array of Independent Disks). Protecting the data in such cases is done, first, by dividing it into blocks. Each block is coded independently. Since a code cannot protect against loss of many bits (or generally, symbols) in any block, the system must be designed so that bits in a block are more or less independent. (For example in the RAID, each bit of a coded block is stored on a separate disk. It is relatively unlikely that two disks will fail at once.)

[0005] Backing up computer files is typically done by making a duplicate copy of all the data. This means that the backup copy requires just as much space as the original, and to keep costs reasonable, backups are generally consigned to tape or another capacious but slow medium. Backups are so slow and inconvenient that they are often created and managed by a separate organization, and are often not worth using, except in case of catastrophe. Backups are extremely voluminous and time-consuming to make and unwieldy to organize.

[0006] In case of logical faults, it is relatively likely that all symbols in a single logical file will be corrupted or erased at the same time, in contradistinction to the physical fault model, discussed above. Computer viruses are one source of logical faults. The unit corrupted by a computer virus is not a disk track or similar physical unit, but a single computer file (usually an executable file). Another example is a user's own inadvertent erasure, change, or over-writing of a valuable computer file.

[0007] It is an object of the present invention to overcome the problems of known backup schemes and systems.

[0008] It is another object of the present invention to provide a scheme which allows to quickly and efficiently restore data in case of logical faults.

[0009] It is a further object of the present invention to provide a scheme which can be used for anti-virus protection.

[0010] The present invention concerns a method for protecting data against logical faults by means of error correction codes allowing to create vastly smaller backups than conventional systems. Under certain circumstances, it may also be employed against physical faults.

[0011] The present invention, in order to protect against corruption of logical units data (‘files’), employs coded blocks consisting of cross-sections across a number of files, rather than bits from within a single file.

[0012] For maximal efficiency (best mode), the data blocks should be cross-sections across all files, in which case the code blocks may be of variable length rather than the usual fixed length.

[0013] As with most ECC, limited corruption of the coded data results in no loss of original data, while corruption beyond this point may result in substantial loss.

[0014] There is a clear distinction between the present invention and most coding applications to transmission or even storage (including RAID in the latter category). Since the present invention provides for a separate integrity component to stand alongside an independent data system, the codes used must be systematic. To explain, in most applications the data are accessed only through the coded channel. For example, any RAID disk read/write must go through the RAID protocol. In this case the code need not be systematic, which is to say that the data bits need not be present ‘in the clear’ within the coded data. The data bits and parity bits may be all intertwined, and only get sorted out by the coding and decoding procedures.

[0015] According to the present invention, a systematic code is used which acts as an independent component to protect the data integrity of an ‘original’ system, and the data must be ‘in the clear’ for the original system. The systematic code preserves the original data bits. Please note that, unlike the data bits, there is no constraint on the parity bits, which are used only by the coding component and not the original system. In other words, when using a systematic code, the codeword still includes the original bits and any system processing or reading the original bits does not need to know what kind of coding scheme is used. It can simply take and process the original bits included in the codeword.

[0016] One implementation of the present invention uses an erasure code rather than an error-correcting code. An erasure code can restore bits that have been lost, while an error-correcting code can restore bits that have been lost or corrupted. Technically the difference between them is that the erasure code is ‘told’ which bits are bad, while the error correcting code is not given this information; the lesser task demanded of the erasure code allows it to operate more efficiently.

[0017] Another implementation may make use of an error-correcting code.

[0018] In the present applications, the ‘bad’ symbols are those in ‘bad’ files, which can be identified by a verification system to be described.

[0019] The present invention, and modifications thereof, allow realization of systems which efficiently protect against loss of data.

[0020] It is an advantage of the inventive scheme that it is fast and reliable. The storage space needed is much smaller than in case of conventional systems.

[0021] It is another advantage of the present invention that the size of the backup is linked to the size of the anticipated damage, and is completely independent of the total size of the files being protected.

[0022] It is a further advantage of the present invention that it provides protection against limited damage much more economically by employing an error-correcting code.

[0023] The invention also allows rapid update of the redundant coding data comprising the backup as new files are added or deleted. The restore operation is reasonably rapid, so recovery of damaged data is done quickly. The speed and compactness of the present invention allows backups to be done locally and constantly if needed so that the backup is always up to date. This allows putting the entire backup and restore mechanism, according to the present invention, under user control.

[0024] The present invention is well suited for use in portable computer systems, such as palm tops and notebooks.

[0025] The invention is described in detail below with reference to the following schematic drawings:

[0026] FIG. 1 Shows an example of blocks of message bits to be encoded consisting of cross-sections across N files.

[0027] FIG. 2 shows a system, according to the present invention, which is in a verified state with a complete set of parities computed. The files of this system all have the same length.

[0028] FIG. 3 shows a system, according to the present invention, which is in a verified state with a complete set of parities computed. The files of this system are of variable length.

[0029] FIG. 4 shows a system, according to the present invention, which is in an unverified state, with two files being damaged. The parities can be used to restore the two damaged files thereby returning the system to a state that will verify successfully.

[0030] FIG. 5A illustrates a first embodiment of the present invention where files are to be stored in a storage section.

[0031] FIG. 5B illustrates a first embodiment of the present invention where files have been stored in a storage section and a set of parity check bits (parities) has been computed and put in another storage section.

[0032] The expression files used hereinafter is meant to cover any kind of data files, records or other aggregation of data.

[0033] When hereinafter referring to “code”, any systematic code is meant to be addressed which is suited for erasure coding or error-correction coding.

[0034] The expression “message bits” is used in the standard sense of coding theory to refer to the information of interest to be stored or transmitted. Since the present invention applies to data storage rather than transmission, we shall use the terms “message bits” or “data bits” interchangeably. By contradistinction the expression “parity bits” is used in the standard sense of coding theory to refer to bits that are of no intrinsic interest, but are used to recover data bits in case of errors or erasures.

[0035] The present invention, in order to protect against corruption of data, employs coded blocks consisting of cross-sections across a number of files to be protected, rather than bits from within a single file. It is to be noted that these cross-sections across files do not have to be constructed explicitly in order to encode the files to be protected.

[0036] Error Correcting Codes (ECC) are described in “The Theory of Error Correcting Codes”, by F. MacWilliams and N. Sloane, North Holland, 1977, for example.

[0037] An example of blocks consisting of cross-sections 11 across N files 10 is illustrated in FIG. 1. If a virus, for example, now destroys any or all of file 12, an intact copy of the code blocks 11 would allow the restoration of the information originally contained in this file 12, as will be described in the following.

[0038] Instead of storing the blocks 21 being cross-sections of the files 20 to be protected (this would simply double the amount of storage space needed because basically the whole files are duplicated), according to the present invention, these blocks 21 are encoded using a systematic code. The encoding of these blocks 21 generates parity blocks 22 which can also be thought of (and optimally stored) as parity files 23, as schematically illustrated in FIG. 2.

[0039] There is a clear distinction between the present invention and most coding applications to transmission or even storage (including RAID in the latter category).

[0040] The present invention provides for a separate integrity component (the parity bits 22/23) to stand alongside an independent data system (e.g. the files 20) since systematic codes are used. The data (files 20) can still be read without requiring any knowledge about the coding scheme used for protection against data losses.

[0041] The invention works in two phases. An application-specific “verification” module determines whether the files (data) are in an acceptable state, or, if not, which files are damaged.

[0042] Upon such verification, the “coding” phase generates, or updates, encoded parity bits for the current, verified files (data bits). Alternatively, in the event that verification fails, decoding is used to restore the lost or damaged files with the assistance of the parity bits stored.

[0043] In a preferred implementation the inventive verification and encoding schemes are applied to the entire collection of files (set of files) or other logical units, but in general, they may be applied to subsets of them: those in a single directory, in arbitrary fixed-size collections, or in other subsets.

[0044] A preferred code would have the ability to update the parity bits very quickly if only a few data bits (files) have changed. In addition, a preferred code would have the property that the parity bits corresponding to all-zero data bits are very easily determined, for example the parity bits may also be all zeros.

[0045] For a code with the above properties, in a preferred implementation the encoding of a set of data files is computed by a sequence of “updates”. The parity bits begin as all zeros, corresponding to an implicit set of all-zero data files. One by one, the data files are implicitly “updated” from the implicit all-zero state to the actual value, and the parity bits are updated corresponding to this data update.

[0046] In the following, three implementations of the present invention are addressed.

[0047] First Implementation:

[0048] The first implementation is suited for anti-virus protection of files stored in a computer system, for example. A verification is a scan by an anti-virus program. The anti-virus program serves as “verification” module which allows identification of corrupted files. The files of the computer system are in an acceptable state if no viruses are found. If viruses are found, the virus-infected files are identified by the “verification” module (i.e. by the anti-virus program in the present example). Then, these files are restored by using parity bits computed and stored before these files were damaged (perhaps while they were stored). Details will be addressed later. If no viruses are found, the computer system is deemed to be in an acceptable state, and the parity bits may be updated or re-generated in their entirety, to reflect files that are new or changed since the previous verification.

[0049] Second Implementation:

[0050] This implementation is an elaboration of the first implementation, adapting it for a combination with the technique of generic virus disinfection. Generic disinfection is the repair of a file infected with a computer virus that is unknown or not specifically described. Details of generic disinfection are addressed in co-pending U.S. patent application with application Ser. No. 342,520, filed on Nov. 21, 1994, with title “Generic Disinfection of programs infected with a computer virus”. This U.S. patent application is herewith incorporated by reference. To compensate for lack of information about the virus, generic disinfection may require that some information about each file is saved prior to its infection. Typically, the amount of this so-called “redundant”information might be 50 bytes per file, but since each bit or byte can be treated separately, one can talk about it as if it were a single bit. The second implementation saves this redundant bit only in a virtual sense: the redundant data bits for all files are generated and from them a set of parity bits is computed by means of encoding, and then the data bits are discarded. In a preferred implementation the corpus of redundant information is never stored in its entirety, but rather the parity bits are computed as a series of code updates as, file by file, the redundant information is “updated” from an implicit all-zero state to the proper redundant information for that file. Thus if N data files are protected with p parities, instead of saving N bits (for example 50N bytes in total), the inventive scheme saves only p bits (or 50p bytes in total in this example), a much smaller number.

[0051] If viral damage is detected by a “verification” module, a sequence of steps is triggered. First, the 50-byte redundant information for damaged files is recovered by decoding the stored parity information and the 50-byte redundant information for the undamaged files. In a preferred implementation the 50-byte redundant information for the undamaged files is computed on demand, as it is required by the decoding procedure, so that the corpus of such redundant information never needs to be stored in its entirety. The 50-byte redundant information for each damaged file is supplied to the generic disinfection procedure along with the damaged file itself to repair the file. Finally, a fresh verification step may be performed by said “verification” module.

[0052] In other words, the use of the present invention in combination with a virus disinfection system only requires storing the associated backup information and not the 50 bytes of redundant information per file (please note that the number of 50 bytes is an example only, and is used for sake of simplicity).

[0053] If ever a file's generic disinfection is needed, what happens is this. The generic disinfection information is now computed for all uninfected files by the usual procedure. In combination with the stored parity “backup” information, this now allows the restoration of the generic disinfection information for the those files which were infected. Finally, this information is provided to the generic disinfection procedure to repair the infected files in its usual manner.

[0054] Third Implementation:

[0055] The third implementation is for limited backup and restore. If a user has inadvertently deleted a file, the present invention allows him to call for a verification that will report an unacceptable data state. This verification is done by an appropriate “verification” module. The respective files are then restored by decoding of parity bits stored before the inadvertent deletion occurred. Verification may be triggered by the user or automatically, for example at pre-set times, and can take into account user input (such as identities of files he believes he has destroyed) as well as any other evidence of filesystem integrity.

[0056] The encoding and decoding phases may be the same for any of these three exemplary implementations of the present invention. Upon unsuccessful verification, the parity bits are used to restore the data bits from damaged files (which may be thought of as being erased files).

[0057] Upon successful verification, new parity bits may be computed from the data bits (files) if an alteration or modification of these files has occurred or is likely to have been occurred. Please note that new parity bits (referred to as update) do not necessarily need to be computed whenever a verification was successful. This depends on the computing power available as well as on other circumstances.

[0058] Choices of Codes:

[0059] Any systematic error-correcting code or erasure code may be used in the present context. A preferred code is an erasure code

[0060] in which the number of parity bits is as small as possible compared with the amount of data that can be reconstructed;

[0061] in which the encoding to compute the parity bits is as quick as possible; and

[0062] in which the decoding to restore data bits can also be computed reasonably quickly.

[0063] In addition, a preferred code would have the ability to update the parity bits very quickly if only a few data bits (files) have changed.

[0064] The field of error-correcting codes is a vast one, and many codes might be employed for the present purpose. A code is a mapping f: {0,1}^N→{0,1}^N+Rin which N “message” bits of a file are converted to a larger number N+R of “message” (or “data”) bits and “parity check” (or “parity”) bits, with the property that if some of the N+R bits are lost or corrupted, the original N-bit message can still be recovered by decoding.

[0065] A code is called “systematic” if the original N message bits are among the N+R encoded bits. As mentioned above, the present invention requires the use of systematic codes. If a code is systematic, it is sufficient to save the R parity-check bits. If the code were not systematic, it would also be necessary to save the N bits of the codeword. This alone, contributing N encoded bits for every N original bits, would double the size of the data to be stored.

[0066] Another relevant property is whether the code is an “error-correcting code”or an “erasure code”. The preferred implementation of the present invention uses an erasure code, but the invention can also be used in connection with error-correcting codes. Error-correcting codes allow a limited amount of erroneous data to be corrected. Erasure codes are more limited in that for decoding (file restoration in the present context), they require that it be specified which data are in error. The name “erasure code” follows from an equivalence between the erasure of data and the knowledge of what data may be in error. If data is missing, it can be replaced with nonsense data (data known to be erroneous). If it is known that data may be in error, it can be erased. In either case erasure codes are appropriate to repair it.

[0067] In case erasure codes are used for the present invention, means are required which allow determination of whether files are in error. For example, a checksum of each file may be compared to saved checksums. Depending on the implementation, the “verification” module may require appropriate means to allow determination of whether files are in error or not.

[0068] A second property is whether the code repairs all possible errors of the allowed type (e.g. the erasure of any D files), or merely repairs most errors of that type (allowing recovery from the erasure of D files with high probability). The preferred implementation uses codes of the second type, which repair errors or erasures with high probability.

[0069] The time- and space-efficiency of the code are important. If there are N files altogether and the code is to protect against the erasure of up to D files, the question is what the size R of the parity check data is. On information-theoretic grounds R≧D. In a preferred implementation R is about two times D. In other codes, especially error-correcting codes, R can depend on N, which is likely to create overly bulky codes.

[0070] In one implementation of the present invention, the ith bits of all N files being backed up define the N message bits of a codeword. These codewords are herein also referred to as cross-sectional data blocks. The R parity check bits (also referred to as set of parity symbols or simply parities) of that codeword are saved as the ith bits of what are conceptually R backup files. In practice, these may in fact be saved as R files (see 23 in FIG. 2, for example), as we shall assume for simplicity in the following, or they may be combined into one file. If up to D of the data and parity files are erased (or up to D files are damaged and it can be determined which ones), then the inventive scheme allows the data files to be restored, using the surviving N+R−D files. (Depending on the code employed, it may be required that erasures or errors can only occur to the data files, and not to the parity files, and this may be an acceptable constraint if the parities can be stored in a very reliable manner.) The code to be used for the present implementation uses only the exclusive-or (XOR) operation on bits, and may perform this operation on a byte, word, or other unit of bits at once, for increased efficiency.

[0071] Because conceptually a cross-section of the original files is taken (the ith bits of all N files are treated together), according to the present invention, there is an implicit assumption that all files have the same length.

[0072] In cases where files 30 are of variable length, the code blocks may be of variable length rather than the usual fixed length. An example is illustrated in FIG. 3. The file 34 is longer than file 35, for example. Due to this, the blocks consisting of cross-sections across a number of the files 30 are of different length. Please note that block 31 is longer than block 36, for example. The inventive scheme can handle such blocks of different length.

[0073] In case that blocks of different length are not desired or accepted, it is possible to treat all files as if they were of a fixed length. For encoding and decoding purposes, bits past the true end of shorter files are then treated as zeros. Longer files may be ignored, in which case they would not be recoverable by means of the present invention. Alternatively, longer files may be broken into blocks of the fixed length, the last block padded with zeros, if necessary. In this case, the invention's erasure tolerance will apply to a number of blocks rather than files, and the inventive scheme will be able to restore erased data if, for example, a small number of large files is deleted, or a large number of small files.

[0074] One code which could be used to implement the present invention is described by Michael G. Luby, Michael Mitzenmacher, M. Amin Shokrollahi, Daniel A. Spielman, and Volker Stemann in the article “Practical Loss-Resilient Codes”, appearing in the Proceedings of the 29th Annual ACM Symposium on the Theory of Computing, 1997.

[0075] For whatever code is to be used in connection with this invention, one wishes the amount of redundancy (the number of parity bits) to be small, and encoding to be fast. In particular, if encoding the data on a disk, it is desirable to be able to update the coded words in constant time when a single bit of data is updated. One implication of this taxing constraint is that each original bit of a file affects on average only a constant number of “parity bits”. Finally, decoding must be practicable, although if error events are rare, it need not be as fast as encoding. Decoding time linear in the size of the total data volume is acceptable.

[0076] In a preferred implementation of the invention, the code used is a “two-tree” binary code, presented in the following, which has all the desirable properties enumerated above. It guards N-bit words against a small number D=o(N) of random erasures, using only 2D+o(D) parity check bits. Encoding takes linear time, O(N), while updates take time O(1) when a single message bit is changed. Decoding is almost always successful, takes time O(N) in the worst case, and takes expected time O(NE/D) if only E bits were erased.

[0077] We note that while coding against random erasures can be accomplished with O(D) parity check bits, any code protecting against arbitrary erasures, random errors, or arbitrary errors provably requires Θ(D lg(N/D)) parity bits.

[0078] In addition to being compact, the two-tree codes for the random erasure case are systematic, linear, and sparse: the message bits (or data) of the codeword are the original bits, the parity bits of the codeword are a linear function of the original bits, and the matrix describing this linear function is sparse.

[0079] The two-tree code for random erasures detailed in the following is strikingly simple. It is an elaboration of a “tree code”, also described in the following, whose data bits are edges between parity-bit vertices, and where the value of a “parity” vertex is the exclusive-or of all its incident “message” edges. Decoding is accomplished by identifying the forest induced by the erased bits, recovering from the values of leaf vertices the values of their incident edges, then stripping away these leaves, repeating until there is nothing left.

[0080] Sparse Linear Codes:

[0081] A binary linear code is described by a N×(N+R) generator matrix G of bits, so that the codeword x corresponding to a word u is x=uG. Such a code is also systematic if one may write G=[I|G′], the concatenation of the identity matrix and a N×R matrix G′, so that x is the concatenation (u, uG′) of the message bits u and parity bits uG′.

[0082] In the following, the efficient codes that may be used in connection with the present invention are introduced.

[0083] An R-bit “tree code” is a systematic binary linear code with generator matrix [I|G′], where G′ is an N×R matrix having 2 ones in each row, and zeros elsewhere, and the ones are positioned randomly and independently for each row.

[0084] One may think of G′ as the incidence matrix of a random graph with N edges (the message bits) on R vertices (the parity bits). Under certain circumstances, if the errors that occur are themselves random, it may be possible to relax the requirements on the randomness of the code.

[0085] Theorem 1:

[0086] The encoding for a tree code can be computed in time O(N).

[0087] Method: In the graph view, the value of each parity bit (vertex) is the sum (modulo 2) of its incident message bits (edges). If G′ is represented, for example, as a linked list of the edges incident on each vertex, these sums can be computed in time linear in the number of edges, which is exactly N.

[0088] Theorem 2:

[0089] Suppose that in a codeword x of a tree code given by G′, at most D message bits are erased, and no parity bits are erased. Further suppose that the subgraph of the incidence matrix of G′ given by the erased parity bits (edges) is a forest. Then the message u, with x=[I,G′]u, can be decoded correctly, in time O(N), and in expected time O(ND/R).

[0090] Method: In the full graph G′, all vertices (parity bits) have known values (by assumption none were erased), all edges have values 0, 1, or unknown, and the value at each vertex is the sum (modulo 2) of the values (known or unknown) on its incident edges.

[0091] This “summation property” is preserved if each known edge value is subtracted from (equivalently, added to) the values of its incident vertices, and then deleted from the graph. This can be done in time O(N). What remains then is the graph of “erased” message bits (edges with unknown values) on the parity bits (vertices with known values). By assumption it is a forest.

[0092] A leaf-edge's value can be decoded immediately as the value of the leaf. Subtracting the edge's value from its two incident vertices, and deleting it from the graph, preserves the summation property. This can be done in time O(R), for a total time of O(N+R)=O(N).

[0093] Expected time O(kb/r) is achieved if instead of deleting all known-value edges initially, we only delete edges as needed. Specifically, it is only necessary to delete known-value edges which share vertices with edges of unknown value. Since there are D edges of unknown value, there are at most 2D vertices incident upon them. The average degree of a vertex is 2N/R, so the expected number of known-value edges on the selected vertices is at most 4ND/R.

[0094] Suppose that in coding for correction of random erasures, it is given that no parity-check bits will be erased. For example, in an application where we are to protect stored data, perhaps the parity-check bits can be stored separately, in a high-integrity medium.

[0095] Theorem 3:

[0096] Let c<½ be a constant. From a codeword x from a tree linear code with N message bits and R parity bits, let cR message bits be erased at random. Then all but o(R) of the bits of x can be recovered, with probability tending to 1 as R→∞.

[0097] Proof: The subgraph of erased message bits is likely to be dominated by a forest, as follows from elementary facts about random graphs; see for example E. M. Palmer's “Graphical Evolution”, Wiley, 1995. The Theorem then follows from Theorem 2.

[0098] Now, one just augments the code so that one will be able to pick off the remaining bits of a file.

[0099] An (R, R′)-bit two-tree code is a binary systematic linear with generator matrix [I|G′], where G′ is the concatenation of an N×R matrix G1 having 2 ones in each row, zeros elsewhere, and an N×R′ matrix G2 also having 2 ones in each row, zeros elsewhere, and the ones are positioned randomly and independently in each row of G1 and G2.

[0100] Theorem 4:

[0101] Let c<½ be a constant. From a codeword x from a two-tree code with N message bits and R+R′ parity bits, let cR message bits be erased at random. Then all the message bits can be recovered, with probability tending to 1 as R→∞ and R′→∞. The decoding takes time O(N), and if E≦cR message bits are erased, takes expected time only O(NE/R).

[0102] Method: From the first set of r parity bits, i.e. the matrix G1, with probability tending to 1 all but ω(R)=o(R′) of the erased message bits can be recovered as previously described. From the second R′ parity bits, i.e. the matrix G2, with probability tending to 1, the same method recovers absolutely all the remaining ω(R) bits. The fact that the second phase can with high probability recover absolutely all the remaining erased bits, not just almost all of them, follows from the fact that the number of such bits is o(R′), and from elementary facts about sparse random graphs; see for example E. M. Palmer's “Graphical Evolution”, Wiley, 1995.

[0103] As our final remark on the two-tree codes, we note that under certain constraints on the number of message bits, the number of parity bits, and the erasure probability, the same methods will with high probability recover all the message bits when random erasures occur over the entire codeword, i.e. over both the message bits and the parity bits.

[0104] Further implementation details:

[0105] As mentioned, systematic codes are used to generate the parity bits. There is no constraint on the number of parity bits, because these bits are used only by the encoding and decoding component and not the original system which still can read the files, or data thereof, without having to know anything about the codes and coding scheme used.

[0106] As explained above, either an erasure code or an error-correcting code can be used.

[0107] An inventive system for storing files and parity bits for later recovery is illustrated in FIGS. 5A and 5B. A set 50 of files 52 which are to be stored, is shown on the upper left hand side of FIG. 5A. The system comprises an encoding/update unit 55, a first storage section 56, and a second storage section 57. These two storage sections may be provided within one and the same storage device (e.g. a disk drive system) or the two sections may be provided on different systems. If no files are stored in the first storage section 56, this storage section is padded with zeros, as indicated by the blank boxes 60. Also the parity bits are set to zeros, as indicated by the blank boxes 59.

[0108] The files 50 to be stored are now fed via line 51 to the encoding/update unit 55 where they are processed. These files 52 can either be fed into the unit 55 sequentially or in parallel. Depending on the coding scheme used, registers may be employed in unit 55 to store all or part of the files while being encoded. As described above, cross-sections of the files 50 are encoded using a systematic code. As a result of this encoding process, a set of parity bits is generated which is stored via line 54 into storage section 57, as illustrated in FIG. 5B (the grey boxes 61 are the parity bits generated). At the same time, the files 50 are forwarded via line 58 to the first storage section 56, as shown in FIG. 5B (the black boxes 62 correspond to the files 50). Please note that there is a feedback line 53 between the section 57 where the parity bits are stored and the encoding/update unit 55. If an update of the files 62 is required, e.g. because some files were corrupted, or if part of these files have been intentionally modified, for example, the parity bits 61 and the files 62 are fed back to the encoding/update unit 55 for decoding and updating. These updated files are then written into the storage 56. Please note that, due to the fact that systematic codes are used by the encoding/update unit 55, the files 62 in storage section 56 can be read as usual. I.e. these files 62 are stored in a form “in the clear”.

[0109] In a preferred implementation the encoding and update are done by one and the same unit. Please note that the encoding part and the update part of the encoding/update unit 55 can also be implemented separately. Under certain circumstances the encoding (e.g. the initial encoding) can be done in a different way than the updating.

[0110] Verification modules are addressed in the following. A verification module is employed to determine whether the stored files (data) are in an acceptable state, or, if not, to identify which files are damaged. The verification module is application-specific. There are many different way to realize such a verification system. It may either be implemented in hardware, in software (e.g. anti-virus software), or as combination of software and hardware. The module may comprise a block which detects viral damage, for example. In another implementation, the verification process may be triggered by the user, if he/she has inadvertently deleted a file, for example. In turn, the verification process will report an unacceptable data state.

[0111] Optionally, a verification may be triggered automatically, for example at preset times. It may take into account user input as well as any other evidence of filesystem integrity.

[0112] A verification module may be employed which calculates the checksum of all files stored. These checksums are then saved for later use. If verification is needed, the checksum of each file may be compared to said saved checksums. As a result, a list, table of files or pointers to files which seem to have been damaged or lost is output by the verification module. As illustrated in FIGS. 5A and 5B, the verification module (not shown) may be connected to line 63. The verification process provides the required input to the encoding/update unit 55.

[0113] Upon such verification, the “coding” phase generates, or updates, encoded parity bits for the current, verified files (message bits). Alternatively, in the event that verification fails, those files which were identified by the verification module as being damaged or lost are restored as follows. Those data files and parities which are intact (which are not lost or corrupted) are decoded to generate the corrected data files. These files are then written back into the storage section.

[0114] The present invention and modifications thereof, allow realization of systems which efficiently protect against loss of data.

[0115] It is an advantage of the present invention that it is fast and reliable. The storage space needed is much smaller than in case of conventional backup systems.

[0116] The present invention can also be used if data loss is caused by independent events, such as cosmic rays striking the storage medium where the files are stored.