Queryable cache in real-time data management system

[0001] The present invention relates to the field of real-time data management infrastructure. More particularly, the present invention relates to a queryable cache in a real-time data management system.

[0002] Data warehousing involves the gathering, storage, and retrieval of large amounts of information. In the modern age, it is common to perform the data acquisition aspect of data warehousing in real-time, where streaming information may be placed into databases as it arrives. This is especially useful in areas such as manufacturing, where sensor information from various machines may arrive simultaneously and at a high rate of speed.

[0003] Simply acquiring the data in real-time, however, is only part of the equation. It is also beneficial to have the data stored in a way that makes it easy to mine. Relational databases are often used for storage, but the organization of the data in the relational database can be critical to efficient mining later. It therefore becomes a priority to not only store the data in real-time, but to store it in a proper format for easy mining.

[0004] Additionally, relational databases are typically not used for real-time data analysis. The delay in actual storage times into the relational database, and the complexity of the storage design, typically makes it impractical for real-time analysis of incoming data. In the manufacturing world, however, real-time data analysis can be critical in keeping the production line as efficient as possible. For example, in a microchip fabrication plant, it would be beneficial to have a “feedback loop”-type system where information from sensors examining the production of one portion of the chip may be used in real-time to modify or delay the production of another portion of the chip (or another chip entirely). This allows a manufacturer to correct for deficiencies that might have otherwise resulted in an expensive loss. The feedback loop would aid manufacturers in getting their yield up.

[0005] Unfortunately, this problem has proved difficult to solve, as the vast amount of data gathered in these types of systems in a short amount of time make scaling of any solution a major impediment.

[0006] Previous solutions have attempted to solve the problems mentioned above by utilizing a name-value schema to quickly store real-time information into persisted memory. While such solutions are indeed able to achieve very fast save rates, queries to the data are very slow, making them unusable for the real-time data analysis described above.

[0007] What is needed is a solution that overcomes these deficiencies.

[0008] In order to provide real-time data analysis of high speed data, a query control mechanism may be provided and coupled to one or more caches. The caches may temporarily store the incoming high-speed data, and the data from the caches may be periodically swept into a relational database. Upon receipt of a query for real-time data, the query control may determine if one of the caches can fulfill the query. If so, the query is transferred to that cache. If not, the query may be transferred to the relational database. The query control can also combine information from the cache and the relational database. The query control can accomplish this by tracking the time frame of information stored in each cache, such that it is aware of whether or not a particular cache can fulfill a query without first contacting the cache.

[0014] Embodiments of the present invention are described herein in the context of a system of computers, servers, and software. Those of ordinary skill in the art will realize that the following detailed description of the present invention is illustrative only and is not intended to be in any way limiting. Other embodiments of the present invention will readily suggest themselves to such skilled persons having the benefit of this disclosure. Reference will now be made in detail to implementations of the present invention as illustrated in the accompanying drawings. The same reference indicators will be used throughout the drawings and the following detailed description to refer to the same or like parts.

[0015] In the interest of clarity, not all of the routine features of the implementations described herein are shown and described. It will, of course, be appreciated that in the development of any such actual implementation, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the art having the benefit of this disclosure.

[0016] In accordance with the present invention, the components, process steps, and/or data structures may be implemented using various types of operating systems, computing platforms, computer programs, and/or general purpose machines. In addition, those of ordinary skill in the art will recognize that devices of a less general purpose nature, such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like, may also be used without departing from the scope and spirit of the inventive concepts disclosed herein.

[0017] The present invention utilizes a cache system which is queryable in real-time in a data management system in order to allow the real-time data analysis of high-speed data. The present invention also has the capability to simultaneously store the data in a relational database in a format that makes mining operations easier.

[0018] FIG. 1 is a block diagram illustrating a data management system in accordance with an embodiment of the present invention. One or more caches 100, 102, 104, may be provided to temporarily store the incoming real-time high-speed data from various machines 106, 108, 110. In an embodiment of the present invention, each cache is linked to a specific machine, but one of ordinary skill in the art will recognize that there could two or more machines could share a cache, or two or more caches may be provided for different input streams from a single machine.

[0019] As incoming data is received by the caches, the data may be immediately persisted into backup files 112, 114, 116, to guard against power failures and other significant problems that would result in the loss of data in the cache. Each of the caches 100, 102, 104, may be coupled to a storage coordinator 118. The storage coordinator 118 may act like a funnel, slowing down the writing of information to the relational database 120 to an efficient block transaction that can be handled by the interface to the relational database. The storage coordinator 118 may also precondition the data so that it is an format that is easy to mine. This greatly improves the speed at which a user may mine the data at a later time. This preconditioning will be described in more detail later.

[0020] In order to provide real-time data analysis, a query control 122 may be coupled to each of the caches 100, 102, 104. In an embodiment of the present invention, the query control 122 may keep track of the time frame of information stored in each cache. For example, cache 100 may contain the last 3 seconds of real-time data, whereas cache 102 may contain only the last 1 second of real-time data. The query control 122 may then determine when a query request arrives whether or not the query can be satisfied with information in one of the caches, or if it needs to go to the database (or a combination of the two). For example, if the query control 122 receives a request for information from machine 106 that is 2 seconds old, then the query control 122 is able to determine that such a request can be satisfied by cache 100 as cache 100 corresponds to machine 106, and it holds the last 3 seconds of real-time data. It therefore may serve the query request on the cache 100. However, if a similar request came in for machine 108, it would know that the request cannot be satisfied by cache 102, even though cache 102 corresponds to machine 108, because cache 102 only holds the last 1 second of real-time data. It therefore knows it needs to instead query the relational database 120.

[0021] In another embodiment of the present invention, each cache may have an associated event cache. Events are one type of data received from the data source. Events are valid for a moment in time, for example start job and end job. These discrete events may have been received from the input source in addition to the time-series “trace” data (gas pressure, for example). The query control can use the event identifications to look up time markers that can then be used to look up data in the cache or relational database, such as return all gas pressure data between start job and end job.

[0022] The time frame of information stored in each cache may be tracked by the query control 122 using time stamps pointing to offsets in each file. The timestamps may indicate the starting time of each block and the ending time of each block.

[0023] By passing query requests that cannot be fulfilled by the caches to the relational database, the present invention greatly reduces the number of queries sent to each cache. This reduces the load on the caches and enables queries that can be fulfilled by the cache to be fulfilled in real time.

[0024] The information from the query may be utilized by a process control mechanism fault detection and classification (FDC), or advanced process control (APC) 124, for real-time analysis, which may pass the information to some sort of machine control tool, such as Manufacturing Execution Systems (MES) 126. This may then be a part of a feedback loop back to the machines 106, 108, 110 to improve yield.

[0025] Referring back to the preconditioning of data by the storage coordinator 118, because the system does not know the names of the various pieces of the data ahead of time it is necessary to make the preconditioning dynamic. In an embodiment of the present invention, this is accomplished by creating 3 tables, a string table, a floating point table, and an integer table. These make it easier to create an efficient data structure. Each of these 3 tables may be organized as a two dimensional table having time on one axis, the name on another axis, and the value of the data in the body of the grid. An example of this is illustrated in FIG. 2. Separately, a mapping layer may then be built that indicates which name gets stored in which column of which table. This structure allows a user to mine the data quickly by referring the mapping layer first, before going directly to the appropriate column of the appropriate table.

[0026] FIG. 3 is a flow diagram illustrating a method for managing data from one or more high-speed source in accordance with an embodiment of the present invention. Each act in this method may be executed by hardware, software, or any combination thereof. At 300, the data may be stored in one or more caches. In one embodiment, each of the caches corresponds to a single data source, for example, a manufacturing machine having sensors. At 302, the data may be backed up from the one or more caches into one or more persisted backup storages coupled to each of the caches. At 304, the storage of the data from the one or more caches into a relational database may be coordinated. This may be done on a periodic basis, and may include parsing the data into a string table, a floating point table, and an integer table, and creating a mapping layer indicating which portions of the string table, floating point table, or integer table contain the data. The string table, floating point table, and integer tables may each contain two axes and a body, wherein one of the axes corresponds to time and a second of the axes corresponds to name, wherein the body contains values for the data for corresponding name/time combinations.

[0027] At 306, a query for real-time data may be received. At 308, it may be determined if the query for real-time data can be fulfilled by information in one of the one or more caches. This may include examining information as to the time frame of information being stored in each of the one or more caches and determining if a time indicated by the query is inside the time frame of information for a cache corresponding to a data source indicated by the query. If so, then at 310 the query may be transferred to the one of the one or more caches that can fulfill the query. If not, however, then at 312, the query may be transferred to the relational database.

[0028] While embodiments and applications of this invention have been shown and described, it would be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts herein. The invention, therefore, is not to be restricted except in the spirit of the appended claims.