Processor configured to perform a batch process on structured data

[0001] This application is a continuation of U.S. application Ser. No. 17/860,549 filed Jul. 8, 2022, now U.S. Pat. No. 12,045,230, which, in turn, is a continuation of U.S. application Ser. No. 16/909,052 filed Jun. 23, 2020, now U.S. Pat. No. 11,386,083, both of which are incorporated by reference herein.

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62/887,577 filed Aug. 15, 2019, the entire disclosure of which is incorporated by reference herein.

[0003] Portions of the documentation in this patent document contain material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office file or records, but otherwise reserves all copyright rights whatsoever.

[0004] A new batch processing method is disclosed using relational database tables without the sequential reading of data. Preferred embodiments of the present invention make processing large datasets in a batch process using tables possible and it also eliminates the need for using a cursor processing method available in Relational Database Management System (RDBMS). The new method described herein is 100 to 300 times faster than the currently available processing method when used in column store databases. In fact, random batch processing is the only viable option when processing large datasets with billions of rows in tables when an SQL process is required to mimic a batch process written in COBOL or any other programming language.

[0005] Currently there is only one method available to process data in tables in batch/sequential mode. The cursor processing method fetches data row by row, sorted by key, from one or multiple tables, applies the business rules, updates the current row, and then fetches the next row in the sequence. The cursor processing method is an extremely slow process. Tests with the cursor method indicate that the processing rate is only 10,000 to 20,000 rows per minute. At this rate, processing tens of billions of rows would be impossible within a reasonable timeframe. Accordingly, there is a need in the art for other options to process large datasets in tables. The present invention fulfills this need.

[0006] The batch processing method described herein is easy to develop with simple SQL skills. The method can be up to 300 times faster than the currently available processing method when used in column store databases, depending on the total number of accounts/keys to process and the number of transactions per account or key.

[0007] Random batch processing enables organizations to migrate all of their batch processes to a database centric process with greater performance than legacy systems. Legacy systems are expensive to maintain and human resources available to maintain them are diminishing in supply.

[0008] To convert any legacy system to a database centric process, the flat files in the legacy system need to be converted to tables using any of the extract, transform, load (ETL) tools and then apply the method described herein. In one example use of the present invention, a legacy system that processes 15 billion rows every month on mainframe was converted to a database driven process in Sybase IQ database.

[0013] Certain terminology is used herein for convenience only and is not to be taken as a limitation on the present invention. The words “a” and “an”, as used in the claims and in the corresponding portions of the specification, mean “at least one.”

[0014] One preferred embodiment of the present invention is implemented via software code shown below. The software code is subject to the “Copyright Notice and Authorization” stated above.

[0015] Every batch process has at least one master file (MF) and a transaction file (TF), or two or more transaction files.

[0016] This is true for tables as well. Master files and transaction files must be sorted in the same order on the same set of keys for successful matching. Every row in the master file is paired with every row in the transaction file. If a match is found, it is processed and updated based on the business rules. Sequential reading is essential for this process so that the balance from the current row will be carried over to the next row to be processed. The method described herein is the only way to do this process without reading one row at a time from both the master and transaction files (or tables). The method described herein processes one row per account/key simultaneously.

[0017] The principle idea in the batch process described herein is that an additional key is added within each natural key on both tables/files for identifying the sequence of the rows in MFs and TFs. During each loop through the process, the sequence to match on MFs and TFs is recalculated.

[0018] Referring to FIG. 1, the following steps disclose one preferred embodiment of the process. Similar results can be achieved with less or more steps, as long as the principle idea of adding additional keys to identify the order is accomplished. Details for these steps depend on the requirements of the process but they are similar in nature for all MF and TF processing.

Step 1₁

[0019] Add a unique key to the master table, matching the order to the rows to be processed and insert the result to a temporary table This SELECT must have a condition to eliminate the rows that are already processed. The example below is the BALANCE_AMT.

[0020]

SELECT ACCOUNT_KEY, COL1, TRANS_DT, COL3, TRANS_BALANCE_AMT,
...ROW_NUMBER( )OVER (PARTITION BY ACCOUNT_KEY,TRANS_DT..) SEQ INTO
#TEMPDEBIT_INIT_SEQ_ FROM MF_DEBIT_TABLE WHERE ASSMNT_AMT > 0 AND
TRANS_BALANCE_AMT > 0.

[0021] In this example, the SQL Windows function is used to generate a row number for the sequence of rows to process the order.

Step 2₁

[0022] Add a unique key to the transaction table, matching the order the rows to be processed, and insert the result to another temp table. This SELECT also must have a condition to eliminate the rows that are already processed. The example below is the BALANCE_AMT. Note that the balance_amt is stored as ‘-’ in the transaction table and that the <0 indicates that a credit amount exists in this account.

[0023]

SELECT ACCOUNT_KEY, COL1, TRANS_DT, COL3, TRANS_BALANCE_AMT,
...ROW NUMBER( )OVER(PARTITION BY ACCOUNT_KEY ORDER BY
ACCOUNT_KEY,TRANS_DT..) SEQ INTO #TEMPCREDIT_INIT_SEQ FROM
TF_CREDIT_TABLE WHERE PAYMENT_AMT *-1 > 0 AND TRANS_BALANCE_AMT <
0.

[0024] In SQL Windows functions, PARTITION BY is done on the natural key of the data. ORDER BY columns are based on business rules for match order. It is very important for any batch process to have the same key columns and same sort columns for both MF and TF tables/files.

Step 3₁

[0025] Remove the rows from the TF temp table if the account (or key) is not in the MF temp table and also remove the rows from the MF temp table if the account is not in the TF temp table. You need to select the accounts/keys which have at least one row to process in both tables.

[0026]

DELETE FROM #TEMPDEBIT_INIT_SEQ WHERE ACCOUNT_KEY NOT IN (SELECT
ACCOUNT_KEY FROM #TEMPCREDIT_INIT_SEQ).
DELETE FROM #TEMPCREDIT_INIT_SEQ WHERE ACCOUNT_KEY NOT IN (SELECT
ACCOUNT_KEY FROM #TEMPDEBIT_INIT_SEQ).

[0027] The remaining steps in this process depend on the requirements (business rules). One may or may not need all the steps below in all batch process programs. The steps above already identified the rows to match/update in the process by adding additional key (SEQ) within every account. If one joins the two temp tables above with the natural key (ACCOUNT_KEY) and the SEQ generated by the SQL PARTITION BY function, this will get only one row (if one uses AND SEQ=1 in the condition). This will be the first row to be processed in each account. The following steps demonstrate how this idea applies to business rules in a specific batch process requirement.

Step 4₁

[0028] Select all rows from both temp tables where SEQ=1 and insert it into new temp tables. Keep a copy of these tables to store the original amounts after the first tables are updated.

[0029]

SELECT * INTO #TEMPDEBIT_SEQ FROM #TEMPDEBIT_INIT_SEQ WHERE SEQ = 1;
SELECT * INTO #TEMPCREDIT_SEQ FROM #TEMPCREDIT_INIT_SEQ WHERE SEQ = 1;
SELECT * INTO #TEMPDEBIT_SEQ_COPY FROM #TEMPDEBIT_SEQ;
SELECT * INTO #TEMPCREDIT_SEQ_COPY FROM #TEMPCREDIT_SEQ;

Find the row counts in both tables. This is to control the number of loops later. SET @DEBITCOUNT=(SELECT COUNT(*)FROM #TEMPDEBIT_SEQ); SET @CREDITCOUNT=(SELECT COUNT(*)FROM #TEMPCREDIT_SEQ);
Step 5₁

[0030] Perform all updates to the #TEMPDEBIT_SEQ and #TEMPCREDIT_SEQ tables based on the business rules. Use copies of the tables if needed for applying the business rules.

[0031]

UPDATE #TEMPDEBIT_SEQ SET D.TRANS_BALANCE_AMT =
D.TRANS_BALANCE_AMT + C.TRANS_BALANCE_AMT FROM #TEMPDEBIT_SEQ D,
#TEMPCREDIT_SEQ_COPY C WHERE D.EMIS_TIF_KEY = C.EMIS_TIF_KEY =
C.EMIS_TIF_KEY AND D.SEQ = C.SEQ AND D.TRANS_BALANCE_AMT >= -1*
C.TRANS_BALANCE_AMT;
COMMIT;
UPDATE #TEMPCREDIT_SEQ SET C.TRANS_BALANCE_AMT = 0,
C.CREDIT_FULLY_USED = ‘Y’ FROM #TEMPCREDIT_SEQ C,
#TEMPDEBIT_SEQ_COPY D WHERE C.EMIS_TIF_KEY = D.EMIS_TIF_KEY AND
C.SEQ = D.SEQ AND D.TRANS_BALANCE_AMT >= -1*C.TRANS_BALANCE_AMT;
COMMIT;
UPDATE #TEMPCREDIT_SEQ SET C.TRANS_BALANCE_AMT =
C.TRANS_BALANCE_AMT + D.TRANS_BALANCE_AMT FROM #TEMPCREDIT_SEQ
C, #TEMPDEBIT_SEQ_COPY D WHERE C.EMIS_TIF_KEY = D.EMIS_TIF_KEY AND
C.SEQ = D.SEQ AND C.TRANS_BALANCE_AMT <> 0 AND C.TRANS_BALANCE_AMT
*-1 >= D.TRANS_BALANCE_AMT;
COMMIT;
UPDATE #TEMPDEBIT_SEQ SET D.TRANS_BALANCE_AMT = 0,
D.DEBIT_FULLY_PAID = ‘Y’ FROM #TEMPDEBIT_SEQ D, #TEMPCREDIT_SEQ_COPY
C WHERE C.EMIS_TIF_KEY = D.EMIS_TIF_KEY AND C.SEQ = D.SEQ AND
C.TRANS_BALANCE_AMT <> 0 AND C.TRANS_BALANCE_AMT *-
1>=D.TRANS_BALANCE_AMT;
COMMIT;

Step 6₁

[0032] Use natural keys only to update the original tables in the database with the data in the temp tables above.

[0033]

UPDATE MF_DEBIT_TABLE SET TRANS_BALANCE_AMT =
TEMPD.TRANS_BALANCE_AMT FROM MF_DEBIT_TABLE D, #TEMPDEBIT_SEQ
TEMPD WHERE D.EMIS_TIF_KEY =
TEMPD.EMIS_TIF_KEY AND D.TRAN_SEQ = TEMPD.EMIS_TIF_KEY AND
D.TRAN_SEQ = TEMPD.TRAN_SEQ;
UPDATE SET TRANS_BALANCE_AMT = TEMPC.TRANS_BALANCE_AMT FROM
TF_CREDIT_TABLE C, #TEMPCREDIT_SEQ TEMPC WHERE C.EMIS_TIF_KEY =
TEMPC.EMIS_TIF_KEY AND C.TRAN_SEQ = TEMPC.TRAN_SEQ;

One row in each account in the MF table is processed if there is a Transaction row in the TF table for the account/key. If there are 30 million accounts/keys in the transaction table, 30 million transaction rows are processed from the TF and MF tables simultaneously.
Step 7₁

[0034] Drop all #temp tables created.

[0035]

DROP TABLE #TEMPCREDIT_INIT_SEQ; DROP TABLE #TEMPDEBIT_INIT_SEQ;
DROP TABLE #TEMPCREDIT_SEQ; DROP TABLE #TEMPDEBIT_SEQ;
DROP TABLE #TEMPCREDIT_SEQ_COPY; DROP TABLE #TEMPDEBIT_SEQ_COPY;
Loop through steps 1 to 7 until there are no more rows to process in the MF or TF table (until
@CREDITCOUNT = 0 or @DEBITCOUNT = 0).

[0036] The number of rows to process decreases in every loop of the process. The performance of this process and the benchmark results for each loop are listed below for test dataset tables in a specific system with the SAP SYBASE IQ 16 database.

• • [0037] 7133405 rows affected
  • [0038] Execution time: 55.075 seconds

[0039] The statistics above show that 7.1 million rows were updated on the TF table and MF table within 55 seconds. This was done in the first loop of the process.

• • [0040] 4383786 rows affected
  • [0041] Execution time: 109.383 seconds

[0042] The statistics above show that 4.3 million rows on both tables were updated in the second loop of the process. This means that a total of 11.4 (7.1+4.3) million rows were processed in 109 seconds. Note that the execution time shows the total time (2 loops), but the number of rows affected shows the count in the last update process which is the last update in the second loop. Running the test for 10 loops averaged a performance of 3 million rows per minute for the tested tables.

[0043] The batch processing technique described above will revolutionize the current batch processing methods in the industry. The method assigns new temporary keys to the files/tables used to establish the processing order and stores the new tables in temporary tables. It then applies business rules to the tables/files joining with the natural key and the new temporary key. The process then updates the original table using natural keys. This is repeated until all rows are processed. The method can update one row within every account instead of one row within the entire file/table for each update, which is what the current method does. The current (prior art) method known as “cursor processing” can only process several thousand rows per minute while the method can process several million rows per minute.

[0044] FIG. 2 is a more detailed flowchart representing the same process described above, and illustrated in FIG. 1, with two additional explanatory steps.

• • [0045] Step 1₂: Generate sequence numbers in the master table by sort order.
  • [0046] Step 2₂: Generate sequence numbers in the transaction table by sort order.
  • [0047] Step 3₂: Delete the rows that are not present in both temporary tables, matching them by natural key.
  • [0048] Added explanatory step: Create another set of tables selecting all rows where generated key=1.
  • [0049] Steps 4₂, 5₂: Update the master and transaction tables using both the natural key and sequence number.
  • [0050] Step 6₂: Update the permanent tables using only the natural key only.
  • [0051] Added explanatory step: Drop all temporary tables.
  • [0052] Step 7₂: Repeat the process.

[0053] The same process repeats until all rows in the Master Table or Transaction Table or both are processed. Rows that are already processed are excluded from the next loop by using the appropriate selection criteria.

[0054] An alternative (modified) embodiment of Step 5 combines first and second SQL statements into a single statement and combines third and fourth SQL statements into another single SQL statement. This reduces the lines of code and improves the performance. The modified step 5 is given below:

Perform all updates to the #TEMPDEBIT_SEQ and #TEMPCREDIT_SEQ tables based on business rules.

[0055]

UPDATE #TEMPDEBIT_SEQ
SET D.BALANCE_AMT = CASE
WHEN D.BALANCE_AMT >= -1*C.BALANCE_AMT THEN
D.BALANCE_AMT + C.BALANCE_AMT
WHEN C.BALANCE_AMT *-1> D.BALANCE_AMT THEN 0
END,
D.APPLIED_AMT = CASE
WHEN D.BALANCE_AMT >= -1*C.BALANCE_AMT THEN -1*
C.BALANCE_AMT
WHEN C.BALANCE_AMT *-1> D.BALANCE_AMT THEN
D.BALANCE_AMT
END
FROM #TEMPDEBIT_SEQ D, #TEMPCREDIT_SEQ_COPY C
WHERE D.EMIS_TIF_KEY = C.EMIS_TIF_KEY AND
D.SEQ = C.SEQ AND D.BALANCE_AMT <> 0;
COMMIT;
UPDATE #TEMPCREDIT_SEQ
SET C.BALANCE_AMT = CASE
WHEN D.BALANCE_AMT >= -1*C.BALANCE_AMT THEN 0
WHEN C.BALANCE_AMT *-1> D.BALANCE_AMT THEN
C.BALANCE_AMT + D.BALANCE_AMT
END ,
C.APPLIED_AMT = CASE
WHEN D.BALANCE_AMT >= - 1*C.BALANCE_AMT THEN -1*
C.BALANCE_AMT
WHEN C.BALANCE_AMT *- 1> D.BALANCE_AMT THEN
D.BALANCE_AMT
END
FROM #TEMPCREDIT_SEQ C, #TEMPDEBIT_SEQ_COPY D
WHERE C.EMIS_TIF_KEY = D.EMIS_TIF_KEY AND
C.SEQ = D.SEQ AND C.BALANCE_AMT <>0;
COMMIT;

[0056] FIG. 3 shows an architectural (schematic) diagram with tables used in the process. The base tables reside in the database. The Master Table (MT) and the Transaction Table (TT) are the base tables. Both tables have natural keys to identify the account and transactions associated with it. The Master Table (MT) contains all debit transactions and the Transaction Table (TT) contains all credit transactions. The Master Table key is identified as A in FIG. 3 and Transaction Table key is identified as B in FIG. 3.

[0057] Table MT1 is created after the first step of the process and table TT1 is created on the execution of step 2. Note the additional key columns C and D generated in these tables. Step 3 is applied on the same table which deletes all rows without a matching natural key on the TT1 and MT1 tables.

[0058] Tables MT2 and TT2 are created by selecting the rows with a generated key value of 1. Step 4 described above gives the details for this process. Note that this step is optional. It makes the SQL simpler in syntax for updates in the following step.

[0059] Step 5 is applied on table MT2 and TT2 by matching both natural keys and generated keys. This is where all business rules are applied.

[0060] Step 6 updates the original tables in the database (MT and TT) matching the natural key only. The generated key is ignored. If a unique identifier for a row is not available in the original tables, the row number will be used as an identifier and will be added as a column to the tables.

[0061] The process mentioned above is the only batch process that takes advantage of the Massive Parallel Processing (MPP) capabilities of column store databases.

[0062] The process can be used in any set of tables where sequential processing is required for matching data between two or more tables, datasets, or views.

[0063] Sample use cases are described below.

[0064] Batch processing is performed on transactions in multiple tables. The multiple tables include a first and a second permanent table, and a first and a second temporary table which are generated from the respective first and second permanent tables. Each table has a plurality of transactions posted thereto in respective rows of the tables. Each transaction is associated with a natural key, and each transaction has at least (i) a transaction date, (ii) a transaction amount, and (iii) a transaction identifier. In one preferred embodiment, the natural key is an account number. However, other types of natural keys are within the scope of the present invention. The method operates as follows:

• • [0065] (a) For each of the temporary tables, sequence numbers are generated for each set of transactions in a respective table that has the same natural key. The sequence numbers are ordered by their respective transaction date. The sequence numbers are natural numbers beginning with one.
  • [0066] (b) Ignore for further processing any rows in the first temporary table that do not have transactions with the same natural key as a transaction in the second table, and ignore for further processing any rows in the second temporary table that do not have transactions with the same natural key as a transaction in the first table. In one preferred embodiment, the rows are ignored for further processing by deleting the respective rows from the respective temporary tables.
  • [0067] (c) For all rows in the first and second temporary tables wherein the sequence number equals one, the transaction amounts are updated for each natural key in each of the first and second temporary tables. The updated transaction amounts in each of the first and second temporary tables will have either a zero, or non-zero value.
  • [0068] (d) Update the transaction amounts for each natural key in the first and second permanent tables using any respectively updated transaction amounts in the first and second temporary tables. The updating in the first and second permanent tables uses the natural keys and the transaction identifiers to identify the appropriate rows in the first and second permanent tables for updating. Together, the natural keys and the transaction identifiers define a unique natural key.
  • [0069] (e) Iteratively repeat the steps above (steps (a)-(d)). The iteration includes regenerating the first and second temporary tables with any rows in the corresponding first and second permanent tables wherein the transaction amount is a non-zero value. The regenerated first and second temporary tables are then used in steps (a)-(d). The iteration repeats until either the first or second temporary table has no further rows wherein the transaction amount is a non-zero value.

[0070] In one preferred embodiment of the method described above, the first and second permanent and temporary tables are each transaction tables.

[0071] The transaction date may simply be just a date, or it may also include a date and time (date/time stamp), both of which are collectively referred to as a “transaction date.” The nature and circumstances of the transaction will determine whether a time value will be included in the transaction date.

[0072] FIGS. 4A-4G show selected table entries and changes made to the table entries as a result of the multi-step processes shown in FIGS. 1 and 2 for Example 1. For illustration purposes, some entries are shown with strikethroughs, indicating the previous and new values on the same figure, instead of showing the changes in new tables on separate figures. There are three loops (2 iterations) in this example because the maximum sequence number for the rows of data is 3. To simplify the illustration, only the first and second temporary tables are shown in these figures.

[0073] Batch processing is performed in transactions in multiple tables. This embodiment differs from Example 1 because the first tables do not maintain any transactions, but only have an account balance that is updated using transactions in the second tables.

[0074] The multiple tables include a first and a second permanent table, and a first and a second temporary table which are generated from the respective first and second permanent tables. The first permanent and first temporary tables have a plurality of account balances posted thereto in respective rows of the tables. Each account balance is associated with a natural key. The second permanent and second temporary tables have a plurality of transactions posted thereto in respective rows of the tables. Each transaction is associated with a natural key, and each transaction has at least (i) a transaction date, (ii) a transaction amount, and (iii) a transaction identifier. In one preferred embodiment, the natural key is an account number. However, other types of natural keys are within the scope of the present invention. The method operates as follows:

• • [0075] (a) For the second temporary table, sequence numbers are generated for each set of transactions in the second temporary table that has the same natural key. The sequence numbers are ordered by their respective transaction date. The sequence numbers are natural numbers beginning with one.
  • [0076] (b) Ignore for further processing any rows in the first temporary table that do not have account balances with the same natural key as a transaction in the second temporary table, and ignore for further processing any rows in the second temporary table that do not have transactions with the same natural key as an account balance in the first temporary table. In one preferred embodiment, the rows are ignored for further processing by deleting the respective rows from the respective temporary tables.
  • [0077] (c) For all rows in the second temporary table wherein the sequence number equals one, update the transaction amount for each natural key in the second temporary table, and update the account balance for each natural key in the first temporary table. The updated transaction amounts and account balances in each of the first and second temporary tables will have either a zero, or non-zero value.
  • [0078] (d) Update the account balance for each natural key in the first permanent table using any updated account balances in the first temporary table, and update the transaction amount for each natural key in the second permanent table using any updated transaction amounts in the second temporary table. The updating in the first permanent table uses the natural keys of the first permanent table and the first temporary table to identify the appropriate rows in the first permanent tables for updating. The updating in the second permanent table uses the natural keys and the transaction identifiers of the second temporary table and the second permanent table to identify the appropriate rows in the second permanent tables for updating. Together, the natural keys and the transaction identifiers define a unique natural key.
  • [0079] (e) Iteratively repeat the steps above (steps (a)-(d)). The iteration includes regenerating the first temporary table with any rows in the corresponding first permanent table wherein the account balance is a non-zero value, and regenerating the second temporary table with any rows in the corresponding second permanent table wherein the transaction amount is a non-zero value. The regenerated first and second temporary tables are used in steps (a)-(d). The iteration repeats until either the first temporary table has no further rows wherein the account balance is a non-zero value, or the second temporary table has no further rows wherein the transaction amount is a non-zero value.

[0080] In one preferred embodiment of the method described above, the first and second permanent and temporary tables are each transaction tables.

[0081] In one alternative embodiment, prior to step (b), for the first temporary table, a sequence number of one is assigned for each row in the table, and the updating in step (c) occurs for all rows in the first and second temporary table wherein the sequence number equals one. This embodiment is optional because where there is only row per account, it is not necessary to create sequence numbers. However, this alternative embodiment allows the same programming rules to be applied as in Example 1, namely, by selecting all rows for updating by identifying for rows having a sequence number of equal to one.

[0082] The transaction date may simply be just a date, or it may also include a date and time (date/time stamp), both of which are collectively referred to as a “transaction date.” The nature and circumstances of the transaction will determine whether a time value will be included in the transaction date.

[0083] FIGS. 5A-5L show selected table entries and changes made to the table entries for Example 2. There are three loops (2 iterations) in this example because the maximum sequence number for the rows of data is 3. To simplify the illustration, only the first and second temporary tables are shown in these figures.

[0084] This embodiment is a more generic version of Example 2 because the table data may be any type of structured data, and any type of structured data transactions. Structured data refers to any data that resides in a fixed field within a row (record) or file. This includes data contained in relational databases and spreadsheets. Accordingly, a structured data transaction refers to an entry made in a table, such as a child table, here a second temporary and second permanent table, and which is subsequently populated to a parent table, here, a first temporary and first permanent table.

[0085] In structured data, one defines the fields of data to be stored and how that data will be stored, including the data type (e.g., numeric, alphabetic, name, date, address), and any restrictions on the data input (e.g., number of characters).

[0086] In one embodiment, the structured data and structured data transactions include financial data, such as shown in Example 2. However, in other embodiments, the structured data and structured data transactions include non-financial alphanumeric data, as shown in FIGS. 6A-6I.

[0087] Batch processing is performed on structured data in multiple tables. The multiple tables include a first and a second permanent table, and a first and a second temporary table which are generated from the respective first and second permanent tables. The first permanent and first temporary tables have one or more fields of structured data posted thereto in respective rows of the tables. The one or more fields of structured data are associated with a natural key. The second permanent and second temporary tables have a plurality of structured data table entries posted thereto in respective rows of the tables. Each structured data table entry is associated with a natural key. The method operates as follows:

• • [0088] (a) For the second temporary table, sequence numbers are generated for each set of structured data table entries in the second temporary table that has the same natural key. The sequence numbers are ordered by a sort order of one or more columns of the second temporary table in addition to the natural key. That is, the natural key and one or more other columns of the second temporary table are used for the sort order. The sequence numbers are natural numbers beginning with one.
  • [0089] (b) Ignore for further processing any rows in the first temporary table that do not have structured data with the same natural key as a structured data table entry in the second temporary table, and ignore for further processing any rows in the second temporary table that do not have structured data table entries with the same natural key as structured data in the first temporary table. In one preferred embodiment, the ignoring occurs by deleting the respective rows from the respective temporary tables.
  • [0090] (c) For all rows in the second temporary table wherein the sequence number equals one, update the structured data for each natural key in the first temporary table with the corresponding structured data table entries having the same natural key in the second temporary table.
  • [0091] (d) Mark the row of each updated structured data table entry in the second temporary file with a status indicating that the structured data table entry was processed.
  • [0092] (e) Update the structured data for each natural key in the first permanent table using any updated structured data in the first temporary table, and update the structured data table entry for each natural key in the second permanent table using any updated structured data table entries in the second temporary table. The updating in the first permanent table uses the natural keys in the first permanent table and first temporary table to identify the appropriate rows in the first permanent tables for updating. The updating in the second permanent table uses the natural keys and one of the columns of the second temporary table and the second permanent table to identify the appropriate rows in the second permanent tables for updating.
  • [0093] (f) Iteratively repeat the steps above (steps (a)-(e)). The iteration includes regenerating the first temporary table with the updated rows of the first permanent table, and regenerating the second temporary table with any rows in the corresponding second permanent table wherein the structured data table entry is not yet marked as being processed. The regenerated first and second temporary tables are used in steps (a)-(c). The iteration repeats until the second temporary table has no further rows with a structured data table entry that is not yet marked as being processed.

[0094] In one preferred embodiment, the natural key is a unique identifier of an entity, such as a social security number.

[0095] In one preferred embodiment, each structured data table entry has a date stamp, the second temporary table thereby having a date stamp column. The sort order uses the date stamp column of the second temporary table. The updating in the second permanent table uses the natural keys and the date stamp column of the second temporary table and the second permanent table to identify the appropriate rows in the second permanent tables for updating. The date stamp may be a date that an event occurred, or it may be a date that the table entry was made. The date stamp may simply be just a date, or it may also be a date/time stamp, both of which are collectively referred to as a “date stamp.” The nature and circumstances of the structured data will determine what date to use for the date stamp, and whether a time value will be included in the date stamp.

[0096] Regarding each of the examples above, the table regeneration may automatically incorporate the step of ignoring for further processing non-matching rows by natural key. That is, these rows may be ignored (e.g., deleted) at the same time that the first and second temporary tables are regenerated. These two approaches are equivalent to one another because they require performing the same processing, and achieve the same result.

[0097] FIGS. 6A-6I show selected table entries and changes made to the table entries for Example 3. There are three loops (2 iterations) in this example because the maximum sequence number for the rows of data is 3. To simplify the illustration, only the first and second temporary tables are shown in these figures.

[0098] The present invention may be implemented with any combination of hardware and software. If implemented as a computer-implemented apparatus, the present invention is implemented using means for performing all of the steps and functions described above.

[0099] When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.

[0100] The present invention can also be included in an article of manufacture (e.g., one or more tangible computer program products) having, for instance, non-transitory computer readable storage media. The storage media has computer readable program code stored therein that is encoded with instructions for execution by a processor for providing and facilitating the mechanisms of the present invention. The article of manufacture can be included as part of a computer system or sold separately.

[0101] The storage media can be any known media, such as computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium. The storage media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present invention as discussed above.

[0102] The computer(s)/processor(s) used herein may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computer may be embedded in a device not generally regarded as a computer but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone or any other suitable portable, mobile, or fixed electronic device.

[0103] Such computers/processors may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.

[0104] The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0105] The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present invention as discussed above. The computer program need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present invention.

[0106] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and the like, that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.

[0107] Data structures may be stored in non-transitory computer-readable media in any suitable form. For simplicity of illustration, data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a non-transitory computer-readable medium that conveys relationship between the fields. However, any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements.

[0108] The scope of the present invention thus further includes a tangible computer program product for performing a batch process on transactions and structured data in multiple tables, wherein the computer program product comprises non-transitory computer-readable media encoded with instructions for execution by a processor to perform the methods described above.

[0109] Preferred embodiments of the present invention may be implemented as methods, of which examples have been provided. The acts performed as part of the methods may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though such acts are shown as being sequentially performed in illustrative embodiments.

A. First and Second Tables

[0110] In the examples described above, the first tables are permanent tables and the second tables are temporary tables. However, the first tables are more broadly considered to be “source tables” and the second tables are more broadly considered to be “processing tables.” Thus, the scope of the present invention includes first and second tables that are not necessarily permanent and temporary tables.

B. Additional Tables

[0111] In the examples described above, only one secondary table is illustrated. However, there may be multiple sets of secondary tables. If so, the same steps described above for each of the examples are performed for each set of secondary tables. Consider, for example, Example 3 wherein there are two different sets of secondary permanent (e.g., source) and secondary temporary (e.g., processing) tables. The steps are performed for the first set of secondary permanent and secondary temporary tables until second temporary table of the first set has no further rows with a structured data table entry that is not yet marked as being processed. Then the same steps are performed for the second set of secondary permanent and secondary temporary tables until second temporary table of the second set has no further rows with a structured data table entry that is not yet marked as being processed. If there are additional sets of secondary tables, the steps are repeated again using the additional sets.

C. Tables and Views

[0112] Tables can be logical or physical structures. A table can be constructed as a physical table which resides in physical storage such as hard drives, or views, which are virtual tables defined by a query, or temporary tables which reside in memory. The scope of the present invention covers all such table equivalents.

D. Sort Order

[0113] Sort order is defined by the business rules of the batch process. The sort order is not necessarily limited to a single column, such as a time stamp, but may have as many columns or combination of columns as defined by the business rules. An example of sort order is when statistics on where specific products or sold geographically are required. In this example, the sort order would be the combination of the Product id and then the Zip code of the purchase. Another example is using a customer id, transaction id, and transaction date to reconcile and aggregate consumption of a customer in the utility industry.

E. Sequence Values

[0114] The examples provided above, and illustrated in the figures, use sequence numbers. However, the sequence number does not necessarily have to be a number, per se. More broadly, the sequence feature may be implemented by any set of sequence values, which may include numbers, alphanumerics, or characters that can be “sequenced.” Thus, a set of sequence values may be any set of values (e.g., numbers, alphanumerics, or characters) that can be generated in a sequential order. Examples include the following:

• • [0115] (1, 2, 3, 4, 5, . . . )
  • [0116] (A1, A2, A3, A4 . . . ),
  • [0117] (1.1, 1.2, 1.3, . . . ) and
  • [0118] (AAA, AAB, AAC, . . . , ZZZ).

[0119] In each of these examples the resulting set of values are in sequential order. Stated another way, each set of sequence values begins with a lowest value, followed by sequentially increasing values. In the embodiments above, the sequence numbers are natural numbers beginning with one, and for each loop in the process, only the rows having a sequence number equal to “one” are processed. However, in embodiments wherein the sequence values are not natural numbers beginning with one, for each loop in the process, only the rows having a sequence value that equals the lowest value in the set are processed. In addition, the sequence values do not necessarily need to have a lowest value, followed by sequentially increasing values, but instead may have a highest value, followed by sequentially decreasing values. In this alternative scenario, the updating occurs for the structured data wherein the sequence value has the highest value, not the lowest value. These two embodiments are equivalent to each other, and accomplish the same function.

[0120] Various embodiments of the invention have been presented above. However, the invention is not intended to be limited to the specific embodiments presented, which have been presented for purposes of illustration. Rather, the invention extends to functional equivalents as would be within the scope of the appended claims. Those skilled in the art, having the benefit of the teachings of this specification, may make numerous modifications without departing from the scope and spirit of the invention in its various aspects.