Mining and characterization of data

[0001] This invention is related to identification and characterization of information items on a specified topic in a database.

[0002] Today, a knowledge worker is deluged with written or otherwise-published information from books, reports, periodicals, the Internet and other sources, so much so that relevant information is often mixed and/or saved with information that is irrelevant or of no present interest to the worker. Much, although not all, of this information is often stored in one or more “data mines” for possible review at a later time. As this information accumulates, it becomes more and more difficult for the worker to distinguish the relevant from the irrelevant and, more important, to distinguish the more relevant from the less relevant.

[0003] What is needed is a system that (1) allows a worker to identify “found” publications, in electronic format, that contain one or more uses of a specified word or word combination and (2) provides the worker with an assessment of the relative significance of phrases that often appear in the found publication, and of each found publication itself, based on a sequence of criteria with user-specified parameters. Preferably, the significance index should take into account one or more of the number of times and relative frequency of occurrence of the specified word/combination, separation distance between nearest-neighbor occurrences, occurrence of synonyms for the specified word/combination, and time interval(s) for the publication.

[0004] These needs are met by the invention, which provides a system and associated method for searching for one or more relevant publications in a collection of publications. In one embodiment, the method involves:

• • [0005] receiving a search-phrase whose occurrence is to be searched for in a selected collection of publications;
  • [0006] performing a search for the search-phrase within the collection; and
  • [0007] when at least one occurrence of the search-phrase is found in at least one publication, referred to as a “found publication”, assigning a date to the found publication and aggregating the found publications as a super-document;
  • [0008] identifying selected “found phrases” that occur in one or more of the found publications, and assigning a measure of significance to the occurrence each of the found phrase in all found publications;
  • [0009] displaying at least one of the found phrases and the measure of significance assigned to the corresponding found phrase; and
  • [0010] for each of at least one selected time interval, counting and displaying the number of found phrases having an assigned date within each of the at least one selected time interval.

[0011] The invention allows a user to specify one or more search phrases, perform a search of a database of publications in an electronic format, identify a group of one or more “found publications” that contain the search phrase(s) found, individually and collectively analyze the found publications to determine one or more characteristics of the found phrase within each found publication, and display the results of the analysis. This allows a user to extract relevant objective information from a collection of publications, without having to read and analyze each of the documents, and to highlight or otherwise identify topics of significant interest in the time interval covered by the data base.

[0018] The following several paragraphs set forth definitions that are adopted for purposes of disclosing and claiming the invention.

[0019] A “phrase” includes a word and a group of two or more consecutive words.

[0020] A “search-phrase” includes a word, a group of two or more consecutive words, and a grouping of two or more words or phrases that are separated by no more than n words (n≧1), and two or more words or phrases that are associated by one or more Boolean connectives.

[0021] A “found phrase” is an m1-word phrase (e.g., m1=1 or 2 or 3) that appears at least m2 times in a found publication, where m1 and m2 (≧1) are selected positive integers. Optionally, common articles, possessives and connectives, such as “a”, “an”, “the”, “their”, “our”, “and”, “or” and “but” may be ignored in identifying found phrases within a found publication.

[0022] A “Boolean connective” includes the Boolean logical connector operators AND, NAND, OR, NOR, XOR, XNOR and NOT.

[0023] A “Boolean grouping” is a group of phrases that includes “X B Y”, where X and Y are phrases and B includes at least one Boolean connective.

[0024] A “monotonically increasing function” of a real variable x is a function f(x) that satisfies f(x1)≧f(x2) wherever x1≧x2. Note that this definition includes, but is not limited to, strictly monotonically increasing functions.

[0025] A “publication” includes writings, formally published documents, informally published information, Internet articles and announcements and similar public releases of information, all available in electronic format.

[0026] A “measure of significance” is a numerical value assigned to a group of one or more found publications, based on one or more criteria representing the perceived relevance of the found publication(s).

[0027] An “information portrait” is a characterization of one or more found phrases, based upon one or more measures of significance associated with the found phrase.

[0028] A “release date” is a publication date assigned to a publication.

[0029] A synonym search-phrase is another search-phrase that has at least one meaning that is substantially the same as at least one meaning of the given search-phrase. A set of “synonyms” for a selected word, such as “frequency” may include any or all of {cycles per second, Hertz, Hz, kHz, MHz, GHz, period (for an oscillating signal)}.

[0030] A “separation distance” between two words, A and B, in a publication is the minimum number of words that separate an occurrence of “A” and an occurrence of “B” in the publication. For example, in the sentence “It is well known that A is inconsistent with B in certain situations.”, the separation distance for this sentence is 3 (words). Two words that are contiguous will have a separation distance of 0. A separation distance can be measured and assigned where two words (i) appear in the same sentence, (ii) appear in different sentences, (iii) appear in different paragraphs and/or (iv) appear in different sections of the publication.

[0031] A “relative temporal distance” Δt(rel) of a found publication from a user-specified time interval is either 0, if the found publication has a release or publication date within that time interval, or the difference in time between the release or publication date and the nearest time point within that time interval (if positive).

[0032] FIG. 1 illustrates a general approach of the invention, which optionally includes two procedures. In a first (direct task) procedure, a user provides a search and analysis (“SAA”) system with a theme, for example, by specifying a search-phrase whose occurrence is to be sought in a specified database of publications. A publication in the specified database is an electronic (“soft”) copy of a document, which may have been downloaded from a network, such as the Internet, may have been produced in electronic format originally, or may have been scanned in from a hard copy. The search-phrase provides a (first) theme, and this theme is projected onto the database of publications to search for and identify which publications contain at least one matching theme, referred to as “found publications.” This identified group G of found publications is treated as a single super document SD for initial analysis, and provides an “information portrait” for the group G.

[0033] In a second (reverse) procedure, the super-document SD is analyzed to determine the number m of times a selected m1-word phrase (e.g., m1=1, 2 and/or 3) appears in the super-document. Each such phrase that appears at least once in the super-document is assigned a primary measure of significance, based on the number m and upon one or more other attributes associated with the phrase. For example, for a particular database of approximately one million publications, covering a time interval from 01 Mar. 2000 though 31 Dec. 2001, use of the search-phrase “George Bush” yields 60,034 found publications, all of which are aggregated into a super-document SD for initial analysis.

[0034] Each found publication is analyzed to determine the number m of times a selected m1-word phrase appears in the found publication, where m1≧1; if m≧m2 (a selected positive integer), the phrase becomes a “found phrase,” which is not limited to the original search phrase. A primary measure of significance is assigned to each found phrase in a found publication, based upon a weighted sum of two or more measures of significance computed for the found phrase within the found publication. A secondary measure of significance is assigned to each found phrase in the super-document, based upon a sum of the primary measures of significance assigned to that found phrase in all found publications in the super-document.

[0035] FIG. 2 sets forth an example, using the search-phrase “George Bush”, of the 25 one-word phrases and 25 two-word phrases from the super-document that received the highest one-word and two-word secondary measures of significance, respectively. The top five one-word phrases are “Gore”, “Republican”, “Florida”, “Cheney” and “Democrat”, and the top five two-word phrases are “President George”, “Al Gore”, “White House”, “George Bush” and “electoral vote”, with other phrases having lower secondary measures of significance as shown.

[0036] A date is assigned to each found publication, and the number n1 of found publications in each of a sequence of one or more selected (preferably non-overlapping) time intervals is counted and displayed. In FIG. 3, the selected time interval is one of the calendar months running from Mar. 1–31, 2000 through Dec. 1–31 2001. A different selected time interval, such as a seven-day week or a ten-day time interval or a three-month time interval, can also be used here. FIG. 3 indicates that the search-phrase “George Bush” appeared in 127 to 6360 of the found publications for each of the 22 calendar months from March 2000 through December 2001, with the “peak” occurring in November 2000.

[0037] Optionally, the system selects found phrases (e.g., one-word or two-word or three-word) from the lists associated with FIG. 2 and determines and graphically displays, in FIG. 4, the values of the highest measures of significance for phrases appearing in the found publications having an assigned date within any of a selected sequence of one or more time intervals. Here, the time intervals here may, but need not, coincide with the time intervals used in connection with the search-phrase in FIG. 3. FIG. 4 graphically displays a plot (colors optional) of the values of measures of significance associated with several selected one-word found phrases, “Afghanistan” through “terrorism” for each of the 22 calendar months from March 2000 through December 2001.

[0038] Optionally, a short summary of a selected number of one or more of the found publications can be displayed so that the user can initially determine the relevance of that found publication to the user's goal. FIG. 5 illustrates a summary of two found publications for the search-phrase “George Bush”, both found publications involving the Elion Gonzales controversy in the year 2000.

[0039] FIG. 6 is a flow chart of an initial procedure for receiving and processing a search-phrase for a database of publications. In step 61, the system receives a search-phrase, and performs a search for and identifies each found publication containing the search-phrase within the database. In step 63, the system determines if the search-phrase is found in at least one publication in the database. If the answer to the query in step 63 is “no”, the system informs the user of this non-occurrence, in step 65, and optionally awaits a further user response.

[0040] If the answer to the query in step 63 is “yes”, the system identifies each publication (referred to as a “found publication”) in which the search-phrase occurs at least once and optionally assigns a date to each found publication, in step 67. In step 69 (optional), the system aggregates all found publications into a super-document SD. In step 71, the system determines a number N1 of times each of N2 selected one-word, two-word or three-word phrases (optionally including the search phrase) occurs in the super-document, assigns a primary measure of significance to each of these N2 “found phrases”, and displays the N2 found phrases in order of the primary measure of significance value for each, where N1 (≧1) and N2 (≧1) are selected integers.

[0041] In step 73 (optional), the system counts the number of found publications that have an assigned date in each of a first sequence of one or more selected time intervals and displays the number n1 of found publications with an assigned date in each of the time intervals in this first sequence.

[0042] In step 75 (optional), the system counts and displays the values of measures of significance associated with each of a selected sub-group of the found phrases (from step 71) that appears in a found publication having an assigned date in one of a second sequence of at least one selected time interval. The first and second sequences of time intervals may coincide or may differ in one or more time intervals.

[0043] In step 77 (optional), the system optionally displays a summary of one or more of the found publications.

[0044] A second operation illustrated in FIG. 1 begins with the subset of found publications and subjects each subset member to a further analysis to determine one or more measures of significance that indicate how relevant this found publication is likely to be for the user. Here, the user may specify additional attributes or constraints that help “sharpen” the analysis.

[0045] FIG. 7 illustrates an embodiment of a procedure for determining a primary measure of significance assigned to a found phrase in a found publication. In step 81, the system receives or is provided with a group of found publications, each having at least one occurrence of the specified search-phrase, assigns each found publication an integer in a consecutive list of integers (n=n0, n0+1, . . . , n1), and initializes a counting index to n=n0. In step 83, the system identifies and places in a list for subsequent analysis and display each found phrase in the found publication no. n. A found phrase includes all m1-consecutive word phrases that occur at least m2 times within the found publication no. n, where m1 and m2 are system-specified or user-specified integers, with m1=1 and/or 2 and/or 3, and m2≧1. A selected number M of the found phrases is optionally displayed, where M is a selected integer (M≧0). In step 85, the system determines the number W(n) of words in, and the number N(n) of times a found phrase occurs in, the found publication no. n. In step 87 (optional), the system optionally determines a fraction N(n)/W(n) representing the frequency of occurrence of each found phrase in the found publication no. n.

[0046] In step 89, the system assigns, to each found phrase in the found publication no. n, a numerical value representing a first measure of significance, based on at least one of the number N(n), the fraction N(n)/W(n) and a fractional difference N(n)/W(n)−f(avg), where f(avg) is an estimate of an average frequency of occurrence for the found phrase in a selected group of publications (which may be, but is not necessarily, the original collection of publications). In step 91 (optional), for the found publication no. n, the system displays at least one of (i) the at least M m-consecutive word found phrases that occur, (ii) the number N(n), (iii) the fraction N(n)/W(n), the fractional difference N(n)/W(n)−f(avg), and (v) an indicium representing the first measure of significance.

[0047] In step 93 (optional), where a found phrase occurs at least twice in the found publication no. n, the system determines a smallest separation distance d(min;W(n);N(n);n) between any two occurrences of the found phrase within the found publication no. n, representing a “nearest neighbor” separation distance within the found publication no. n. In step 95 (optional), the system assigns a second measure of significance, based upon at least one of (i) the minimum separation distance d(min;W(n);N(n);n) for the found phrase and (ii) at least one parameter used for the first measure of significance.

[0048] In step 97 (optional), the system or user provides at least one synonym-phrase for one or more found phrases and determines a number N(syn;n) of times the at least one synonym-phrase occurs within the found publication no. n. In step 99 (optional), the system assigns a third measure of significance, based upon at least one of (i) the number N(syn;n) for the found phrase and (ii) at least one parameter used for the first measure of significance.

[0049] The user may specify which of the second and/or third measures of significance is to be determined, and optionally displayed, and may specify one or more parameters, if any, to be used to determine to determine such measure(s) of significance.

[0050] In step 101, the system increments the index n (n→n+1). In step 103, the system determines if n>n1, where n1−n0+1 is the number of found publications. If the answer to the query in step 103 is “no,” the system returns to step 83. If the answer to the query in step 103 is “yes,” for each found phrase (n=n0, n0+1, . . . , n1), the system continues to step 105 and computes a weighted sum, using non-negative weight coefficients, of the first, second and third measures of significance for each found phrase in all the found publications. Each weight may be 0 or positive. This weighted sum is referred to herein as the “primary” measure of significance for each found phrase and is displayed in the analogue of FIG. 2.

[0051] One approach to determination of the first measure of significance uses a binomial distribution b(W;N;p) to compute and compare probability values for a found phrase in the found publication no. n, $\begin{array}{cc}1-\mathrm{P1}=\sum _{k=N\left(n\right)+1}^{W\left(n\right)}C\left(W\left(n\right);k\right){p^k\left(1-p\right)}^{W\left(n\right)-k},& \left(1\right)\end{array}$ $\begin{array}{cc}1-\mathrm{P2}=\sum _{k=N\left(\mathrm{avg}\right)+1}^{W\left(n\right)}C\left(W\left(n\right);k\right){p^k\left(1-p\right)}^{W\left(n\right)-k},& \left(2\right)\end{array}$
p=f(avg)=N(avg)/W(n),  (3)
C(x;y)=x!/{y!(x−y)!},  (4)
where f(avg) is an estimated average frequency of occurrence of the specified found phrase in the collection of publications. Note that P 1 and P2 are increasing functions of the respective variables N(n) and N(avg). In one approach, if P2≧P1, the found phrase is assigned a relatively low measure of significance, and this measure of significance increases monotonically with at least one of the following two variables: the ratio P1/P2 and the difference P 1−P2. Definitions of the quantities 1−P1 and 1−P2 in Eqs. (1) and (2) may be replaced by more general probabilistic definitions, such as
1−P1=F(N(n);W(n);p),  (1′)
1−P2=F(N(avg);W(n);p),  (2′)
where F(u;v;p) is a cumulative probability density function that is monotonically decreasing in u, for fixed v, and is monotonically increasing in v, for fixed u.

[0052] A similar probabilistic calculation and comparison can be carried through for the second measure of significance, involving the separation distance d(min;W;N). One easily verifies that
d(min;W(n);N(n);n)=0(W(n)<2N(n)−1).  (5)
For W≧2N(n)−1 and for a uniform distribution of the search-phrase, one verifies by consideration of cases that the average distance of separation is approximately
d(avg;W(n);N(n)=2;n)={2W(n)²−7W(n)+6}/(6W(n)),  (6)
which has an asymptotic limit of W(n)/3 as W(n) increases without limit. For larger values of N(n), it appears that the average distance of separation is limited by d(avg;W(n);N(n))≦W(n)/(N(n)+1) in the asymptotic limit of large W(n). For N(n) (≧2) occurrences of the found phrase in a found publication, the measured value d(min;W(n);N(n);n) is compared with d(avg;W(n);N(n);n) in determining the second measure of significance. In one approach, if d(avg;W(n);N(n);n)≦d(min;W(n);N(n);n), the found phrase is assigned a relatively low measure of significance, and this measure of significance increases monotonically with at least one of the following two variables: the ratio d(avg;W(n);N(n);n)/d(min;W(n);N(n);n) and the difference d(avg;W(n);N(n);n)−d(min;W(n);N(n);n).