Categorization and presentation tool for code resources

[0002] 1. Field of the Invention.

[0003] This invention relates to computer system resource management systems and methods in general, and more specifically, to methods for indexing and locating assets, such as application programs which may be written in a variety of different programming languages, in a distributed or network environment.

[0004] 2. Description of the Related Art.

[0005] One of the most common uses of computers is to create, store, and index data for later retrieval. As a result of the burgeoning growth of computer usage, the number of data files available for searching has grown exponentially, leading to an information overload that can overwhelm a data searcher.

[0006] To help manage the access to these massive numbers of files, also known as “assets”, a process called “data mining” has evolved. Data mining is defined in Newton's Telecom Dictionary (15th Edition, Miller Freeman Publishing, New York, N.Y.) as “[U] sing sophisticated data search capabilities that use statistical algorithms to discover patterns and correlations in data.” In essence, computers are used to “crawl” through masses of data files, analyze the information contained in the files according to criteria input by the user, and output results to the user which the user can use to study the information further.

[0007] To support the explosive growth of computer usage, software development has become a key part of any company engaged in high-technology business. Large companies may have many software development groups located at numerous locations throughout the world, with each group employing hundreds or thousands of employees.

[0008] As used herein, complete programs (e.g., Microsoft Word™) developed by the programmers are referred to as “software assets” and the various subroutines used to produce the software asset (e.g., C++ subroutines and programs used to create Microsoft Word™) are referred to as “code assets.” These assets may number in the thousands or more for a single company and vary substantially in complexity, function, and size. For example, an asset may be a single program comprising hundreds of thousands of lines of computer code and designed to perform a multitude of tasks; at the other end of the spectrum, an asset may be a single subroutine comprising three lines of code.

[0009] With large numbers of employees focusing their work on the development of these assets, management becomes a critical task. With multiple groups within a company at different locations developing software for a variety of tasks, it is inevitable that duplication of effort will occur.

[0010] To avoid such duplication, it is desirable for all of the members of design groups, as well as all of the design groups within a company, to be able to share with each other the assets that they develop, and systems have been developed to assist in the management of such assets. In the software development field, the management, indexing, and retrieval of assets introduces an additional level of complexity not necessarily found in other asset management schemes. In particular, within a single group, assets may be developed in several different programming languages (e.g., Java, C/C++, COBOL, HTML, and/or XML) at the same time. Searching for code assets increases the complexity and difficulty of the search, since programmers typically want to search for language-specific constructs/semantics, such as inheritance relation, in object-oriented languages which cannot be captured using standard free-text searches. This makes it difficult for the users of the system to thoroughly search all of the assets.

[0011] Accordingly, it would be desirable to have an asset location system which offers the ability for free-text “search engine” style queries, attribute-specific queries, or a mixture of free-text queries and attribute-specific queries.

[0012] In our copending application Ser. No. 09/473,554 of common assignee herewith, and hereby incorporated by reference, there is disclosed a method and system for locating assets that provides a capability for the gathering of information about assets contained in data repositories. The technique is adapted to gathering information from either a single data repository or a plurality of data repositories, possibly in disparate locations of an enterprise. The captured information is then consolidated into a single database for access by multiple users. While this technique represents an improvement over conventional techniques of asset retrieval, nevertheless this technique does not provide automatic categorization to facilitate search and navigation by the users. Manual categorization has several drawbacks. When done by repository organizers, it requires the presence of an expert in each resource domain at all times. This is a very expensive solution and because it is not scaleable as in modern systems, the expert can never keep up with new resources which are added/updated to the repository every day. On the other hand, categorization which is done by users is also less than satisfactory, because users dislike the necessary overhead of describing the category for every resource, and moreover, they may be unqualified to categorize their resources.

[0013] It is therefore a primary object of some aspects of the present invention to improve the efficiency of asset location in computer system resource management systems.

[0014] It is another object of some aspects of the present invention to categorize code resources automatically using predefined taxonomy in computer system resource management systems.

[0015] It is a further object of some aspects of the present invention to enable users of code repositories to navigate through the repository of code resources in a computer system resource management system according to the category taxonomy without having to actually compose any queries.

[0016] It is yet another object of some aspects of the present invention to provide an improved tool for presenting categorization results in a computer system resource management system based on the category information and on the language semantics

[0017] These and other objects of the present invention are attained by a tool that is capable of categorizing code resources automatically into predefined taxonomy trees, that is into a set of predefined categories) . This tool enables the users of code repositories to navigate through the repository according to the category taxonomy without having to actually compose any queries. Moreover the category information can be used as part of the query criteria by the users.

[0018] In addition this invention describes a tool for presenting the categorization results in a novel method, based on the category information and on the language semantics.

[0019] The invention provides a computer-implemented method for indexing and locating assets stored on a storage device, which is performed by defining asset-specific categories for classification of asset-specific information, identifying stored assets, extracting the set-specific information from the stored assets, classifying the extracted information in the set-specific categories according to a predefined set of rules, and storing the classified textual and semantic information for retrieval.

[0020] According to a further aspect of the invention, extracting the set-specific information is performed with a language specific parser.

[0021] According to yet another aspect of the invention, the predefined set of rules includes a plurality of predefined sets of rules, wherein each of the predefined sets of rules is applied to a different language specific group.

[0022] According to still another aspect of the invention, a syntax of rules in the predefined set of rules is “<condition>→<category>@<weight-factor>”.

[0023] According to an additional aspect of the invention, the storage device includes a plurality of storage devices linked in a communications network.

[0024] According to an aspect of the invention, the method includes automatically updating the steps of identifying, extracting, and classifying when a new resource is stored in the storage device.

[0025] The invention provides a computer-implemented method for locating assets stored on a storage device, comprising the steps of identifying stored assets, extracting asset-specific information from the stored assets, classifying the extracted information according to a predefined set of rules, storing the classified textual and semantic information for retrieval, and displaying the stored information in a tree view. The tree view has a first hierarchy of the classified textual and semantic information and a second hierarchy of the classified textual and semantic information, wherein nodes that are represented in the tree view comprise nodes of the first hierarchy and nodes of the second hierarchy.

[0026] According to a further aspect of the invention, displaying is achieved by selecting a displayed element in a first display area to define a selected element, and displaying all categories of the stored information in which the selected element is classified.

[0027] According to another aspect of the invention, the selected element is classified according to a category relevancy score.

[0028] According to an additional aspect of the invention, the first hierarchy includes categories of a computer programming language that were identified in the step of classifying, and the second hierarchy includes a hierarchy of instances of the categories.

[0029] According to an aspect of the invention, the second hierarchy includes a class package.

[0030] According to still another aspect of the invention, the second hierarchy includes a name space.

[0031] According to an aspect of the invention, there is a step of preclassifying information that was obtained in the step of extracting, according to a plurality of language specific groups.

[0032] According to another aspect of the invention, the predefined set of rules includes a plurality of sets of rules, wherein each set of rules is applied to a different language specific group.

[0033] The invention provides a computer software product, comprising a computer-readable medium in which computer program instructions are stored. The instructions, when read by a computer, cause the computer to perform the steps of identifying stored assets on a storage device associated with the computer, extracting asset-specific information from the stored assets, classifying the extracted information according to a predefined set of rules, and storing the classified information for retrieval, and, on a monitor connected to the computer, in a first display area of the monitor displaying the stored information in a tree view which has a first hierarchy of the classified information and a second hierarchy of the classified information, wherein nodes of the first hierarchy are integrated with nodes of the second hierarchy.

[0034] According to another aspect of the invention, the step of displaying also includes selecting a displayed element in the first display area, and in a second display area of the monitor, displaying all categories of the stored information in which a selected element is classified.

[0035] According to a further aspect of the invention, the selected element is classified according to a category relevancy score.

[0036] The invention provides a computer system, comprising a storage device for storage of assets therein, a display monitor, a memory for storage of program instructions, and an execution unit that accesses the program instructions in the memory for execution thereof, wherein the program instructions cause the computer to perform the steps of identifying stored assets on the storage device, extracting asset-specific information from the stored assets, classifying the extracted information according to a predefined set of rules, storing the classified textual and semantic information for retrieval, and displaying in a first display area of the display monitor the stored information in a tree view has a first hierarchy of the classified textual and semantic information and a second hierarchy of the classified textual and semantic information, wherein nodes of the first hierarchy are integrated with nodes of the second hierarchy.

[0037] For a better understanding of these and other objects of the present invention, reference is made to the detailed description of the invention, by way of example, which is to be read in conjunction with the following drawing, wherein:

[0038] FIG. 1 is a pictorial representation of a data processing system which may be utilized to implement the present invention;

[0039] FIG. 2 is a diagram of a networked computing environment in which the present invention may be practiced;

[0040] FIG. 3 is a high level flow diagram illustrating the procedure according to some aspects of the invention; and

[0041] FIGS. 4 and 5 represent computer displays of the presentation phase in accordance with some aspects of the invention.

[0042] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances well known circuits, control logic, and the details of computer program instructions for conventional algorithms and processes have not been shown in detail in order not to unnecessarily obscure the present invention.

[0043] FIG. 1 illustrates a representative workstation hardware environment in which the present invention may be practiced. The environment of FIG. 1 comprises a representative single user computer workstation 10, such as a personal computer, including related peripheral devices. The workstation 10 includes a microprocessor 12 and a bus 14 employed to connect and enable communication between the microprocessor 12 and the components of the workstation 10 in accordance with known techniques. The workstation 10 typically includes a user interface adapter 16, which connects the microprocessor 12 via the bus 14 to one or more interface devices, such as keyboard 18, mouse 20, and/or other interface devices 22, which can be any user interface device, such as a touch sensitive screen, digitized entry pad, etc. The bus 14 also connects a display device 24, such as an LCD screen or monitor, to the microprocessor 12 via a display adapter 26. The bus 14 also connects the microprocessor 12 to memory 28 and long term storage 30 which can include a hard drive, tape drive, etc.

[0044] The workstation 10 communicates via a communications channel 32 with other computers or networks of computers. The workstation 10 may be associated with such other computers in a local area network (LAN) or a wide area network, or the workstation 10 can be client in a client/server arrangement with another computer, etc. All of these configurations, as well as the appropriate communications hardware and software, are known in the art.

[0045] FIG. 2 illustrates a data processing network 40 in which the present invention may be practiced. The data processing network 40 includes a plurality of individual networks, including LANs 42 and 44, each of which includes a plurality of individual workstations 10. Alternatively, as those skilled in the art will appreciate, a LAN may comprise a plurality of intelligent workstations coupled to a host processor.

[0046] Still referring to FIG. 2, the data processing network 40 may also include multiple mainframe computers, such as a mainframe computer 46, which may be preferably coupled to the LAN 44 by means of a communications link 48. The mainframe computer 46 may be implemented utilizing an Enterprise Systems Architecture/370, or an Enterprise Systems Architecture/390 computer available from the International Business Machines Corporation (IBM). Depending on the application, a midrange computer, such as an Application System/400 (also known as an AS/400) may be employed. “Enterprise Systems Architecture/370” is a trademark of IBM; “Enterprise Systems Architecture/390”, “Application System/400” and “AS/400” are registered trademarks of IBM.

[0047] The mainframe computer 46 may also be coupled to a storage device 50, which may serve as remote storage for the LAN 44. Similarly, the LAN 44 may be coupled to a communications link 52 through a subsystem control unit/communication controller 54 and a communications link 56 to a gateway server 58. The gateway server 58 is preferably an individual computer or intelligent workstation which serves to link the LAN 42 to the LAN 44.

[0048] Those skilled in the art will appreciate that the mainframe computer 46 may be located a great geographic distance from the LAN 44, and similarly, the LAN 44 may be located a substantial distance from the LAN 42. For example, the LAN 42 may be located in California, while the LAN 44 may be located in Texas, and the mainframe computer 46 may be located in New York.

[0049] Software programming code which embodies the present invention is typically stored in permanent storage of some type, such as the permanent storage 30 of the workstation 10. In a client/server environment, such software programming code may be stored with storage associated with a server. The software programming code may be embodied on any of a variety of known media for use with a data processing system, such as a diskette, or hard drive, or CD-ROM. The code may be distributed on such media, or may be distributed to users from the memory or storage of one computer system over a network of some type to other computer systems for use by users of such other systems. The techniques and methods for embodying software program code on physical media and/or distributing software code via networks are well known and will not be further discussed herein.

[0050] The preferred embodiment of the invention is explained herein in terms of a programming language domain; however the invention is also applicable to other types of knowledge domains. As shown in FIG. 3, in a preliminary phase 51, prior to the process of code resource categorization, a category taxonomy of the code resources is defined by a domain expert, either manually or using artificial intelligence techniques known to the art. This taxonomy is relatively static, and its preparation can be viewed as a non-recurring task. The task of the domain expert comprises several subtasks. Initially he defines a taxonomy. It should be emphasized that this definition is for the particular domain type. Then he sets out the significant features of the domain resource type that are needed in order to properly enable a categorization decision. Lastly he defines a set of categorization rules which apply to the significant features, and which can later be applied to categorize a domain resource type.

[0051] The classification process itself is composed of the following three main phases.

[0052] In the feature extraction phase 53 all the information from the code resource which is significant for the categorization decision is identified, preferably using the techniques taught in the above noted application Ser. No. 09/473,554, and then extracted. Essentially the feature extraction phase 53 is a technical process performed by a parser, which extracts the significant features identified by the domain expert from the resources. Both textual information and language specific semantic information are extracted. For example in the case of a Java source code resource, methods and fields names are extracted.

[0053] In a categorization phase 55 the material extracted in the feature extraction phase 53 used to classify the resources into categories. The classification procedure applies the predefined set of rules which were developed in the preliminary phase 51 in order to classify the code resource. The classification rules are written only once. The classification process is done automatically for each new or updated resource in the repository. A code resource may be classified into more than one category. Since the categories are subject and language oriented, a classification into more than one category means that the code resource is related to more than one subject or has a semantic relation to other resources in the category. The categorization results, which are the names of the categories in which a resource was classified, are then stored in step 56 for later use, preferably according to the teachings of the above noted application Ser. No. 09/473,554. It should be noted that the categorization results are not indexes. While it may be advantageous to index the categorization results for other purposes, this is outside the scope of the present invention and will not be discussed.

[0054] In a presentation phase 59 a presentation model is constructed based on the domain taxonomy, the categorization results and based on the semantic information of the resources. The semantic information is used to build a secondary hierarchy as will be disclosed in further detail hereinbelow.

[0055] The feature extraction phase 53 will now be explained in further detail. This activity makes use of a language specific parser to extract the “interesting” features from each of the resources. The features are arranged into several groups according to the language semantics. For each group a different set of rules is applied during the categorization phase 55 which is disclosed hereinbelow.

[0056] For Java resources the following features are extracted: text, class name, parent, parent name, methods, fields, imports, package name, and field types. In general several heuristics are employed for dividing composite names into single words in order to fully exploit all the information available from names in the resource. For example one of these heuristic algorithms searches for capital letters inside identifiers (strings), and each time a capital letter is encountered, the name is divided such that an individual word begins with the encountered capital letter. The original form of the name is stored as well. For example, the class name “ResourceAnalyzer” is divided into the words Resource and Analyzer. This approach has proven to be very useful for those Java resources in particular that make use of this coding convention.

[0057] Moreover, the following groups of features relating to Java resources have been identified:

[0058] comments—text;

[0059] inames—class name, parent name;

[0060] names—class name, parent name, methods, fields; and

[0061] types—imports, package, field types.

[0062] The motivation for the selection of these groups is as follows. The comments group holds all the textual information extracted from the resource and is the equivalent of the textual information extracted from documents, which have been subjected to ordinary text categorization. The next group, inames, is the equivalent of the title of a document. The group inames includes the class and parent names because the inventors have observed that these names have a special importance, and in many cases, the class name expresses the meaning of the entire class. The third group, names, includes all the names that describe the behavior of the class, which is mainly encapsulated in the method and field names. It is considered that classes that have in common several similar method names share some common behavior and should reside in the same category. The fourth group, types, includes all the additional type information implied by the class. The group types has been selected because the inventors have observed that classes which share the same package and make use of similar classes (implied by the imports feature), and have a similar set of types defined as fields (i.e., instance variables) also share some common behavior. Therefore, it is preferred that they reside in the same category.

[0063] For C++ resources, the following features are extracted: class name, parent name, member functions, member fields names, and include statements.

[0064] The following groups of features relating to C++ resources have been identified:

[0065] comments—text;

[0066] iname—class name, parent name;

[0067] names—class name, parent name, member functions, member fields; and

[0068] types—includes, field types.

[0069] The motivation behind the group selection for C++ is identical to Java and therefore is not described in the interest of brevity.

[0070] During the categorization phase 55 a set or rules is applied to the four groups created in the feature extraction phase 53. We have constructed sets of rules for both Java and C++, which are disclosed in Listing 1 and Listing 2 respectively.

[0071] The syntax of the basic rule is as follows:

[0072] <condition>→<category>@<weight-factor>,

[0073] where the symbol “→” stands for “categorize to” and the rule semantic is “categorize the resource into the category if the condition on the left side evaluates to true and set a weight factor for this categorization”. The basic format of <condition> is as follows:

[0074] <group feature>|<keyword><occurrence-condition>,

[0075] where the symbol “|” stands for “contains the keyword” and the semantics of the condition “the keyword appears in the group feature according to the occurrence-condition. The symbol “→” stands for “categorize to”.

[0076] For example the first rule states that if the feature group “comments” (i.e. the one including the textual information for Java categorization) includes the keyword “sqlexception”, and the expression is such that the keyword has to appear more then once, then it is associated with the “Database” category with a factor weight of “1”. The weight in each rule stands for how strong this rule should be evaluated in the categorization decision. This allows one to obtain a confidence score for each category to which a resource was classified. In addition, it is possible to write more complex conditions by concatenating two simple conditions on the left side of the “—” with the “&” sign.

[0077] In a prototype of the preferred embodiment of the invention, the rules were constructed in two phases. In the first phase, a learning tool was employed. This produced a set of rules based on a set of training resources that were categorized manually by a domain expert. During the second phase a domain expert verified and extended these rules in order to improve both the precision and the total number of the resources being categorized. Each rule defines a condition and a category to which a resource would be classified if the condition of that rule is satisfied. A resource satisfying more than one rule can be classified in more than one category. For example, the following rule “names | host >1→ Network” will cause any resource having the name “host” occurring more than one time in the “names” group to be classified to the “Network” category.

[0078] Referring now to FIGS. 3, 4 and 5, for use in the presentation phase 59, the user is shown a screen 60 on a computer monitor. A category visualization tool is presented as two main window panes, a left pane 62, and a right pane 64. The left pane 62 presents the taxonomy tree, i.e., the categories, and the code resources that belong to each category. The right pane 64 presents all the categories in which a selected resource of the left pane 62 is classified. In most cases, categories are presented as a tree view 66. In its most simple usage, this approach is not scalable, since once a node (e.g. a category) includes several thousand resources the user cannot efficiently browse it. There are several alternative solutions for this problem, such as sorting the resources in each category. According to some preferred embodiments of the invention, a view is presented which makes use of the language semantics. More specifically a tree view is implemented that is composed of two integrated hierarchies, unlike the conventional tree view presentation, where only one hierarchy is presented within a tree view. The technique according to this preferred embodiment of the invention applies to many types of tree based hierarchies. In the presently preferred implementation, the hierarchies are integrated in the following manner: the category tree is used as the top level hierarchy while the class package information forms the second hierarchy. The second hierarchy can start at any of the nodes of the top level hierarchy, but only in cases where at least one resource belongs to that hierarchy. The user who is presented with the tree view 66 navigates along the category tree until he reaches the desired subject or category. At this point, he may begin to navigate further according to the second hierarchy, an example of which is the package tree model shown in screen 60. The user is not required to browse a long flat list of resources.

[0079] In order to emphasize the two different hierarchies, each having different semantics, in the screen 60 the nodes representing a category node have been decorated with a “Key” icon 68, while the nodes representing nodes from the package hierarchy are decorated with a “Folder” icon 70. IN THE CASE OF C++, one can use the name spaces for building the second hierarchy level.

[0080] For example, in FIG. 4, one can observe that the node 72 named “Collections” is a sub-category of the “DataStructures” category 74 and together they compose part of the top level hierarchy. While the nodes “com” 76, “grasp” 78, “java” 80, “javax” 82, that are under the Collection node 72 are part of the second hierarchy, i.e. the package hierarchy.

[0081] The categories in the tree view are organized in a hierarchical manner so that the top level nodes represent the major categories, while the children of these nodes stand for subcategories. By selecting a category node the user is presented with a list of all subcategories, if there are any, and a list of all code resources that belong to the selected category. When a code resource is selected in the left pane 62, the list of all categories in which it was classified is displayed in the right pane 64. The categories in the right pane 64 are sorted according to their categorization rank, or category relevancy score, which was calculated during the categorization process. FIG. 5 shows a second example of the category visualization tool.

[0082] In a prototype version, using the techniques disclosed herein, successful categorizations were obtained 80% of the time on average. That is, 80% of the resources subjected to classification were placed into the correct category.

[0083] Category navigation according to preferred embodiments of the invention is a powerful method to explore code resources in a repository. It enables the user to find quickly what exists in the repository, and to find code resources, which are relevant to a concept of interest. This feature saves search time and improves the search results. Since the code resources are “grouped” into categories, one can perform “more like this” queries to find other code resources in the same category. In some preferred embodiments of the invention this functionality is supported by invoking the query on the index representing the entire repository and filtering out only those resources that belong to the desired category. The query capability is preferably extended by allowing users to formulate and execute queries for code resources that belong to or relate to a certain category.

[0084] The fact that the classification is done automatically provides a scaleable solution. Although the classification definition is done manually, it requires expert human input only during the preliminary phase 51 during which, the expert's knowledge is encapsulated into the rules used by the categorization engine. Therefore, the expert's presence is required only once.

[0085] Using conventional manual categorization, each update of the repository requires either the user's or an expert's knowledge in order to classify the code resource into the correct categories. In some preferred embodiments of the invention, classification of new or updated resources is done automatically. No additional input is required from the users and no experts are needed to maintain the repository. This fact makes this solution economically attractive and scaleable allowing to host hundreds of thousands of resources on a single server.

[0086] While this invention has been explained with reference to the structure disclosed herein, it is not confined to the details set forth and this application is intended to cover any modifications and changes as may come within the scope of the following claims: