LAYERED CONTEXTUAL CONFIGURATION MANAGEMENT SYSTEM AND METHOD AND MINIMIZED INPUT SPEECH RECOGNITION USER INTERFACE INTERACTIONS EXPERIENCE

[0001] This system and method relates to the field of software programming. More precisely, the invention provides systems and methods for contextual configuration management of various software applications in such a way that the software according to the present invention can enhance or modify any aspect of various software applications without change or access to the source code. In particular, speech interfaces are provided for non-native speech interface applications. The present invention also provides an efficient method of processing user input in a speech recognition interface by adding a graphical layer over a typical graphical user interface that is non-disruptive.

[0002] Software system architectures typically provide application software, operating system software, and possibly utility software. The operating system provides a set of known resources for the applications, and thus allows a software application writer to generate a program which relies on a standardized interface to the hardware, other applications, and the environment. Utility software may be designed to interact with an application, and may modify the way that it operates. An example is a macro recorder and playback utility, which records and recreates user interface inputs, to allow automated control of a software application.

[0003] According to a first aspect of the invention, a layered contextual configuration management system and method is provided.

[0004] One aspect of the invention is to calculate or compute the contextual configuration provided the given state of a computer operating system running software applications (called context). Such calculated contextual configuration can then describe desired enhancements of non-proprietary software (i.e., software separate or independent from the present invention) that can be implemented from proprietary software (i.e., software according to the present invention) while requiring only a minimum of input from a software user about the non-proprietary software into the proprietary software, and not requiring coding into the non-proprietary software for the purpose of enhancement or modification. See, U.S. Pat. No. 6,071,317, expressly incorporated herein by reference.

[0005] A successful implementation of this embodiment avoids the need for access to, and use of, most Software Development Kits (SDK) to adapt non-proprietary software without sacrificing any flexibility from the user or software developer's perspective, provided the fact that targeted proprietary software can allow enhancement or modification of functionality from the proprietary software.

[0006] For example, an implementation according to a preferred embodiment of the present invention is related to speech recognition proprietary software that enhances non-proprietary software through the addition of speech recognition functionality, without requiring additional coding into the non-proprietary software to access that functionality. The proprietary software implementing this embodiment can run (as a daemon process) on an operating system while monitoring for changes in contexts, like a change in the top-application or a top-window or a top-keyboard-focus edit field. Once a change is detected, the proprietary software can refer to stored configuration files (e.g., one per level) so that a single configuration for the context is calculated while consolidating the requirements from each level. It is noted that it is not required that a static stored configuration for each level be provided, and for example, the configuration may be dynamically generated, for example based on a set of available applications, or a status of respective applications. The speech recognition proprietary software can then activate or deactivate features on the operating system or application(s) as a consequence of this context configuration that was calculated. Such a method and system consequently provides a flexible and efficient way of enhancing non-proprietary software without requiring code change at that end.

[0007] According to a second embodiment of the invention, a system and method for minimized input speech recognition user interface interactions experience is provided.

[0008] In an effort to improve the speech recognition experience, and more particularly, to respect the human brain limitations in regards to how many voice commands can be memorized by an average person, this aspect of the present invention provides systems and methods for interaction with an automated device though voice. Through the use, for example, of superposed speech recognition related content on a computer screen, that does not disrupt commonly known state-of-the-art input methods—like mouse and keyboard focus in a window of most modem operating systems, modem operating systems are complemented with a speech recognition dedicated user-interface.

[0009] According to a third embodiment of the invention, a computer display or dialog from non-proprietary software or an operating system is enhanced through the addition of “hot spots”, which are, for example, graphic indications of interactive elements on a graphic user interface generated by the non-proprietary software which have been “enabled” or enhanced with functionality or alternate access capability by the proprietary software. For example, a text-labeled screen button may be linked to a speech command, such that receipt and processing of the speech command will activate the screen button. The hot spot is graphically indicated, for example, by a green spot adjacent to the screen button, indicating to a user that a text label associated with a graphic user interface interactive element is enabled for immediate activation by the proprietary software.

[0010] “Levels” (L): A natural division of inclusion—L_i has L_(i+1), or L_(i+1) is in L_i—occurring into a computer operating system. L_i is used to identify the level i. The range of i is from 0 to N.

[0011] “Attributes” (A): A placeholder for a value or a list of values that can be changed by a user. For example, a placeholder for the volume level, and a placeholder for a list of e-mail addresses on a computer system are attributes. A_j is used to identify the j^th attribute. The range of j is from 1 to P. A_ji is used to identify the j^th attribute on level i (or in C_i).

[0012] “Configuration” (C): Stored values allocated to attributes (which may be stored in file or in memory) representing desired enhancement or desired behavior for a targeted aspect of processing. For example, the grouping of the ‘sound output device’, ‘sound input device’ and ‘volume’ values for the corresponding attributes could constitute a ‘sound’ configuration. C_i is used to identify the configuration of level i.

[0013] “Layered Configuration” (LaC): Optional stored values allocated to attributes subset (which may be stored in file or in memory) overwriting the Configuration (C_i) and representing a potentially different desired enhancement or desired behavior for a targeted aspect of processing. LaC_k is used to identify the k^th Layered Configuration. LaC_ki is used to identify the k^th Layered Configuration of level i. k may range from 0 to Q.

[0014] “Level Criteria” (LC_i): A unique level identifier describing the context for the corresponding level. LC_i is used to identify the level criteria of level i. LC_it is used to identify the level criteria of level i at time t. For example, in the preferred embodiment of this invention, for the application level (L₁), LC₁ could be the application name; the string “Microsoft Word” can be used for LC_1t if Microsoft Word is the foreground application at the time t.

[0015] “Context” (Cx_t): The differentiating set of level criteria at a given time. Cx_t={LC_0t . . . LC_Nt} is used to identify the context at time t.

[0016] “Level Contextual Configuration” (LCC): LCC_i is the resulting calculated configuration from CLF_i based, in part, on LCC_i−1 and C_i if i>0, or C_i if i=0. LCC_i is used to identify the level contextual configuration of level i. LCC_it is used to identify the level contextual configuration of level i at time t.

[0017] “Contextual Configuration” (CC): CC_t is LCC_Nt after GCMP processing where N is the value representing the highest level at a given time t. Obtaining this value is the goal of this invention.

[0018] “Configuration Level Filtering” (CLF): CLF_i is the process by which LCC_i−1 and C_i, if i>0, or C_i, if i=0, are consolidated into LCC_i. CLF_i is used to identify the configuration level filtering of level i.

[0019] “Promotion Attributes” (PA): PA_ik is the promotion attribute that relates to a subset of attributes in C_i (stored in file or in memory) describing the desired consolidation of LCC_(i−1)t and C_i into LCC_it during CLF_i. PA_ik is used to identify the k^th promotion attribute of level i. k is expected to range from 1 to M. A promotion attribute is an attribute.

[0020] “Global Configuration” (GC): A unique configuration that describes changes to make on targeted attributes prior to promoting to contextual configuration (CC).

[0021] “Global Configuration Modifier Process” (GCMP): The Global Configuration Modifier Process (GCMP) is a process by which the highest level contextual configuration (LCC_N) is optionally modified provided the content of the Global Configuration (GC).

[0022] “Window Id”: The window Id is a unique description of the hierarchy to locate or differentiate a window or an edit field into a window. For example, on the Windows operating system: {OpusApp,“Microsoft Word *”}, {MsoCommandBarPopup,“Border Color”} where OpusApp and MsoCommandBarPopup are class Ids, and “Microsoft Word *” and “Border Color” are window names (and the ‘*’ character in the window name is a wild card identifier).

[0027] The use of configuration in software is well known. However, configuration has mainly been seen as a homogenous input that is provided by a user from which the only apparent use from a software is to statically refer to it. The novelty exposed by this invention is that contextual configuration at any given time t (CC_t) can be calculated based on the context managed by levels (L_0t . . . L_Nt), and filtered based on promotion attributes (PA₀₁ . . . PA_NM). With this invention, the user is allowed all the flexibility that is sometimes required to fully customize a non-proprietary application without more restriction than what a Software Development Kit (SDK) would normally impose, and without modifying the source code of the non-proprietary software—contrarily to the experience when an SDK is used. In the process, this invention also allows for the configuration maintained by the user to be optimally propagated and results into minimal input required while still providing full-control to the user over the management of his or her customization (instead of relying on a software coder).

[0028] At any given time t, an automated device like a computer operating system can be associated a context (Cx_t) from a finite set of level criteria (LC_o . . . LC_N) corresponding to each level (L₀ . . . L_N)—each of them potentially holding fixed configuration (C_i) and fixed promotion attributes (PA_i1 . . . PA_iM). For the preferred embodiment of this invention, the levels are:

[0029] The base level (L₀)

[0030] The top application (L₁)

[0031] The top window in the top application (L₂)

[0032] The edit item that has keyboard focus in top window of top application (L₃)

[0033] The lowest level (L₀) is the most universal level, and the highest level (L_N) is the less universal one. The corresponding level criteria (LC_i), in the preferred embodiment of this invention are:

[0034] NULL for LC₀.

[0035] The top application name for LC₁. For example: “Microsoft Word”.

[0036] The window Id of the top window in the top application for LC₂.

[0037] The window Id of the current edit field in the top window of the top application for LC₃.

[0038] The configuration on all levels (C₀ . . . C_N) hold values for the same attributes (A₁ . . . A_p) and promotion attributes (PA₁ . . . PA_M). The values assigned to the attributes and promotion attributes are unrelated to each others on different levels; i.e. they are not expected to be the same nor different. One aspect of the present invention therefore calculates a unique contextual configuration at a time t (CC_t), based on the context at that time t (Cx_t), and a fixed set of configuration maintained by the user for each level (C₀ . . . C_N). In the process of calculating CC_t, LCC₀ . . . LCC_N are generated as a residue, and are transient by nature.

[0039] In the preferred embodiment, the base level (L₀) holds a configuration (C₀) that is influencing the contextual configuration (CC_t) for all times t. LC₀ never changes since the base level applies for all contexts. In the preferred embodiment, there is always a top application running on a computer operating system. Consequently, a top application running level (L₁) and its associated configuration (C₁) is always influencing the contextual configuration (CC_t) being calculated. For the preferred embodiment, the top application (LC₁) can be defined as the application that holds some level of focus. That is, the application that has the unique edit item holding keyboard focus (LC₃), or, if no edit item has keyboard focus (LC₃=NULL), the application that has the window holding graphical user-interface focus (LC₂). In the event that no edit item has keyboard focus (LC₃=NULL) and no window has graphical user-interface focus (LC₂=NULL), the top application (LC₁) is simply the application at the top of the operating system process chain.

[0040] To calculate the contextual configuration (CC_t) for a given context at a given time t (Cx_t) with the base level criteria (LC₀), a determined top application (LC_1t), a determined top-window (LC_2t where LC_2t can be NULL), a determined current edit item holding keyboard focus (LC_3t where LC_3t can be NULL), all applicable levels (L₀ . . . . L_N) can have some associated configuration (C₀ . . . C_N) maintained by the user. In the event that one or more level does not have associated configuration stored (C_i=NULL), some default configuration and default promotion attributes can be generated potentially differently for each applicable level. At initialization time of the process (t=0), or when one or more level criteria change is detected, i.e. there is an i for which LC_it1≠LC_it2 (for given times t1 and t2) the method calls for the contextual configuration (CC_t) to be recalculated. Such recalculation of the contextual configuration (CC_t) is described in FIG. 1. For the preferred embodiment, the method starts with the lowest or most universal level (L₀) and loads its associated configuration (C₀). For the preferred embodiment, it then passes the configuration to its associated configuration level filtering (CLF₀) to calculate the corresponding level contextual configuration (LCC₀). Although not the preferred embodiment of this invention, this lowest configuration level filtering at the base level (CLF₀) is not essential for the good execution of the invention, and the base level configuration (C₀), instead of the base level contextual configuration (LCC₀), can be passed directly to the upper configuration level filtering (CLF₁). For the preferred embodiment, the contextual level configuration (LCC_i) is then passed to the next configuration level filtering (CLF_i+1). The next level's configuration level filtering (CLF_i+1) also loads its associated configuration (C_i+1), and consolidates both configuration provided promotion attributes (PA_(i+1)0 . . . PA_(i+1)M) in its own level contextual configuration (LCC_(i+1)). Each level configuration (C₀ . . . C_N) also stores 1 to M promotion attributes (PA_ik) related to attributes representing enhancement or modification to a non-proprietary software. PA_ik is associated to a subset of attributes in the configuration C_i that are all mutually exclusive to each others. For example, for the speech recognition implementation using this invention, the promotion attributes may relate to ‘command and control’ attributes, ‘spell mode’ attributes, ‘dictation’ attributes or ‘mouse control’ attributes. Each promotion attribute (PA_ik) is to then used by each corresponding configuration level filtering (CLF_i) to calculate the corresponding level contextual configuration (LCC_i). For the preferred embodiment, the possible promotion attributes (PA_ik) values are:

[0041] Promote (PROMOTE)—available only for PA₁ . . . PA_N.

[0042] Do not promote, do not use current (DNPDNUC)—available for PA₀ . . . PA_N.

[0043] Do not promote, use current (DNPUC)—available for PA₀ . . . PA_N.

[0044] Merge (MERGE)—available only for PA₁ . . . PA_N and only when corresponding to attributes that hold a list.

[0045] In the preferred embodiment, filtering on a level (CLF_i) refers to promotion attributes (PA_i1 . . . PA_iM) to calculate the associated level contextual configuration (LCC_it) at that time t.

[0046] In the event that a promotion attribute (PA_ik) is PROMOTE, the corresponding attributes (A_ji) related to the promotion attribute (PA_ik) of the current level configuration (C_i) are ignored, and the corresponding attributes (A_j(i−1)) related to the lower level's contextual configuration (LCC_(i−1)) are affected into the contextual configuration from the current level (CC_i). In the event that the promotion attribute (PA_ik) is DNPDNUC, the corresponding attributes (A_ji) related to the promotion attribute (PA_ik) are re-initialized and/or flagged as disabled. In the event that the promotion attribute (PA_ik) is DNPUC, the corresponding attributes (A_ji) related to the current's level configuration (C_i) are affected to the current level contextual configuration (CC_i) and the corresponding attributes (A_j(i−1)) of the lower level contextual configuration (LCC_(i−1)) are ignored. In the event that a promotion attribute (PA_ik) is MERGE (available for list attributes and levels higher than 0 only), the corresponding attributes (A_ji) related to the current's level configuration (C_i) are merged with the corresponding attributes (A_j(i−1)) of the lower level contextual configuration (LCC_(i−1)) into the current level contextual configuration (LCC_i). The contextual level filtering is repeated for all levels. In the preferred embodiment of this invention, when all levels have calculated their level contextual configuration, the highest level contextual configuration (LCC_N) is passed to the Global Configuration Modifier Process (GCMP) which also refers to Global Configuration (GC) in order to consolidate both inputs into the Contextual Configuration (CC_t). This last step of processing prior to generating CC_t is useful to change some attributes globally. For example, in the speech recognition implementation of this invention, the Global Configuration GC may hold some information like stating that the current user is blind, or deaf, etc. . . . Since the user maintaining, or at least deploying the initial version of Level Configuration (LC_i) can be a different user than the user at the time t being calculated, adding the flexibility for the user at time t to change globally its configuration is important. Should the GCMP detect an attribute stating that a user is blind, for example, the GCMP can adapt the text-to-speech attributs to be widely used when LC₀ . . . LC_N would not advocate the use of text-to-speech. This makes it easy for the end-user to globally change his configuration while also not limiting a different user to deploy configuration for non-proprietary software adaptation or modification without taking all these factors (people being blind, deaf, personal preferences) into consideration, and while still providing useful input for the process.

[0047] Up to this point, nothing has been mentioned about Layered Configuration (LaC). Layered Configurations (LaC) are not required for the invention to be functional. Nevertheless, they allow an additional dimension of flexibility. As stated earlier, Configurations (C_i) need to hold a value for all Attributes (A_j) in order for the Contextual Configuration to be calculated. When Layered Configurations (LaC_ki) are used (Q>0), each Configuration (C_i) stays the same, requiring that a value be set for each Attribute (A₀ . . . A_p). The difference between a Layered Configuration (LaC_ki) and a Configuration (C_i) is that the Layered Configuration (LaC_ki) needs to hold a value only for the Attributes A_j that is desired to overwrite from the Configuration (C_i). As the Configurations (C_i) go forward in the process, if a non-empty Layered Configuration is encountered (LaC_ki), since only the Attributes (A_j) that are desired to overwrite the Configuration C_i are kept, other original Attributes from C_i will stay untouched.

[0048] While referring to FIG. 2, it is possible to see the effect of Layered Configuration being factored in the User's Configuration box of FIG. 1. FIG. 2 is the preferred embodiment of the invention in regards to the User's Configuration. It assumes that most of the work is done by the user 1 at deployment time, that to make it as easy and straight-forward as possible for the end-user (user 3) to become productive. User 1 fills the Configurations C₀ . . . C_N for the possible Level Criteria LC_i. Once that is done, user 1 can deploy its Configuration to the world. Once it is deployed, in a large institution for example, standardization may be welcome. For that reason, the Administration layered configuration (LaC₁) is introduced. The hypothetical large site's administrator (user 2) would be the exclusive owner of that layer (for example, password protected or by other methods of securing electronic data) and files related to this layered configuration reside on a server, and are synchronized to the local hard-drive periodically. That way, user 2 can, at any given time, change the configuration of its entire work-force without further complications related to deployment within its own institution. The following Layered Configuration (LaC₂) is allocated to final users (user 3) which may also change configurations prior to them getting to Configuration Level Filtering (CLF_i). Many Layered Configuration can be introduced within the invention (although the preferred embodiment uses 2). Also, some Layered Configuration (LaC_ki) as well as the Configurations (C_i) may follow other rules adopted in the state-of-the-art industry like, password-protection, download-upload synchronization, etc.

[0049] Attributes may also contain information in regards to subsequent Layered Configuration access. That is, a user managing the Layered Configuration LaC_ki may well set a logical flag for each Attributes (A_j) to specify if each is available for edition or not for following Layered Configuration (LaC_(k+1)i). By doing that, for example, in the preferred embodiment of the invention, an administrator (user 2), can disable the accessibility to any Attribute (A_j) for the Preference Layered Configuration users (user 3).

[0050] The present invention provides an improved speech recognition human computer user interface, which respects human cognitive and performance limitations in regards to how many voice commands can be memorized and used by a person. The preferred embodiment uses superposed speech recognition related content on a computer screen that does not disrupt other typical human user input devices and methods, including mouse and keyboard focus in a windowing computer operating system.

[0051] The present example provides, for example, a graphic overlay for a typical graphic user interface which is non-disruptive. Such added graphical layer may relate exclusively to speech recognition input (may be triggered by speech recognition commands) and may be translucent so that the user can still refer to the state-of-the-art graphical user-interface below if desired.

[0052] As shown in FIG. 3, in order to complement a mouse, a speech recognition system may superpose a grid over the actual graphical user-interface in order to map a logical coordinate with an utterance that can be spoken. The grid may be translucent but the bulk of the state-of-the-art graphical user-interface behind has to stay visible. That way, the speaker is communicated a set of coordinates that it may use to perform operations on. For example, in the preferred embodiment of this invention, the coordinates are composed of 2 digit numbers pairs. Valid coordinates could be “23-51”, or “21-55”. A speaker may then say a command like “click twenty three fifty one” and a click would happen a the corresponding location in the state-of-the-art graphical user-interface under the number 23-51 of the superposed user-interface. But the user may also say a command like “move to twenty one fifty five” followed by the voice command “Drag to twenty three fifty one”. That would in fact emulate a drag in a state-of-the-art graphical user-interface without using an actual mouse but speech recognition instead.

[0053] It is obviously not possible to fill the entire automated device's screen with coordinates, so holes are to be expected. In the event when a speaker needs to perform a mouse operation in an area that is within a hole of the communicated coordinates in the superposed user-interface, he may use ‘shift’ voice commands. By saying “shift right”, for example, the entire set of coordinates would shift to the right. He could then shift the grid until a coordinate is over the desired point for his operation, and then continue by uttering his operation normally.

[0054] That same concept may also apply on limited areas of a state-of-the-art graphical user-interface so that the entire screen would not be filled of the superposed coordinate system.

[0055] For cases where a graphical user-interface may be required as a response to a voice command on a speech recognition system, translucency can be used. Furthermore, that potential translucent graphical user-interface needs not to be disruptive towards commonly known state-of-the-art input methods (keyboard and mouse input). If the content to be communicated to the speaker cannot fit into a single screen, this embodiment of the present invention provides that the entire content needs to be scrolled at limited speed for the speaker to have enough time to read and react accordingly. That way, all the information that needs to be communicated to a speaker can be displayed without further knowledge on how to say any other voice commands to navigate through that complement graphical user-interface.

[0056] One embodiment of the invention provides a graphic user interface enhancement for operating systems and applications wherein the screen text, or objects within the computer that give rise to the screen text, are analyzed and made “speech enabled”. Indeed, objects represented in a graphic user interface not associated with text or semantic labels may also be speech enabled. Thus, many common types of graphic user interface elements, which would normally require a point device initiated event to select and manipulate, can instead be selected or manipulated by an alternate user input, for example speech or keyboard. Preferably, a “hotspot” is presented in the graphic user interface to show screen objects which are recognized and available for manipulation by the alternate input. The hotspot is provided as an overlay, and therefore does not generally interact with the basic screen display elements. A typical layout is shown in FIG. 4, wherein a set of menu options each have an associated spot which indicates that the alternate input has recognized the graphic user interface object and it is available for manipulation. Alternately, for example in a browser context, the hotspots may be generated by modifying the page being displayed through appropriate code manipulation.

[0057] Typically, applications and shells of graphic user interface systems define display configurations by adopting parameters for use of a set of predefined objects, which are then displayed. The proprietary software can therefore analyze the parameters or resulting objects, making them accessible through alternate means from a normal pointing device. In some cases, a “map” or non-standard user interface is used, which does not provide defined high level objects; in that case, a graphic analysis system may be employed to process the display, and determine elements that are likely intended to be graphic user interface elements, for example by performing character recognition algorithms on text. Preferably, this alternate is a speech recognition system. In that case, each user interface object is assigned a label, which should be unique, and which is typically the corresponding spoken version of a text label or common description. Typically, the label will be a single word, often prominently displayed in association with the graphic user interface object. In the event that it is not possible to ensure the uniqueness of a label, the speaker may invoke it by stating the shared label. This invention would then proceed to a disambiguation interaction with the speaker by highlighting all components triggered by the voice command. Only after a successful disambiguation phase between the speaker and the system will a graphical user-interface interaction be generated.

[0058] Therefore, in operation, at least one text label is associated with each object. The text labels are then provided to a speech recognition engine, or the output of a speech-to-text engine used to determine a match with the labels. In either case, immediately after a match is found, or after a successful disambiguation phase, a pointing device event is generated at the location of the associated graphic user interface object, for example a mouse-click event. In some cases, a more complex event is desired, such as a double-click or drag. In that case, a preliminary modifier may be spoken, such as “double-click” or “drag” preceding the label. In the case of a “drag”, a target position is then specified (unless the operation is to be concluded by a different user input device). The target position may itself have a label, or may be represented by a screen location, for example indicated by the grid shown in FIG. 3. Thus, the user could say, and have appropriately recognized, “double-click word”, meaning that the proprietary software has recognized a Microsoft Word icon in Explorer, and that this icon is labeled “word”, the position of which (i.e., center or within a discrete boundary) is then used to generate a double-click event, which would open the “Word” application. In the case of a drag operation, a document file icon with name “Letter” may be opened in Microsoft Word, by speaking “drag . . . Letter . . . to . . . Word”, which corresponds to generating a mouse pointer down event at the “Letter” icon, repositioning the cursor location at the “Word” icon, and generating a mouse pointer up event, thus opening the Letter file in Microsoft Word.

[0059] As an alternate to the hotspots shown in FIG. 4, the display text may be rendered or overlay with a designated text style or display attribute or overlay, for example italic, pink or dynamically changing, to indicated that they are speech enabled. This option is particularly appropriate for use in browsers, since a number of different attributes are controllable, for example in HTML, without altering the screen or page layout and spatial arrangement.

[0060] The foregoing description of the preferred embodiments of the invention is by way of example only, and other variations of the above-described embodiments and methods are provided by the present invention. Components of this invention may be implemented using a programmed general purpose digital computer, using application specific integrated circuits, or using a network of interconnected conventional components and circuits. The embodiments described herein have been presented for purposes of illustration and are not intended to be exhaustive or limiting. Many variations and modifications are possible in light of the foregoing teaching. The invention is limited only by the following claims.