Walk-through human/machine interface for industrial control

[0002] The present invention relates to industrial control systems and, in particular, to a human/machine interface for the programming and monitoring of control programs used for the control of factories and the like.

[0003] Industrial controllers such as those described in U.S. Pat. Nos. 3,810,118; 3,942,158; 4,165,534; and 4,442,504 are high-speed computers that may be connected to industrial equipment, such as automated assembly lines or machine tools, to operate such equipment in accordance with a stored control program. The stored program includes instructions, which when executed, examine the condition of selected inputs to the controller from sensing devices on the controlled equipment, and energize or de-energize selected actuators or outputs from the controller to operate devices on the controlled equipment.

[0004] Inputs to the industrial controller may be discrete binary signals, such as those from switches, which may detect limits of process variables such as motion, temperature, time, or other quantities, or the inputs may be analog measures of process variables which are generally then converted to digital words for processing, or the inputs may be multi-variable information.

[0005] Similarly, the outputs of the industrial controller may be either discrete binary signals as produced, typically, by mechanical or solid state relays; analog outputs produced by means of a digital to analog converter; or multi-variable commands. Some of the inputs and outputs to the centralized controller may be remotely located and connected to the controller by means of a digital communications link. Typically, the network connects the controller with a remote I/O rack at which a number of inputs and outputs are clustered.

[0006] Current programming tools may display a view of the instructions of the control program as graphical elements (such as relay coils and contacts or function blocks) to make the control program easier to understand. Certain instructions may include captions indicating the state of their data or a view of the data may be displayed in table form. Animated two-dimensional representations of machines, including for example, pumps or motors or reaction vessels, have been used to provide a view of certain process variables such as “on” or “off” states or liquid level or temperature.

[0007] Present generations of industrial controllers may be extremely complex with thousands of control points distributed over a factory having dozens of machines coordinated through the operation of the control program. Unlike conventional computer programs of equal complexity, the operation of the control program is highly dependent on intervening real-time physical processes (the operation of the machines of the factory). While the state of this controlled equipment is largely defined by its input and output data, viewing and understanding this data in the context of the control program is difficult. Further, each of the controlled machines normally includes some processing capability so the data to the machine may not directly represent its state. To the extent that the control program is divided among a central processor and machines themselves, gaining a complete understanding of the control process as is necessary for design and monitoring of the control process is difficult.

[0008] Current limitations in representing large control programs and its associated data creates an effective limit to the complexity of control systems even as larger control structures coordinating the operations of larger systems are demanded. What is needed is a way to integrate views of large and possibly distributed control programs, their associated control data and the actual physical processes in a logical and intuitive manner.

[0009] The present invention creates an intuitive multi-viewed representation of a control system using the paradigm of the factory floor. Software “objects” are created representing the machines, their status, their programming, and their data each having a defined visual representation and three-dimensional spatial coordinate. In this way, relevant portions of the control program and data may be associated with a particular machine using the mnemonic of physical location.

[0010] These spatial “objects” may also serve as “containers” allowing certain objects to be placed within other objects in nested configuration providing yet another dimension of organization.

[0011] Specifically, the present invention provides a human/machine interface for use with an industrial control system including at least one electronic computer controlling the operation of an industrial process through the use of a stored program. The stored program receives data from the industrial process in the form of control inputs and provides data to the industrial process in the form of control outputs. The industrial process includes a plurality of machines operating in a factory and the industrial control system communicates with the machines over a network. The invention includes a visual display and a multi-dimensional input device (such as a mouse) communicating with a computer on the network. The computer executes an interface program to provide a three-dimensional representation of a virtual factory with images of the machines dispersed in three-dimensions within the virtual factory and images of portions of the control program associated with the control of the given machine linked to the image of the given machine. The program responds to inputs from the multi-dimensional input device to change the viewpoint allowing the user viewing the visual display and operating the multi-dimensional input device to obtain the perception of moving through the virtual factory.

[0012] Thus, it is one object of the invention to provide an intuitive framework for organizing and displaying the data and program of an industrial control system. The framework of spatially defined objects dispersed in three-dimensions is easily understood by human operators.

[0013] The images displayed may include an image of the data and the computer may further execute the interface program to animate the images in response to signals on the electrical network to depict operation of the industrial process. The type of animation may include the state of operation of a machine represented by the image, the values of data used by the machine represented by the image, a point of execution in the portion of the control program represented by the image; or a flow of data between machines.

[0014] It is thus another object of the invention to integrate a view of control data and machine state into the interface of the present invention as physical objects.

[0015] The interface program executed on the computer may link images by placing images of program portions, data or machine state visually within the images of the machines.

[0016] Thus, it is yet another object of the invention to allow for a visual parsing of an extremely complex control program among controlled devices. Collecting objects to be contained within its machine simplifies the overwhelming complexity of large industrial control programs.

[0017] The interface program executed on the computer may respond to input from the multidimensional input device to allow the user to select images and a selected image may be moved by further input from the multidimensional input device to change the linking between images of portions of the control program and images of machines.

[0018] It is another object of the invention to provide a powerful human/machine interface making use of the paradigm of object oriented programming. By representing software objects as tangible three-dimensional objects that may be moved around a virtual factory, use and reuse of software objects under the paradigm of object oriented programming is made immediate and clearer.

[0019] The foregoing and other objects and advantages of the invention will appear from the following description. In the description, reference is made to the accompanying drawings which form a part hereof and in which there is shown by way of illustration, a preferred embodiment of the invention. Such embodiments do not necessarily represent the full scope of the invention as claimed.

[0020] FIG. 1 is a simplified perspective view of a control system useful for controlling and coordinating machines within a factory;

[0021] FIG. 2 is a fragmentary perspective view of a factory showing the control system of FIG. 1 and its associated machinery as located within the factory;

[0022] FIG. 3 is a three-dimensional representation of a virtual factory corresponding to the factory of FIG. 1 showing spatially linked and spatially independent objects;

[0023] FIG. 4 is a figure similar to that of FIG. 3 showing use of two of the spatially independent objects providing tool functions and showing spatially linked objects within other spatially linked objects, the later rendered transparent;

[0024] FIG. 5 is a figure similar to that of FIG. 4 showing use of the spatially independent cursor object to move a spatially linked program code object between machines;

[0025] FIG. 6 is a graphic representation of a scene graph employed by the present invention to track the hierarchy and association of different spatially linked objects the latter represented by nodes;

[0026] FIG. 7 is a blocked diagram of the various portions of the operating system of the present mention including a VRML rendering engine for rendering the scene graph of FIG. 6 and a node editor for modifying the nodes of the scene graph of FIG. 6; and

[0027] FIG. 8 is a flow chart showing the operation of the node editor in modifying the nodes of the scene graph of FIG. 6.

[0028] Referring now to FIG. 1, an industrial control system 10 includes a central control unit 12 typically providing a housing with a common back plane into which one or more modules 14 may be inserted including a central processing unit 16 and a communications adapter 18.

[0029] The central processing unit 16 is connected to a terminal 20 providing a visual display 22 and one or more user input devices 24. The display may be a cathode ray tube or other flat panel type display, a head mounted display, such as goggles or glasses employing a small liquid crystal or cathode ray tube elements for each eye to provide stereo optic effects, or a virtual retinal display in which an image is projected directly on to the retina of the eyes. Alternatively, the display may be a so-called ‘cave’ in which multiple electronic projection screens form an enclosed virtual environment. The cave may be used with shutter glasses or the equivalent to provide a three dimensional effect. Such display types are well known in the art.

[0030] The input device 24 may be a conventional keyboard and mouse, or a spaceball (as shown) the latter providing a sphere with sensors allowing six degrees of input, or other similar devices such as a space-mouse, ‘wand’, haptic or data-glove as are known in the art. The input device may further be a ‘haptic’ device, providing force feedback as well as input, or a voice input system or eye motion input system (gaze tracker), all well known in the art for providing for multi-dimensional input for a human operator.

[0031] The central processing unit 16 communicates with the communications adapter 18 which is connected to a common network 26 which in turn connects with one or more remote units 28. In a conventional control system, remote units 28 provide input output functions only, however, they may also provide separate computing capability to execute portions of a control program also executed by the central processing unit 16.

[0032] Each remote unit 28 includes an I/O interface 30 having lines 32 exchanging input and output signals with controlled equipment (not shown in FIG. 1). The I/O interface 30 may be essentially free-standing as a highly distributed control node 33 also connected to network 26.

[0033] Referring now also to FIG. 7, the central processing unit 16 in the central control unit 12 includes a processor 38 communicating with a memory 40 to execute the control program 42 contained therein. In the case where program portions 48 of the control program 42 are contained in the remote units 28 (or in the controlled machines themselves) copies of those program portions 48 may also be contained in memory 40 for reference. Typically, also the control program 42 as executed by the processor 38 will have portions 49 uniquely associated with particular machines 34 insofar as the portion 49 produce outputs relevant to that machine 34 or is responsive to inputs from the machine 34.

[0034] During operation of the control system 10, input and output data is received through the communications adapter 18 and stored in an I/O image table 44 contained in the central processing unit 16 and accessible by the processor 38. The communications adapter asynchronously updates the I/O image table 44 to contain current values of the input data. The processor 38, then executes the control program 42 to read inputs from the I/O image table 44 and based on the logic of the control program 42, write new outputs to the I/O image table 44. These outputs are then asynchronously transmitted to the controlled machines 34 by the communications adapter 18.

[0035] Referring now to FIG. 2, the controlled machines 34 connected to I/O interface 30 of FIG. 1 may be located about a factory environment 36, interconnected by network 26. The factory environment 36 presents a volume in which each machine 34 has a defined three-dimensional coordinate location. The locations are generally determined by the physical processes executed by the machines and their interaction in executing those processes.

[0036] Referring now to FIGS. 2, 3 and 7, the present invention uses the factory environment to provide a three-dimensional paradigm for presenting information about the control system 10 useful for design, programming, control and maintenance of factory processes. This is accomplished by a human/machine interface provided by an “application oriented” operating system 50 as may be run in the processor of the terminal 20.

[0037] A principal feature of the applications oriented operating system 50 is that it provides a three-dimensional virtual environment 52 representing the factory environment 36, through which a human operator may move. This is accomplished by means of a VRML rendering engine 54 capable of reading a scene graph 56 and producing three-dimensional representations of the factory environment 36 and the machines 34, and displaying the same on the display 22 from different viewpoints determined by signals received from the input device 24. One commercially available VRML rendering engine 54 suitable for use with the present invention, is the (3D Studio Max by Kinetix) Cosmo Player by Silicon Graphics. .

[0038] Referring to FIGS. 3, 6 and 7, prior to the three-dimensional rendering of the virtual environment 52, the factory and its machines must be is described by a scene graph 56 (shown in FIG. 6) and presenting a mathematical description of the walls 60, floors and doorways of the factory represented in the virtual environment 52. The scene graph 56 preferably follows the VRML (virtual reality markup language) well known in the art in which scene elements are arranged as nodes on a tree structure. The graph structure is implemented by a corresponding data structure stored in memory as ASCII text with each node being a record and indicating other records that are its children and holding data describing the node, termed “properties”.

[0039] An advantage of the VRML scene graph implementation is that each nodes spatial coordinates are defined relative to its parent node and hence children nodes “move” with the parent node when the coordinates of the parent node are changed. Generally the coordinates include x, y and z Cartesian coordinates as well as rotative coordinates of roll, yaw, and pitch. This allows visual objects, represented by children nodes, to be placed “withthin” other visual objects represented by parent nodes.

[0040] The scene graph 56 links to a root or “world” node 82 a number of wall nodes 83 collected by a group node 84 labeled “room”. Each node, as well as having a place in the hierarchy of the scene graph 56 and relative coordinates, as described above, has a number of properties including generally a shape and texture and can perform certain “methods” as will be described. In the present invention the textures 63 may be conventional building materials or may be fanciful textures such as the sky or the like. Multiple rooms of arbitrary complexity may be created by adding new nodes 84 and 83.

[0041] The virtual environment represented by nodes 84 and 83 may serve as an organizational structure for the large amount of information relevant to the control system 10 represented as objects 64. Each object 64 is represented by one node in the scene graph 56, for example, a manipulator on the factory floor may be represented by a node 64. Henceforth the same number will be used to designate the object and the node.

[0042] The node 64 for the manipulator provides a three dimensional depiction of the manipulator through a number of children “machine element” nodes 86 having shape and texture properties (and possibly composed of other children nodes not shown) to depict the superficial aspects of the machine 34. The depiction of the manipulator may be animated using conventional VRML techniques responsive to variables provided by the node editor as will be described.

[0043] The nodes of the present invention provide more than visual representations of the objects but also include an extensive list of properties that reflect the data associated with the represented object, for example, its input and output data and the portions of the control program relevant to the object. The nodes also provide “methods” or software routines allowing them to process data represented by properties of their own or other nodes (particularly parent nodes). The interconnections of the scene graph also provide paths of data flow between methods of the nodes.

[0044] Referring still to FIG. 6, and as mentioned above, the organization of the scene graph 56 allows nodes for particular objects 64 to be associated with other nodes of object 62 (both with respect to space and data connections) for intuitive organization of the great amount of data represented by the factory automation system. Thus, the manipulator node 64 may have a dependent computer node 83 having its own properties and providing (through other nodes not shown) a similar depiction of a component computer, which according to the hierarchy of the scene is contained in the manipulator 64. As well as being placed in the manipulator physically, the computer node connects to the properties of the manipulator node so as to, for example, animate itself according to the manipulator properties, through its internal methods. Similarly, the manipulator node may respond to the operation of the computer node, again through a linking of properties.

[0045] The computer node 88 in turn holds nodes for a status object 89, a code node 87, and a communications port object 109, each of which provides a visual representation of the associated object, even though, in this case, some of the objects have no physical presence. The code node 87, for example, provides a representation of the portion 49 of the control program 42 related to the manipulator object of node 64, a status node 89 being a state representation indicating the status of the manipulator 64 and having no physical counterpart, and a communications port node 109 providing a window to data communicated over the network 26 to the manipulator 64.

[0046] Generally each node includes the property of visibility and thus its associated object may become invisible or transparent allowing this nesting of objects in other objects to be properly displayed on the visual display 22 and the display to be simplified when all components to nodes need not be displayed.

[0047] Referring again to FIG. 7, generally, the VRML rendering engine 54 accepts inputs from the terminal 20 from the multi-dimensional input devices 24, defining the coordinates of the user, to allow viewing of the objects 64 via their images in the scene graph 56 as will be described. Other inputs from the user are used to manipulate the scene graph 56 in more fundamental ways including moving objects and changing their properties.

[0048] Most VRML rendering engine 54 allow for the depiction not only of spatially linked objects 64, fixed with respect to the coordinates of the virtual environment 52, but also of spatially independent objects 67 that move with the human operator as the operator moves through the virtual environment 52. This is done by establishing a user node 88 whose coordinate properties mirror the user's coordinates in the virtual factory. The user node 88 serves as a parent node for objects 67 that follow the user around. In the present invention, these objects 67 are used to represent tools that can be accessed by the user as the user is at any location within the three-dimensional virtual environment 52.

[0049] In the preferred embodiment, five tools are provided. The first is a meter tool 68 for reading qualitative data from objects 64 whose physical counterparts have physically measurable parameters associated with the quantifiable data. A watch tool 70 is used for measuring real-time occurrences in the factory environment as reflected in its virtual depiction or for modifying the relationship between virtual time and real time in instances where troubleshooting and simulation of the factory may be desired. A stethoscope tool 72 is used to permit the user to “observe” qualitatively physical aspects of the machinery represented by the object 64, such as sound or vibration. A magnifying glass tool 74 allows enlargement of the three-dimensional virtual environment 52 allowing yet a fourth effective dimension of movement. Finally a cursor 76 allows interaction between the user and the object 64 either to select, move, copy or otherwise rearrange those objects 64.

[0050] Referring still to FIG. 7, the present invention augments what would otherwise be a passive representation of the factory by means of a node editor 78 which passes external data to the nodes which may respond, through their methods, to such data by animation or the like in coordination with input output data from the I/O image table 44 the latter which reflects the state of the objects 64. The state may be a physical movement of the object 64 in the case of a machine 34 or when the object represents an intangible aspect of the control process a visual representation of the state of that object. For example, when the object 64 is a program portion 48 or 49 of the control program 42, a visual representation of the executing instruction may be provided.

[0051] The node editor 78 also operates directly on the scene graph 56 to perform normal editing functions such as cutting, copying and pasting. In this way objects 64 may be readily duplicated and reused as is well-known in the art. The node editor 78 also allows the generation of new nodes either by entering the required ASCII text or through the use of higher level tools well known in the art. Importantly the node editor 78 allows the location of the objects represented by the nodes to be moved about the virtual factory by changing their coordinate properties.

[0052] Referring now to FIG. 4, the visibility of a particular object 64 may be changed through the node editor 78 receiving cursor commands from the devices 24 specifically by the user manipulating the cursor 76 to be superimposed over the image of the object 64 within the three-dimensional virtual environment 52 and to toggle the visibility property in the node of that selected object 64 using a predetermined keystroke on the input devices 24. In the example of FIGS. 6 and 4, the manipulator object 83 has been made transparent revealing the status object 89 and the program portion object 87. The coordinates of the cursor are compared to the absolute coordinates of the nodes to identify the relevant node in the scene graph.

[0053] Referring to FIG. 8, the above described selection process is performed by the node editor 78 from commands from the terminal 20 as indicated by process block 94. The keystroke command indicating that the object should be opened to reveal interior objects or closed in the reverse case is indicated by decision block 96. The changing of the visibility property of scene graph 56 is indicated by process block 98.

[0054] Using this procedure, complex interrelationships between the physical, software and state elements of a control system 10 may be logically illustrated and the information organized in an intuitive manner.

[0055] Referring to FIGS. 4, 6 and 8, the cursor 76 may also be used to perform a dragdrop operation familiar to those using mouse based programs but in three-dimensions. In such an operation, the cursor 76 is placed over the image of an object 64 and a predetermined capture key on the input device 24 is pressed and retained in a pressed state until the cursor 76 is moved to a new location within the virtual environment 52 carrying the selected object 64. A drag-drop operation is indicated by decision block 100 of FIG. 8. The capture key like all predetermined key commands, need not be a button pressing but may be a verbal command or the like, or in the case of the use of a data glove, a particular grasping motion of the fingers of the hand.

[0056] If the drag drop operation culminates with the selected object 64 being placed on a portion of the factory where there is no other object 64 as determined at decision block 102, then the object's node coordinates 90 in FIG. 6 are changed if the object is a stationary object 64. If the object is a spatially indifferent object 67, (e.g., a tool) the tool snaps back to its location at the top of the display of the three-dimensional virtual environment 52. These outcomes are indicated by process block 104. Generally, movement of an object 64 which contains other objects 64 moves all of the contained objects 64 as well provided the container object 64 is visible. Invisible objects 64 are generally not moveable until they are made visible. The container relationship and the visibility of objects is determined from scene graph 56.

[0057] At process block 106, if the culmination of the drag-drop operation is that the selected object 64 is on top of another object 64, then the objects 64 of the child node takes its arguments from the parent node's properties and, if necessary, the parent node connects to the child node's properties. For example, the meter tool 68 has an input argument accepting a physical quantity output type that may be a physical quantity reflected in data of a property associated with another object 64. Placement of the meter tool 68 on a particular stationary object 64 provides a reading of physical quantity I/O data in quantitative form that is part of the properties of the node of the stationary object 64. An example stationary object would be a terminal block object 109 providing a visual terminal for electrical values and having a voltage property. As a second example, the communications port object 91 may have communications data as a property readable by the meter tool 68. The identification of objects that should have their arguments connected is by parent child relationships of the scene graph 56. All properties of the nodes, representing data associated with the nodes, may be grouped according to data type so that such linkages may be established automatically. Thus the meter tool methods indicate the type of data they may accept (e.g., voltage, temperature, logic) and when connected to another node as a child may search for relevant data types and automatically connect to these devices by reading the properties associated with the data types.

[0058] The stethoscope tool 72 operates in a similar manner but providing qualitative readings of I/O data such as may be deduced from one or more property types according to general rules. For example, the stethoscope tool 72 may deduce machine operating temperature from measuring a property recording a duty cycle of the machine motor and display the output as temperature or may output a machine sound indicating operation of the machine deduced from sensors indicating machine component motion accessible as yet another property of the node for that component.

[0059] Alternatively, when the selected object was not a tool but was a stationary object 64, the properties connected are common inputs and outputs according to the context of the connection. Thus, the program portion object 87 if moved into the computer object 83 of another manipulator object 83 adopts the arguments of the second manipulator object 83 so as to program it to operate in the same way as the first manipulator. This movement of program portions objects 87 is reflected in a change in the control program 42 or 49 where such portions ultimately reside in the control system 10.

[0060] Once the program portion object 87 is revealed in its container object 64 (by resetting the visibility flag 88 of the container object), the node editor 78 provides for animation of the instructions of the program portion object 87 as it is executed.

[0061] Generally, the program portion object 87 will be a predefined program portion 48 or 49 however when no portion of the program 42 has been expressly allocated to a node, the program portion object 87 may be generated by the node editor 78 which selects instructions for the program portion object 87 that are relevant to the object 64 of its parent node.

[0062] In a similar manner, the status object 89 accepts as its argument properties of the device of its parent node.

[0063] Referring again to FIG. 6, at decision block 108, if the object selected is a spatially indifferent object 67, then after properties are connected per process block 106, the tool method is run, for example, to display a meter output. Otherwise, at process block 112, the hierarchy of scene graph 56 is changed so that the new spatially linked object 64 is inserted into the object 64 on which it is placed, the latter which serves as a container.

[0064] Per conventional scene graph manipulation, nodes representing objects may be copied, cut and pasted thereby duplicating the objects 64 in this manner. As represented in FIG. 5, the cursor 76 may be used to select a program portion object 87 used, for example, in one manipulator, copy it, and move it to a second manipulator so as to program that second manipulator to operate in the same manner.

[0065] As well as providing for organizational structure for the complexity of a control system, the objects 64 may be monitored through the present system to provide for a review or troubleshooting of the industrial control system 10. Animation of the machine objects 64 are such as to represent their actual movement in the factory, however, non-moving objects such as the communications network 26 may also be animated, for example, through the use of moving arrowheads 114 indicating data flow. Non programmed objects 64″ such as manufactured product, may also be animated based on input output signals indicating their presence as sensed by other objects 64 within the factory.

[0066] Thus, the present invention provides a highly intuitive method of organizing a large amount of data associated with an industrial control process including data about the hardware, the operating environment and the particular application programs as well as the states of each.

[0067] Many other modifications and variations of the preferred embodiment which will still be within the spirit and scope of the invention will be apparent to those with ordinary skill in the art. In order to apprise the public of the various embodiments that may fall within the scope of the invention, the following claims are made: