Method and Apparatus for Multi-Window Display With Enhanced Window Manipulation Facilities

1. Field of the Invention

The present invention relates to a multi-window display apparatus for displaying a plurality of image information on a display device of a work station by using windows.

2. Description of the Background Art

One of the important features of a work station is a multi-window display function in which a plurality of windows are manipulated on a display screen in various modes such as moving, enlarging, contracting, and superposing.

An example of a conventional multi-window display apparatus is shown in FIG. 1.

In this apparatus, a graphics controller 101 of the bit block transfer type reads out a bit map from a bit map memory 102, which is a memory for the display which stores bit values of picture elements in one or more bits. The graphics controller 101 then sends the bit map to a frame buffer 103, through which an image given by the values of the picture elements stored in the bit map memory 102 is displayed on a CRT 104. The graphics controller 101 is also connected with a host CPU 106, a main memory device 107, an external memory device 108, a mouse 109, and a keyboard 110 of a work station, through a host bus 105, from which various commands related to the display can be given.

In such a conventional multi-window display apparatus, only rectangular windows are used exclusively, and a user is not provided with choices for the form of the windows, although the size of the windows and the size of letters in the windows can be changed freely. The rectangular windows are also used exclusively in displaying graphic images, regardless of the content of the graphic images to be displayed.

Thus, the conventional multi-window display apparatus has been rather limited in terms of manipulations of windows. For example, it has been impossible to construct a window in such a form from which a content displayed in the window can be easily recognized, or to selectively deal with a part of the window display such as a head portion of a display in a shape of a human body.

It is, therefore, an object of the present invention to provide a method and an apparatus for multi-window display with enhanced window manipulation facilities, which treats all types of the contents of the windows as graphic images.

According to one aspect of the present invention there is provided a multi-window display apparatus, comprising: display means having a display screen; frame buffer means for storing images to be displayed on the display screen of the display means; bit map memory means for storing window images to be displayed in a multi-window display; image processing means for performing various image processing operations on the window images stored in the bit map memory means, by regarding the window images as graphic images; and a graphics controller including bit block transfer means for transferring appropriate sections of window images processed by the image processing means to the frame buffer means.

According to another aspect of the present invention there is provided a method of multi-window display, comprising the steps of: storing window images to be displayed in multi-window display in bit map memory means; performing various image processing operations on the window images stored in the bit map memory means, by regarding the window images as graphic images; transferring appropriate sections of window images processed at the performing step to frame buffer means by using bit block transfer means; and displaying the image in the frame buffer means on display means .

Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings.

FIG. 1 is a schematic block diagram of a conventional multi-window display apparatus.

FIG. 2 is a schematic block diagram of one embodiment of a multi-window display apparatus according to the present invention.

FIG. 3 is a detailed block diagram of an image processing unit of the multi-window display apparatus of FIG. 2.

FIG. 4 is a flow chart for the operation of the image processing unit of FIG. 3 in a contracted display mode.

FIG. 5 is a conceptual diagram of a bit map memory of the multi-window display apparatus of FIG. 2, showing its bit plane structure.

FIG. 6 is an illustration of an example of graphic image to be processed in the multi-window display apparatus of FIG. 2, in a graphical processing mode.

FIG. 7 is an enlarged illustration of a part of the example of FIG. 6 for explaining one method of separating the figure from the background.

FIG. 8 is an illustration of a labelled image obtained by separating out the figure from the background.

FIG. 9 is an illustration of a labelled image incorporating further subdivision.

FIG. 10 is a table of subdivided elements for the labelled image of FIG. 9.

Referring now to FIG. 2, one embodiment of a multi-window display apparatus according to the present invention will be described.

This apparatus comprises a graphics controller 11 for controlling a multi-window display, which includes a bit block transfer unit 111 and a table look up unit 112, a bit map memory 12 for storing images to be displayed in the multi-window display in terms of values of picture elements, an image processing unit 13 for processing the images stored in the bit map memory 12 as graphic images, a CRT 24 for displaying images, and a frame buffer 23 for temporarily storing images to be displayed on the CRT 24. The graphics controller 11 is also connected with a host CPU 26, a main memory device 27, an external memory device 28, a mouse 29, and a keyboard 30 of a work station, through a host bus 25, from which various commands related to the display can be given.

The bit block transfer unit 111 transfers sections to be displayed in the multi-window display selected by the graphics controller 11 from the images stored in the bit map memory 12 processed by the image processing unit 13.

The table look up unit 112 controls the transfer by the bit block transfer unit 111, to be described in detail.

The image processing unit 13 performs various image processing operations on the images stored in the bit map memory 12, by regarding these images as if they are graphic images, regardless of whether they indeed are graphic images or not. Here, all the image processing operations to be performed by the image processing unit 13 are well known operations conventionally performed by a graphic image processing device.

More specifically, as shown in FIG. 3, the image processing unit 13 further comprises an affine transformation unit 14, density transformation unit 15, space filter unit 16, logical filter unit 17, labelling unit 18 and a histogram calculation unit 19, all of which are connected with the bit map memory 12 through a bus 20, and with the graphics controller 11 through control signal line 21.

Briefly, this apparatus operates as follows: first, the image processing unit 13 processes the images stored in the bit map memory 12; then, the graphics controller selects the sections to be displayed in the multi-window display from the processed images; then, the selected sections are transferred to the frame buffer 23 by the bit block transfer unit 111, under the control of the table look up unit 112; and then, the content of the frame buffer 3 is displayed on the CRT 24.

Now, the operation of this apparatus in its various modes of a multi-window display will be described in detail.

MODE 1: Contracted Display Mode.

A so called overlap style multi-window display, in which one window is to be superposed on another, is convenient as a number of windows can be opened, but when the number of windows becomes numerous, it is often difficult to remember the contents of the windows which are overlapped by many other windows.

To cope with this situation, less frequently used windows are contracted and placed around edges of a display screen in this apparatus.

Namely, the less frequently used windows formed in the bit map memory 12 are contracted into a suitable size by the affine transformation unit 14 of the image processing unit 13, and these contracted windows are displayed on the CRT 24 through the graphics controller 11 and the frame buffer 23.

In this contracted display mode, even when the letters of the displayed images becomes too small to read, a pattern of letters arrangement can provide a key for a user to recall the content of the displayed images quickly. In addition, a part of the displayed images may be retained in a readable size, so as to provide an easier clue for the user.

It is noted that in a simple operation, such as this contracted display mode, the processed images obtained by the image processing unit 13 need not be stored in the bit map memory 12, and can be written into the frame buffer 23 directly, without storing.

It is also pointed out that although the simple change of the sizes of the windows and letters can be realized in a conventional work station, the conventional method is inapplicable to a case in which the displayed images include graphic images or tables in addition to the letters. On the contrary, this apparatus can deal with any displayed images including a case involving the graphic images or tables, because the image processing unit 13 treats the entire displayed images as graphic images.

In more sophisticated circumstances, such as those involving graphic images or tables whose contents are difficult to recognize in the contracted display mode, the following operation is carried out according to the flow chart of FIG. 4.

Namely, at the step ST1, a space filter, such as a Sobel operator, is applied to the images by the space filter unit 16, in order to extract edges of the figures in the images. Then, at the step ST2, a binarization is performed on the extracted edges by the density transformation unit 15. Next, at the step ST3, a connected region labelling is performed on the binarized edges by the labelling unit 18. Then, at the step ST4, an area of each connected region is calculated by the histogram calculation unit 19, and the connected regions of small area are discarded by changing their binary values to "0" using the density transformation unit 15, in order to simplify the images. Next, at the step ST5, a logical filter is applied to the simplified images by the logical filter unit 17, in order to thicken the images. For example, the binary values of four nearest neighbor picture elements of a picture element having a binary value of "1" are changed into "1" at this step. Alternatively, the binary value of a picture element having a binary value "0" is changed into "1" when one of its four nearest neighbor picture elements has a binary value of "1" at this step. Finally, at the step ST6, the affine transformation is applied to the thickened images by the affine transformation unit 14, so as to obtain the contracted images. The resulting contracted images are then displayed on the CRT 24 as the images in the contracted display mode. By this process, the characteristic features of the original images are emphasized, so that recognition of the images in the contracted display mode becomes easier.

MODE 2: Figure Manipulation Mode.

In this mode, figure manipulation bits are allocated to each image in the bit map memory 12, in addition to the values of picture elements for the image.

Namely, as shown in FIG. 5, for the image comprising a letter "A", values of picture elements for the letter "A" are stored at a "level 1" bit plane, while a region corresponding to this letter "A" (shaded region) is recorded by the figure manipulation bits at a "level 0" bit plane.

The figure manipulation bits are utilized by the table look up unit 112 of the graphics controller 11 in controlling the transfer by the bit block transfer unit 111. For example, the table look up unit 112 controls the bit block transfer unit 111 such that only those regions which have a binary value "1" for its figure manipulation bit are transferred to the frame buffer 28, while the other regions having the figure manipulation value "0" are ignored, so that the original content of the frame buffer 23 remains unchanged in those region. This enables the bit block transfer unit 111 actually to deal with a rectangular region, while, in effect, dealing with non-rectangular region, i.e., the region of the figure manipulation bit value "1" can be distributed arbitrarily.

Furthermore, the figure manipulation bits can also be utilized in the following ways.

(1) Displaying a Window of an Irregular Form.

The window of an irregular form can be displayed by arranging the region of the figure manipulation bit value "1" in a desired form, on a rectangular (regular) window region formed in the bit map memory 12. The form of the window in this case may be that which is obtained by the image processing unit 13 as a result of a cutting out operation. In a case of writing letters in the window of irregular form, the line should be changed whenever the letter line reaches to the region of the figure manipulation value "0".

(2) Omitting Regions Containing No Information.

For the images of letters in the bit map memory 12, only those regions corresponding to the letters are given the figure manipulation bit value "1", as shown in FIG. 5. Also, a region corresponding to one letter may be allocated for each carriage return code.

In this manner, the remaining regions are ignored in the transfer to the frame buffer 23, so that the content of the previous window which is overlapped by the present window remains visible in those remaining regions. Similarly, in a case of graphic images, only those regions corresponding to the figure elements are given the figure manipulation bit value "1". In such a case of the graphic images, if the regions of the figure manipulation bit value "1" are limited to the immediate vicinity of line elements appearing in the graphic images, it is often difficult to distinguish the content of the present window from the content of the previous window. To cope with this, the connected region labelling is performed for the graphic images, and rectangles enclosing the connected regions are designated for the regions of the figure manipulation bit value "1". Alternatively, the thickening of the graphic images used in the contracted display mode described above can be applied to the graphic images.

(3) Partial Display of a Window.

As with case (1) above, only a part of the window can be displayed by arranging the region of the figure manipulation bit value "1" in a desired form, on a rectangular (regular) window region formed in the bit map memory 12. For example, only a rectangular sub region of the original rectangular window region which frames around the significant information can be displayed by using the appropriate figure manipulation bit values. Alternatively, only a non-rectangular sub region of the original rectangular window region which is shaped in a form of the significant information itself can be displayed by using the appropriate figure manipulation bit values.

MODE 3: Graphical Processing Mode.

Because the entire images in the bit map memory 12 are treated as graphic images by the image processing unit 13 in this apparatus, various processing operations which are conventionally limited to the purely graphic images can be applied to any image to be displayed in the multi-window display.

As an example, a case of processing a graded density level graphic image shown in FIG. 6, which includes three figures A1(cloud), A2(person), and A3(tree) will be described. In this case, the bit map memory 12 stored the figure manipulation bits at the "level 0" bit plane, and an n-th grade density picture element value at an "n-th level" bit plane.

Now, in order to be able to perform operations such as moving, enlarging, contracting, or coloring with respect to each figure independently, the figures A1, A2, and A3 have to be separated out from the background first.

This can be achieved in various different manners. For instance, one method, effective in a case having a sufficient contrast between the figures and the background, comprises the steps of obtaining the histogram for the density levels by using the histogram calculation unit 19, determining an appropriate separation threshold density level, separating out by using the binarization by the density transformation unit 15, and labelling the binarized image by the labelling unit 18. Alternatively, the separation can be achieved by tracing the outlines of the figures A1, A2, and A3 using a pointing device such as the mouse 9, painting the interior regions of the traced outlines, and labelling. The separation may also be achieved by the painting of the figures from which the binary image with figures painted is obtained, and the labelling of the painted out binary image. Another method for achieving the separation is first encircling each figure with an enclosing line M as shown in FIG. 7 specified by using the mouse 9, obtaining the histogram within the enclosing line M, determining a separation threshold, binarizing, and labelling. The mouse 9 may be used for the purpose of correcting in these methods.

As a result of the separation using any one of the methods described above, a labelled image shown in FIG. 8, in which the figure A1 is given a label 1, the figure A2 is given a label 2, and the figure A3 is given a label 3, is obtained.

Now, by utilizing these labels as the figure manipulation bit values, each figure can be regarded as a window of an irregular form, and various image processing operations can be performed with respect to each figure independently.

In addition, as shown in FIG. 9, the labels can be given in a manner distinguishing different parts of the figures. For example, in FIG. 9, the figure A2 of a person is subdivided into a head 2, a right arm 4, an upper torso 5, a left arm 8, a lower torso 8, a right leg 9, and a left leg 10, while the figure A3 of a tree is subdivided into leaves 3 and a trunk 7. With this subdivision, each subdivided part can be processed independently. This subdivision can be specified either automatically by the image processing unit 13, or by the user through the mouse 9 interactively.

In a case of using such a subdivision, a table of subdivided parts shown in FIG. 10 is set up in the table look up unit 112 of the graphics controller 11. In this table of FIG. 10, a number of labelled elements are grouped together in operation level 2 and operation level 3, such that in operating at the operation level 1, each labelled element can be processed independently, while in operating at the operation level 2, an upper body of a person comprising the head 2, right arm 4, upper torso 5, and left arm 6, or a lower body of a person comprising the lower torso 8, right leg 9, and left leg 10, or a tree as a whole comprising the leaves 3 and the trunk 7, can be processed collectively as the bit block transfer unit 111 transfer these elements collectively. Also, in operating at the operation level 3, each one of the figure A1, A2, and A3 as a whole can be processed collectively, as in a case of FIG. 8.

It is pointed out that because the image processing unit 13 of this apparatus is capable of performing various image processing operations, it can be utilized in carrying out image processing on an associated work station. Conventionally, the image processing on a work station can be achieved by connecting an additional image processing device to the work station, in which case it has been necessary to transfer the image data resulting from the image processing which are stored in an image memory, to the bit map memory. In contrast, such a transfer of the image data is unnecessary in this apparatus, so that the display speed can be increased, and the image data and text data can be dealt with in a unified manner.

It is to be noted that a portion of the bit map memory 12 in the above embodiment can be utilized as a frame buffer, instead of providing a separate frame buffer 23 as in the above embodiment.

Besides this, many modifications and variations of the above embodiments may be made without departing from the novel and advantageous features of the present invention. Accordingly, all such modifications and variations are intended to be included within the scope of the appended claims.