Apparatus and method for applying effects to graphical images

[0002] This application claims priority from provisional application Ser. No. 60/042,694, filed Apr. 4, 1997, entitled “APPARATUS AND METHOD FOR VIDEO COMPOSITING” and bearing, which is incorporated herein, in its entirety, by reference.

[0003] The invention generally relates to computer graphical production devices and, more particularly, the invention relates to applying effects to graphical images on a computer system.

[0004] Special effects (“effects”) commonly are added to graphical images in a motion picture by means of conventional graphical editing application programs such as, for example, ADOBE PREMERE™, available from Adobe Systems Incorporated of San Jose, Calif. Such programs may include known effects such as, for example, zooming, rotating, and trimming. Edited motion pictures often may be previewed in a playback window on the computer system.

[0005] The functionality provided by conventional graphical editing programs, however, typically requires extensive utilization of the central processing unit and random access memory within the computer system. More particularly, the central processing unit performs all of the calculations required to apply effects to graphical images. After these calculations are completed by the central processor, the edited graphical image may be directed to a graphics accelerator for display on a display device. One such graphics accelerator is the REALIZM™ video card, available from Intergraph Corporation of Huntsville, Ala. Such graphics accelerator utilizes the commonly known OPENGL™ application program interface (available from Silicon Graphics, Inc. of Mountain View, Calif.) to display the edited graphical image. For more information relating to OPENGL™, See, for example, Inside OLE, two-dimensional. Ed., by Kraig Brockschmidt (Microsoft press 1995); The OPENGL Technical Library, which contains The OPENGL Programming Guide(ISBN: 0-201-63276-4), and The OPENGL Reference Manual(ISBN: 0-201-46140-4), by the OPENGL Architecture Review Board; The OPENGL Graphics System: A Specification, Version 1.0 et seq., by Silicon Graphics, Inc. (1992); Microsoft OPENGL Installable Client Driver Specification, Microsoft Corporation; Computer Graphics Principles and Practice, by Foley van Dam (Addison-Wesley 1996); OPENGL Programming Guide, (ISBN: 0-201-46138-2), by the OPENGL Architecture Review Board. These related documents are incorporated herein, in their entireties, by reference.

[0006] Utilizing the central processing unit for performing such calculations, however, undesirably slows computer system performance. The art has responded to this problem by providing one or more dedicated co-processors for performing the necessary calculations. Although typically more efficient than utilizing the central processing unit, co-processors are expensive, thus increasing the cost of editing graphical images with such graphical image editing programs. Moreover, co-processors typically are difficult to program.

[0007] In accordance with one aspect of the invention, an apparatus and method of applying an effect to graphical data utilizes a graphics processor to apply the effect to a graphical image having the graphical data. To that end, the graphics processor is configured to process graphical data in accordance with a preselected graphics processing format, and the effect and graphical image are defined and converted, respectively, into the preselected graphics processing format. In accordance with preferred embodiments of the invention, the graphics processor is controlled to apply the effect to the graphical image to produce an output graphical image. The output graphical image includes both the effect and the graphical image. In preferred embodiments, the graphics processor is a graphics accelerator card, and the graphics processing format is

[0008] In accordance with other aspects of the invention, the computer system includes a central processing unit, in addition to the graphics processor. The effect may be applied to the graphical image so that the effect is normalized with respect to the graphical image. Specifically, the effect may be applied to the graphical image as a ratio of the size of the graphical image. Moreover, the output graphical image may be stored in volatile or non-volatile memory in the computer system as a frame or other type of graphical image. Alternatively, the effect may be applied to the graphical image within texture memory that is a part of the graphics processor. The texture may be the graphics processor format for the input image.

[0009] In yet other embodiments, the effect may include an initial effect and a final effect. In preferred embodiments, the effect may be applied to the graphical image by interpolating between the initial effect and the final effect to provide a continuous effect. The graphical image may be any known graphical image that may be converted into computer readable form. For example, the graphical image may be a video image having a plurality of fields, or a motion picture having a plurality of frames.

[0010] Preferred embodiments of the invention are implemented as a computer program product having a computer usable medium with computer readable program code thereon. The computer readable code may be read and utilized by the computer system in accordance with conventional processes.

[0011] The foregoing and other objects and advantages of the invention will be appreciated more fully from the following further description thereof with reference to the accompanying drawings wherein:

[0012] FIG. 1 schematically shows an exemplary computer system that may be utilized with a preferred embodiment of the invention.

[0013] FIG. 2 shows a process of applying special effects to a graphical image in accordance with a preferred embodiment of the invention.

[0014] FIG. 3 shows a graphical user interface that may be utilized to apply effects to a graphical image in accordance with a preferred embodiment of the invention.

[0015] In accordance with preferred embodiments of the invention, a graphics accelerator configured for use with the OPENGL™ application program interface (a/k/a OPENGL™ format) applies an effect that is in the OPENGL™ format to a graphical image that also is in the OPENGL™ format. Details of preferred and other embodiments are discussed below.

[0016] FIG. 1 illustrates the system architecture for an exemplary computer system 100, such as an Integraph model number TDZ2000 computer system (available from Integraph Corporation of Huntsville, Ala.), on which the disclosed method and apparatus for applying effects to graphical data can be implemented. The exemplary computer system of FIG. 1 is discussed for descriptive purposes only, however, and should not be considered a limitation of the invention. Although the description below may refer to terms commonly used in describing particular computer systems, the described concepts apply equally to other computer systems, including systems having architectures that are dissimilar to that shown in FIG. 1.

[0017] The computer 100 includes a central processing unit (CPU) 105 having a conventional microprocessor, random access memory (RAM) 110 for temporary storage of information, and read only memory (ROM) 115 for permanent storage of read only information. A memory controller 100 is provided for controlling system RAM 110. A bus controller 125 is provided for controlling a bus 130, and an interrupt controller 135 is provided for receiving and processing various interrupt signals from the other system components.

[0018] Mass storage may be provided by known non-volatile storage media, such as a diskette 142, a digital versatile disk (not shown), a CD-ROM 147, or a hard disk 152. Data and software may be exchanged with the computer system 100 via removable media, such as the diskette 142 and the CD-ROM 147. The diskette 142 is insertable into a diskette drive 141, which utilizes a diskette drive controller 140 to interface with the bus 130. Similarly, the CD-ROM 147 is insertable into a CD-ROM drive 146, which utilizes a CD-ROM drive controller 145 to interface with the bus 130. Finally, the hard disk 152 is part of a fixed disk drive 151, which utilizes a hard drive controller 150 to interface with the bus 130.

[0019] User input to the computer 100 may be provided by a number of devices. For example, a keyboard 156 and a mouse 157 may be connected to the bus 330 by a keyboard and mouse controller 155. An audio transducer 196, which may act as both a microphone and a speaker, is connected to the bus 130 by audio controller 197. It should be obvious to those reasonably skilled in the art that other input devices, such as a pen and/or tablet and a microphone for voice input, may be connected to computer 100 through bus 130 and an appropriate controller. A direct memory access (DMA) controller 160 is provided for performing direct memory access to system RAM 110. A visual display may be generated by a video controller 165, which controls a graphics accelerator 167 and a display device 170. In preferred embodiments, the graphics accelerator is a REALIZM™ video card, available from Intergraph Corporation of Huntsville, Ala. As is known in the art, the REALIZM™ video card is configured for use with the OPENGL™ application program interface (“API”) for rendering three dimensional (“3D”) images on the display device 170.

[0020] A network adapter 190 also may be included to enable the computer system 100 to be interconnected to a network 195 via a network bus 191. The network 195, which may be a local area network (LAN), a wide area network (WAN), or the Internet, may utilize general purpose communication lines that interconnect a plurality of network devices.

[0021] The computer system 100 preferably is controlled and coordinated by operating system software, such as the WINDOWS NT® operating system (available from Microsoft Corp., of Redmond, Washington). Among other computer system control functions, the operating system controls allocation of system resources and performs tasks such as process scheduling, memory management, networking, and I/O services. As suggested above, the operating system preferably includes an OPENGL application program interface for rendering non-trivial 3D graphical indicia on the display device 170.

[0022] FIG. 2 shows a process of applying special effects to a graphical image in accordance with a preferred embodiment of the invention. The process begins at step 200 in which a set of effects is defined in accord with the OPENGL™ format, and then stored in the non-volatile memory 152. These effects may include unique effects created by a programmer, or commonly known effects. Some commonly known effects include geometry transformations that change the shapes of images without changing the image colors, (e.g., “BUMP”, “CROP”, “DEFORM”, and “ROTATE”), “BALLOON”, and “SHINES.” Additional known effects may include “CONCENTRIC BLINDS”, “DISPLACEMENT”, “RADAR BLINDS”, “TOKYO BLINDS” and “VENETIAN BLINDS.”

[0023] In preferred embodiments, effects may be edited and/or combined with other effects to produce yet additional effects. An effect preferably is defined by first developing a prototype of the effect. Such prototype preferably includes a program written in any known programming language (e.g., the C++ programming language) utilizing the OPENGL™ API and WINDOWS NT™ API. In preferred embodiments, effects are designed to manipulate the 3D geometry of graphical images and their applied textures in OPENGL™ to obtain a desired output image result. Once satisfactorily developed, the (effect) program is configured to interface with the particular application program with which it is to be utilized.

[0024] The process then continues to step 202 in which a graphical image to be manipulated is obtained from memory. The image may be read from a file or generated from graphical software. The formatting data for the graphical image can comply with any graphical format and thus, does not necessarily comply with the OPENGL™ API. For example, the graphical image may be a video image from a live feed (i.e., having fields and requiring field rendering), a frame or plurality of frames in a motion picture (e.g., an avi file stored on disk), a bitmap file, or a graphic image file (i.e., a .gif tile). Such images may be three dimensional, two-dimensional, or one dimensional mapped representations of an image on a two-dimensional medium (i.e., the display device).

[0025] Once the image is obtained, a 3D view area is set up in accord with conventional processes (step 204) and then the graphical image is moved into texture memory on the graphics accelerator (step 206). The following OPENGL™ functions may be used to move the image into texture memory:

[0026] glTexEnvi;

[0027] glTexParameteri;

[0028] glEnable; and

[0029] glTexImage2D.

[0030] The image then is defined in a 3D geometry and the texture is applied to it to produce an affected output image (step 208). More particularly, the geometry of the image is manipulated by the application of the texture by the OPENGL™ graphical accelerator 167 to produce the affected output image in 3D. The shape of the 3D image being manipulated, its alignment relative to the view volume, and the manner of mapping the texture to the 3D geometry determine the content of the affected output image. Of course, the calculations for applying the effect to the converted graphical image are performed by the graphical accelerator 167 and not by the central processing unit 105. The OPENGL™ functions that may be utilized for step 208 include:

[0031] glBegin;

[0032] glTexCoord3d;

[0033] glVertex2d or glVertex3d;

[0034] glEnd;

[0035] glEnable; and

[0036] glDisable.

[0037] It then is determined at step 210 if effects are to be applied to additional images and combined with the output image from step 208. The positioning of the different affected 3D output images relative to one another in the view volume determines the interaction between images. The process loops to step 206 if effects are to be applied to additional images. If not, then the process moves to step 212 in which the image is displayed via a frame buffer capture operation. More particularly, the output image preferably is written to a frame buffer and consequently, displayed on the display device 170. An immediate display of this type is preferable, for example, to preview the application of an effect in an editing process. In some embodiments, however, such as those that do not require a preview, the OPENGL™ capture of the output image from the frame buffer is performed in a hidden window. More particularly, the window attributes may be set so that the output image will not be displayed on the display device 170. In other embodiments in which the OPENGL™ API is not utilized, a hidden window may not be necessary. The OPENGL functions that may be used for step 212 include:

[0038] glFlush;

[0039] glReadBuffer; and

[0040] glReadPixels.

[0041] In preferred embodiments of the invention, effects are normalized to be applied to a converted graphical image as a ratio of the size of the converted graphical image. Specifically, rather than specifying an effect with respect to a measurement of the image (e.g., inches or pixels), the effect may be specified as a percentage of the size of the image. For example, for a well known “SPHERIZE” effect, the center of the converted image first may be determined and the effect may process twenty percent of the image extending radially from such center location. Accordingly, preferred embodiments of the invention may apply effects to images having different sizes and still provide consistent output images.

[0042] Multiple effects may be applied to a single graphical image. Most graphical accelerators, however, have a maximum texture size that can be applied to an image. Accordingly, multiple textures for an input image may be mapped into an array associated with the graphical image.

[0043] Effects also may be applied to large images requiring a significant amount of memory by utilizing tiling techniques. More particularly, an input image may be divided (i.e., tiled) into two or more separate images that each are sized to be within the memory constraints of the computer system 100. Once tiled, the effects may be applied to each tile so that when the image is reassembled, the effect is applied to the entire image in a unitary manner. Only the portion of the effect that would be applied to a particular section of the overall image (i.e., within a tile) is applied to an individual tile. For example, if a swirl effect is applied to a large source image, such image may be broken up into four separate tiles (i.e., quadrants). The tiles are arrayed on a 3D geometry such that when the 3D geometry is swirled, the swirl appears to be a single swirl that was applied to the overall image. The boundaries between each of the tiles preferably are imperceptible by utilizing conventional anti-aliasing techniques.

[0044] Tiling techniques also may be utilized to reduce the resolution of a high resolution graphical image that is to be displayed on a lower resolution display device. To that end, the input graphical image may be tiled in accord with the resolution of the display device, and then reassembled for display.

[0045] As noted above, effects may be applied to a series of frames in a motion picture. For example, a “ROTATE” effect may rotate a three-dimensional graphical image in a three-dimensional space. In preferred embodiments, a starting position of the graphical image and an ending position of the graphical image (in the three-dimensional space) may be pre-set without any provision as to the positions of the image between such starting and ending position. To that end, in addition to applying the effect to the starting and ending frames, preferred embodiments of the invention calculate incremental effect changes and accordingly apply the effect to the intermediate frames in the motion picture. Assuming the graphical image is a cube, for example, preferred embodiments of the invention may determine the location of the cube in each intermediate frame of the motion picture between the starting and ending positions. Additional attributes of such intermediate frames such as, for example, shading and light reflections, also are calculated and applied to provide a higher quality display.

[0046] FIG. 3 shows a graphical user interface (“interface 300”) that may be utilized to apply effects to a graphical image in accordance with a preferred embodiment of the invention. To that end, the interface 300 includes an Input/Output page 302 for setting system parameters, and a viewing area 304 showing the effect, input image, and output image. More particularly, the Input/Output page 302 includes a plurality of fields that may be configured by a user editing an input graphical image. Each of those fields is discussed below.

[0047] A buffer field 306 indicates the buffer from which the input graphical image is to be received. Each output image also may be configured to he input for application of additional effects. The buffer field 306 includes a radio button 308 that activates the field 306. In preferred embodiments, the buffer field 306 has a default bufier that operates like a stack.

[0048] A file field 310 indicates the file from which to import the graphical image. In a manner similar to the buffer field 306, the file field 310 also includes a radio button 312 that activates the field 310. A file name may be written directly into this field 310, or a plurality of other files may be browsed via a browse button 313 for selection into this field 310.

[0049] An output field 314 indicates where the output image is to be displayed and/or stored. In particular, the output image may be displayed on the screen, or saved to either of the default buffer or a buffer selected by the user.

[0050] The interface 300 also includes a preview area 315 having three windows. Specifically, the preview area 315 includes an input window 316 for displaying the input graphical image, a parameter window 318 to preview and edit the output image (and effect) prior to saving in nonvolatile memory, and an output window 320 to display the output image that is to be saved in non-volatile memory 152. The interface 300 also includes a tool bar 322 that may be utilized to edit the effect in the parameter window 318. The tool bar 322 operates in accordance with conventional processes. The parameter window 318 also includes a timeline controller 324 that enables a user to move along a number of frames in an effect. The timeline controller 324 is configured to operate in a manner similar to controls of a tape recorder.

[0051] In an alternative embodiment, the disclosed apparatus and method for applying effects to graphical images may be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable media (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. Medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., microwave, infrared or other transmission techniques). The series of computer instructions embodies all or part of the functionality previously described herein with respect to the system. Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies. It is expected that such a computer program product may be distributed as a removable media with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web).

[0052] It should be noted that although preferred embodiments of the invention have been discussed as using the OPENGL™ format, it is contemplated that other three dimensional and two dimensional graphical formats may be utilized to apply effects to input images. For example, a graphics accelerator configured for use with the well known DIRECT3D™ graphical display API (available from Microsoft Corp.) may be used for such purposes. Accordingly, discussion of the OPENGL™ format is not intended to limit the scope of the invention.

[0053] Although various exemplary embodiments of the invention have been disclosed, it should be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the true scope of the invention. These and other obvious modifications are intended to be covered by the appended claims.