Visual programming interface for a three-dimensional animation system for defining real time shaders using a real-time rendering engine application programming interface

[0001] Shaders are programs for three-dimensional graphics applications that define the color at each pixel on a surface. Shaders thus define surface and lighting effects for a shape. Shaders can be developed by combining multiple operations using a technique called a shade tree, such as described in “Shade Trees” by Robert L. Cook, in Computer Graphics, Vol. 18, No. 3, July 1984, pp. 223–231. The actual computation time required to render a surface to which a shader has been applied depends on a number of factors.

[0002] Having a shader operate in real-time depends on the capabilities of the computation hardware used to perform the computations. Currently, several hardware-based real-time rendering engines are available that allow a shader to be defined using operations from an application programming interface (API) for the real-time rendering engine. Typically, a computer programmer writes a computer program that defines a shader using this API to provide commands to the hardware.

[0003] It is currently time consuming to develop custom shaders for surface and lighting effects for real time environments because such development requires knowledge of low level APIs and programming skills. Once developed by a programmer, custom shaders are incorporated by an animator or artist in a three-dimensional animation system to develop content for a target platform. Additional programming is required by the programmer if the artist wants to modify a custom shader in some manner other than by modifying specific parameters defined by the programmer. This process is time consuming, costly, and limits the ability of the artist to create different effects. Also, experimentation with different shaders in this context takes a long time and is thus prohibitive.

[0004] A visual programming interface allows an artist to create real time shaders using a tree of shader nodes. Each shader node represents an operation that can be performed in real time through the real time rendering engine. The visual interface allows the arbitrary combinations of these shader nodes to be made and allows the parameters of the shaders to be manipulated or animated. The visual programming interface may be activated in an interactive animation environment through a designation that a real time shader is to be applied to a surface. By integrating the visual programming interface with an interactive animation environment, an artist can experiment readily with different custom real time shaders. An artist also has the flexibility to create arbitrary real time shader trees and to view them interactively without requiring a programmer to develop or modify a custom shader.

[0005] The shader nodes represent basic operations of the application programming interface for a real time rendering engine, combinations of such operations or a reference to an image. Example basic operations represented by such shader nodes include drawing to a frame buffer (which may include a blending operation with the contents of the frame buffer), a transform operation and lighting operations. A tree of shader nodes may be processed in multiple passes. Each drawing operation in the tree defines a separate pass. The result of each pass is blended with the results of prior passes according to parameters defined for the drawing operation.

[0009] Referring to FIG. 1, a block diagram of an example system that provides a visual programming interface for creating custom real-time shaders will now be described. This system includes a three-dimensional model generation and animation system 10. An example of such a system is the Softimage|XSI animation system from Softimage, Inc., of Montreal, Canada. In such a system an artist may define one or more three-dimensional models 12, and animation of such three dimensional models. The system allows the artist to associate a shader with each surface of a model, for example through a menu associated with the surface. In particular, an animation system generally allows an animator to associate a texture or material with the surface of an object. One material type that may be selected is a “real time shader” type, which can be defined using a visual programming interface 14. The visual programming interface 14 allows the artist to define the real time shader 16 that is associated with the model and animation 12. The real time shader and the model and animation are provided to a real-time rendering engine 18. This real time engine generates rendered images of the model or animation according to the specified real time custom shader. When used in an interactive animation environment, such as described in U.S. Pat. No. 5,977,982, which is hereby incorporated by reference, an artist is provided with an increased capability to experiment with a variety of surface and lighting effects.

[0010] The visual programming interface 14 provides the artist with access to basic shader elements 20 that are defined to correspond to commands in the application programming interface of the real-time rendering engine 18. A real time shader that has been defined using the basic shader elements 20 also may be accessible through the visual programming interface and may be used as an element of a another real time shader. An image clip also may be provided as a shader element 20.

[0011] The real-time hardware engine will now be described in more detail. A real-time rendering engine typically is provided as part of a computer platform on which the three-dimensional authoring application is executed, and typically includes a graphics card that includes a graphics processor with a specified application programming interface. Commands provided through this interface are interpreted and optimized by the hardware to operate in real time. Examples of commercially available real-time hardware rendering engines include the Microsoft XBOX, Sony Playstation, and Nintendo GameCube game platforms and other hardware platforms that support Direct3D, OpenGL and other graphics APIs. Such a platform typically has a frame buffer that stores image data and performs image operations on the data in the frame buffer.

[0012] The visual programming interface for creating real-time shaders that use the real-time hardware engine will now be described in more detail, with reference to FIG. 2. In the visual programming interface, a user selects shader elements that have an output node and may have an input node. Predefined shader elements or previously defined real time shaders may be accessed through a “library” or other similar mechanism through the user interface. Each shader element that is selected is represented graphically on a computer display, for example as a rectangle, as shown at 22, 24, 26 and 28 in FIG. 2. A connection between an output (e.g., 30) of one shader element and an input (e.g., 32) of another shader element is represented graphically by a line (e.g., 34). The connections represent data flow from the output of a process performed by a shader element to an input of another shader element. Real time shader elements thus are combined in a manner similar to an “Effect Tree” that is used in compositing applications to define a combination of image operations. The properties of each of the real time shader elements may be edited in a property editor for the real time shader tree. An effect tree and property editor is described in copending U.S. patent application Ser. No. 09/369,688, entitled “Editor for Effect Tree Parameters,” filed Aug. 6, 1999, which is hereby incorporated by reference. The implementation of the effect tree and its property editor may be applied to create this visual programming interface for real time shaders.

[0013] An example user interface is shown in FIG. 3. A visual programming interface is shown at 300, in which a shader tree 302 is connected to a real time input 304 of a material node 306. Properties of a shader that is selected, such as shader 308, may be edited using the property editor 310, which is similar to the property editor described above. A viewer window 312 displays the result of the rendering of the shader tree.

[0014] The basic real time shader elements from which all other real time shaders are created are defined to correspond to functions that can be performed in real time by the real time rendering engine through its application programming interface. Each of the basic real time shader elements has a number of properties that can be defined by the end user using the property editor, and has an associated function that it performs using those properties.

[0015] Each real time shader element should have access to data defining a rendering context (such as an OpenGL context), data defining the surface (typically a set of vertices, triangles, colors, uv data, normals, materials and textures, and a property page. Given this information, each shader can set render states (vertex/pixel shader, materials, textures, alpha blending, lighting, antialiasing, mip mapping and other state information), change the final geometry by modifying uv data, vertices, colors and normals, and override the drawing code.

[0016] In one embodiment, where the real time hardware engine supports the OpenGL API, example types of real time shader elements are the following:

[0017] Draw. A draw shader corresponds to the OGLDraw command and defines and executes a single OGL rendering pass by determining the OGL state, and then drawing. The draw shader is essential to most realtime shader trees because it draws the result of the chain of shaders leading into it. It is the only core realtime shader that can be connected directly to the material node's real time input. Thus, for example, the shader element 28 of FIG. 2 would be a draw shader that connects to the real time input of a material node.

[0018] Shade. A shade shader corresponds to the OGLShade command and sets lighting characteristics of an object. It defines the ambient, diffuse, specular, and emissive RGBA values of an object, as well as the size of the specular highlight of the object.

[0019] Single Texture. A single texture shader corresponds to the OGLT2D commmand and defines the image source, projection method and other attributes of a single texture image. This shader may be used to identify an image as a relection map, rather than as a directly applied texture.

[0020] Texture Transform. A texture transform shader corresponds to the OGLTCrans command and applies a variety of transformations to input textures. Transformations are animatable, either by keying or by enabling one of several modulation modes, which are are also animatable. The modulation options animate the basic transformation according to one of several available waveforms.

[0021] Combined. A combined shader corresponds to the OGLCom command and is a draw shader, a shade shader, and a single texture shader in a single package. The combined shader is useful in cases where you would otherwise use a single instance of each shader to create the desired effect. Otherwise, using the combined shader can create unnecessary calculations.

[0022] Multiple Texture. A multiple texture shader corresponds to the OGLMulti command and combines four single texture shaders in one property editor. The basic attributes of each of the single texture shaders may be set and these shaders may be blended together using modulation functions. A simpler set of blending functions may be provided for the multiple texture shader. Full control over blending of multiple single texture shaders can be provided by using separate single texture shaders.

[0023] An image clip node also is provided. The image clip merely represents an image and may be attached to an input of any of these shaders.

[0024] More complex real time shaders may be specified by creating a shader tree using these basic shader elements. Each real time shader tree ends with a draw shader having its output connected to the real time input of the material node associated with the surface of an object.

[0025] Using OpenGL, a real time shader is rendered using a multipass process that uses the OpenGL state and state change to build an effect that is drawn onto an object. The real time shader is applied during the traversal of the geometry of the object during rendering by the real time rendering engine. For example, when the OpenGL display is about to set material and texture bindings for an object, then it also checks for the existence of a real time shader applied to the realtime input of the material node associated with the surface of the object. If such a real time shader is present, the render function is executed.

[0026] In a realtime shader tree, the shader nodes are executed sequentially starting with the node farthest from the material node, and ending with the one that is closest. Each draw node in the sequence reads the OGL state and draws a single layer, or pass, based on that state. The draw node connected to the material node draws the cumulative result of the entire shader tree onto the object. Successive passes are drawn one on top of the other, and blended together using OGL alpha blending in a frame buffer to create an overall effect, like compositing. After a pass is drawn, the results become the background that will be blended with the results of the next pass, which become the foreground. The properties of the draw shader specify the blending modes, indicating how any foreground layer (the input to the draw shader) is applied to a background layer (the contents of the frame buffer).

[0027] By integrating such a visual programming interface that exposes the API of a real-time hardware rendering engine to allow an artist to create custom real-time shaders with an interactive rendering environment in a three-dimensional authoring application, the ability of an artist to experiment with different effects is significantly increased.

[0028] Having now described an example embodiment, it should be apparent to those skilled in the art that the foregoing is merely illustrative and not limiting, having been presented by way of example only. Numerous modifications and other embodiments are within the scope of one of ordinary skill in the art and are contemplated as falling within the scope of the invention.