TECHNICAL FIELD
The invention relates to software application programming interfaces (APIs) for multimedia devices in computer systems such as display controllers (video cards), sound devices, and three dimensional graphics accelerators. More specifically, it relates to support in a multimedia (API) for querying capabilities of multimedia devices and software emulators in a computer, and for exploiting these capabilities.
BACKGROUND OF THE INVENTION
When creating an application program for a computer such as a PC, the design of the user interface is a major concern to the application developer. In developing the user interface, the programmer typically has to include code to capture input from the user and provide output using the computer's peripherals. The interaction between an application and peripheral devices can vary significantly depending on the type of application and sophistication of the user interface. In a spreadsheet program, for example, the application needs to be informed when the user has updated a cell in a spreadsheet so that it can update the spreadsheet and display the proper result. Since the display changes rather infrequently and does not contain complex graphics in this example, the performance of the underlying software and hardware that controls the user interface is less critical. As such, the programmer can rely on a high level interface to graphics and input devices without being particularly concerned with the performance of the interface. The user interfaces of many of today's multimedia applications, however, are significantly more complex and have rigorous performance requirements. In multimedia games for example, the interaction between the application and peripherals is critical to achieving a highly interactive and realistic user interface. The interaction between the application can include not only reading input from a joystick and displaying images, but can also include mixing audio files to generate sound effects, rendering three dimensional animation using a graphics accelerator, decompressing and playing video, and updating the display image fast enough to depict realistic scenes.
To control a peripheral devices in this manner, the application program can attempt to control the peripheral devices directly or can perform operations through a software interface. A software interface provides access to certain services through a specified set of operations which can be invoked to request the services. For instance, an interface to sound effect services might include operations to "prepare a sound for playing," "start playing a sound," and "wait until a sound has finished playing." In response to a request for a particular service, the interface attempts to provide the service by taking steps to control the underlying hardware. In effect, the interface does what the application would have to do if it were to try to control the hardware directly. In addition to communicating with the hardware, an interface sometimes provides some resource management so that programs running in the computer can share access to the limited hardware resources.
For the vast majority of applications, application programmers rely on some form of software interface to interact with the computer's peripherals. Programmers typically rely on software interfaces to the peripherals so that they can focus on the specifics of their application rather than on the specifics of controlling a particular device. Unfortunately, many of today's software interfaces cannot provide the level of performance that multimedia applications demand.
There are a number of software products on the market today that provide interfaces between application programs and peripheral devices. These interfaces are sometimes characterized as low or high level interfaces, and device independent or dependent. A high level interface is one whose operations request big-picture strategic services, such as "start playing this sound" or "display this document." A low level interface is one whose operations request tactical services specifically, such as "tell the sound card at I/O (input/output) address 220 to point its DMA buffer to memory address 1000000" or "tell the video card to copy the 64.times.64 pixel region from a location starting at address 0000001 to a location starting at 1000000 in video memory". In general a high level interface may be easier for a programmer to use, but a low level interface may provide better performance and functionality. Ease of use comes from having fewer details to take care of, while better performance comes from taking advantage of special cases the hardware handles well. Hiding details tends to disguise special cases as well.
The terms "device independent" and "device dependent" refer to the extent to which the interface is specific to a particular piece of hardware. Device independent interfaces provide operations that are not specific to a particular brand of hardware device. Instead, the operations hide the detail of the hardware from the application and take care of these details internally. In contrast, device dependent interfaces provide operations to control specific features of a particular piece of hardware. To write an application using a device dependent interface, the application developer has to have a detailed understanding of how the specific hardware operates.
Hardware dependence is usually not favored because it is not flexible to changes in the underlying hardware and can often lead to resource contention problems. Programs written for a device dependent interface can be rendered obsolete by updates to the underlying hardware, and commonly do not work for more than one brand of peripheral. In addition, device dependent interfaces are more susceptible to resource contention problems because an application has access to its state information and can render it inoperable for other applications.
In general, high level interfaces tend to be device independent because they hide details, whereas low level interfaces tend to be device dependent because they reveal details. For instance, "play a sound" does not rely on the details of any sound card, but "tell the sound card at I/O address 220 . . . " obviously does.
While device independent, high level interfaces are generally easier to use for the reasons explained above, they typically are unable to provide the performance and functionality needed for certain types of applications. High level interfaces are often not sufficient for game applications and other multimedia applications because they are often incapable of achieving the desired performance. Games demand higher performance because they must achieve a high degree of user interactivity and visual realism. A game application typically has to collect rapidly changing user input, compute its impact on the current scene, and display the corresponding images and playback sounds with imperceptible delay.
Because they are designed for specific tasks, peripherals are usually much better at performing certain types of functions than the host processor. For example, a video card may have special purpose hardware that can copy pixels much faster than the CPU. A high level interface may not take advantage of this particular feature or may include additional layers of code that consume valuable CPU cycles and time before the operation is even started on the peripheral.
Since many peripherals have their own computing resources, such as processors and memory, performance can be improved by off-loading some tasks to the peripheral rather than consuming the resources of the host CPU. However, without a low level interface to expose these resources to the application, they are not fully exploited.
One drawback of many APIs available today is that they lack the ability to effectively expose multimedia hardware and software capabilities to the application programs. Many media interfaces provide no access to the specific capabilities of the media devices in the computer such as a display controller or sound device. As such, applications are unable to determine the capabilities of the media hardware, and optimize their behavior based on the available resources.
SUMMARY OF THE INVENTION
The invention provides methods for returning software and hardware capabilities in a multimedia API. The invention further provides methods for optimizing API function calls using available software and hardware capabilities. Software capabilities refer generally to software emulators associated with a particular media device such as a display device, a 3D graphics accelerator, or a sound device. Hardware capabilities refer to the capabilities of the actual media device.
One aspect of the invention relates to the support in a multimedia API for returning hardware and software capabilities. The multimedia API maintains the hardware and software capabilities in capabilities structures associated with a particular media device. As an optimization, the multimedia API can also maintain the intersection of the software and hardware capabilities.
Applications can request either hardware or software capabilities for a media device by invoking a function to get the capabilities for specified media device and specifying whether software, hardware or both capabilities are to be returned. In response, the function returns the requested capabilities. In one implementation, an application requests hardware capabilities by specifying a pointer to a capabilities structure to be filled with the hardware capabilities. Similarly, to get software emulator capabilities, the application specifies a pointer to a structure to be filled with the software capabilities. With knowledge of these capabilities, an application can optimize its behavior using available hardware capabilities to enhance performance.
The API includes a number of functions to access media types rendered by the respective media devices. For example, the display device API includes functions to access and manipulate surface memory holding a pixmap to be drawn on a display device, and palettes to be used to specify the entries in the display device's color table. These functions can be optimized by the API by identifying the available hardware or software capabilities needed to process a particular API request, and using the available hardware capabilities where possible to process the call.
Further features and advantages of the invention will become apparent with reference to the following detailed description and accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a general block diagram of a computer system in which an embodiment of the invention can be implemented.
FIG. 2 is a block diagram illustrating the architecture of multimedia device interfaces in which one embodiment of the invention is implemented.
FIG. 3 is a flow diagram illustrating a method for getting hardware and software capabilities.
FIG. 4 is a flow diagram illustrating a method for optimizing an API call based on available hardware and software capabilities.
FIG. 5 is a block diagram illustrating one implementation of a display device interface architecture.
FIG. 6 is a block diagram illustrating one implementation of a sound device interface architecture.
FIG. 7 is a diagram illustrating the relationship between media device objects and related objects in one implementation of multimedia APIs.
DETAILED DESCRIPTION
FIG. 1 is a general block diagram of a computer system 20 in which an embodiment of the invention can be implemented. The computer system 20 includes as its basic elements a computer 22, one or more input devices 24 and one or more output device 26. The computer system can also include a communication device 25 and an auxiliary processing device 27.
Computer 22 generally includes a central processing unit (CPU) 28 and a memory system 30 that communicate through a bus structure 32. CPU 28 includes an arithmetic logic unit (ALU) 33 for performing computations, registers 34 for temporary storage of data and instructions and a control unit 36 for controlling the operation of computer system 20 in response to instructions from a computer program such as an application or an operating system.
Memory system 30 generally includes high-speed main memory 38 in the form of a medium such as random access memory (RAM) and read only memory (ROM) semiconductor devices, and secondary storage 40 in the form of a medium such as floppy disks, hard disks, tape, CD-ROM, etc. or other devices that use optical, magnetic or other recording material. Main memory 38 stores programs such as a computer's operating system and currently running application programs. In some implementations, portions of main memory 38 may also be used for displaying images through a display device.
Input device 24 and output device 26 are typically peripheral devices connected by bus structure 32 to computer 22. Input device 24 may be a keyboard, pointing device, pen, joystick, head tracking device or other device for providing input data to the computer.
Output device 26 may be a display device, printer, sound device or other device for providing output data from the computer.
The communication device 25 can include any of a variety of peripheral devices that enable computers to communicate. For example, the communication device can include a modem or a network adapter (25).
The auxiliary processing device 27 refers generally to a peripheral with a processor for enhancing the performance of the computer. One example of an auxiliary processing device is a graphics accelerator card.
It should be understood that FIG. 1 is a block diagram illustrating the basic elements of a computer system; the figure is not intended to illustrate a specific architecture for a computer system 20. For example, no particular bus structure is shown because various bus structures known in the field of computer design may be used to interconnect the elements of the computer system in a number of ways, as desired. CPU 28 may be comprised of a discrete ALU 33, registers 34 and control unit 36 or may be a single device in which one or more of these parts of the CPU are integrated together, such as in a microprocessor. Moreover, the number and arrangement of the elements of the computer system may be varied from what is shown and described in ways known in the art.
The invention may be implemented in any of a number of well-known computer systems. For instance, the invention may be implemented in a personal computer (PC), such as IBM-AT compatible computers or computer systems based on the 80386, 80486, or Pentium processors from Intel Corporation. Alternatively, the invention may be implemented on any number of computer workstations, such as machines based on a RISC (reduced instruction set computing) architecture. The above systems serve as examples only and should not be construed as limiting the type of computer system in which the invention may be implemented.
FIG. 2 is a block diagram illustrating the architecture of multimedia device interfaces 50 in which an embodiment of the invention is implemented. This diagram illustrates relationships between application programs ("applications") 52, multimedia device interfaces 50, and multimedia device hardware 56. In FIG. 2, there are five multimedia (APIs) including a display device API 58, a 3D graphics rendering (3D) API 60, a sound device API 62, an input API 64 and a communication (COMM) device API 66. Each of the APIs can communicate with the underlying multimedia hardware through device drivers 68-78. The device drivers are device dependent software interfaces used to carry out API functions on specific hardware devices 56. While FIG. 2 illustrates the device drivers 68-78 as separate software components, they can be integrated into the respective hardware devices as well.
FIG. 2 represents the multimedia devices (56) as separate functional components of the architecture. This does not mean that multimedia devices are always implemented as separate devices connected to the host PC. Multimedia devices can be implemented in separate cards that plug into a peripheral bus of a PC, implemented on a single card, or implemented on a chip or chips within the host computer, or on an external bus. In general, the APIs can operate in a number of different hardware platforms.
In the architecture shown in FIG. 2, the display device, 3D, and sound device APIs 58-62 include software emulators or Hardware Emulation Layers (HEL) 80-84. The APIs can invoke a software emulator to carry out an API request such as "copy a block of pixels," "render a stream of primitives based on rendering instructions," or "mix the following four sound files." Software emulation is particularly useful to provide additional functionality not supported in the underlying hardware device. However, as explained further below, a software emulator function can be invoked even in cases where the underlying hardware also supports the function. The APIs can inform the applications of the available capabilities so they can optimize their performance. In addition, the API can attempt to optimize a particular API request using available hardware and software capabilities.
The display device API shown in FIG. 2 provides services relating to the display of 2D images on a display monitor. It communicates with a display controller 86 through a display driver 68.
The 3D device API provides services relating to 3D graphics rendering including transforms, lighting, and rasterizing. The 3D API includes services to perform the steps in the rendering pipeline (transformation, lighting, and rasterization) either in software, or with the 3D accelerator 88. Like the display device API, the 3D also includes a HEL 82 for emulating hardware capabilities not available in the 3D accelerator. The API communicates with the 3D accelerator, if one is present, through the 3D device driver 70.
The sound device API provides services relating to the playback of sounds on a sound device 90. It communicates with the sound device through the sound device driver 72. The sound API includes a HEL 84 capable of emulating sound functions using the host CPU rather than the sound device.
The input device API provides access to digital or analog input devices 92 such as joysticks, touch screens, digitizing tablets, light pens, rudder pedals, flight yokes, virtual-reality headgear, etc. The API communicates with the input devices through device drivers 74.
Finally, the comm APIs provides access to communication services. The comm APIs access communication devices such as a modem or network adapter through a modem driver 76 or network driver 78, respectively. In one implementation of the APIs, the comm API refers to an application interface to support multimedia applications such as games that involve interaction with another computer. For example, in a multi-player game application, two players may be playing a game on separate computers that communicate through a local area network, through a modem connection, or across the Internet. The comm API provides access to the communication protocols necessary to establish communication paths between or among each computer involved in the game.
To invoke a multimedia related operation such as play sound, render a scene or copy a block of pixels, the applications invoke functions in the APIs. In response, the APIs instruct the hardware to carry out the operation, or invoke a software emulator to carry out the function.
In the architecture shown in FIG. 2, the display, 3D, and sound APIs include a service that returns the hardware and software capabilities for the respective media type. The hardware capabilities refer to the attributes and functionality of the underlying hardware, while the software capabilities refer to the functions of the software emulators. This service enables applications to scale to the available functionality. By providing the hardware and software capabilities, an application can choose to exploit a certain function if available in the hardware, and avoid using the function if only provided in software. Also, an application can choose to use its own software emulation if a particular hardware function is not available.
To take advantage of this service, an application can invoke it at initialization to get the available capabilities. The application can then optimize itself based on the available capabilities. If the application does not invoke the service, but instead attempts to use a capability that is not available, the API will tell the application that the capability is not available by returning an error parameter.
The APIs can also optimize a particular API request even in cases where the application has not done so. To optimize an API request, the API breaks down the request into the individual capabilities needed to process the request. It then determines whether the capabilities for the request are available hardware capabilities. If so, it instructs the hardware to carry out the request and passes in the necessary parameters.
If the necessary hardware capabilities are not available to handle the API request, the API determines whether the capabilities for the request are available software capabilities. If so, the API handles the request using the available emulators. Otherwise, the API tells the application that the API request cannot be processed by returning an error in most cases.
In some circumstances, the API can process an API request using a combination of software support in the API and available hardware capabilities. For example, in the case of the sound device API, the API may use the CPU to emulate sound mixing capabilities for some sounds. In the display device API, the API can break a bit block transfer (blt) request into a series of blt requests for the hardware blt capability in the display controller. In one implementation of the 3D API, the graphics rendering pipeline comprises three stages: 1) transform primitives; 2) set the lighting; and 3) rasterize primitives. Each of these stages can either be performed using software capabilities or hardware capabilities. Thus, in response to an API request to render a stream of graphics primitives, the 3D API can render the primitives using software rendering for one stage and hardware for another stage.
Having provided an overview of the APIs, we now turn to the API service for returning hardware and software capabilities. FIG. 3 is a flow diagram illustrating how a multimedia API retrieves software and hardware capabilities in one implementation. The process illustrated in FIG. 3 is typically performed at initialization to determine available hardware and software capabilities before beginning normal execution of a multimedia application.
The process shown in FIG. 3 applies to a number of media devices including the display controller, 3D accelerator and sound device. Therefore, we are not specific to the media in this diagram. More specific examples pertaining to specific media types are provided below and in Appendix A, which is incorporated by reference.
As shown in FIG. 3, the multimedia API queries and saves the hardware capabilities and the software capabilities (100, 102). In one specific implementation, the multimedia API fills one capabilities structure with the capabilities from the hardware device, and another structure with the capabilities from the software emulators. The device driver provides the hardware capabilities while the HEL (FIG. 2) provides the software capabilities in response to a request for the hardware and software capabilities.
As an optimization, the API also finds and stores the intersection between the hardware and software capabilities to determine which functions are available in both the hardware and the software (104). This optimizes how the capabilities are used to process an API request because it simplifies the API's decision whether the hardware device or a software emulator will be used to process the request.
Knowing the hardware and software capabilities returned by the API, applications can optimize their performance based on the available resources. An application can request the API to return capabilities at initialization before tries to use a particular hardware or software resource. In addition, an application can request capabilities during normal operation to determine how to use a given resource at that time. As the application executes, the availability of both the hardware and software resources may change. Thus, an application can operate more effectively if it asks the API to return capabilities before attempting to use a given resource. This is especially true in a multitasking environment such as the Window 95 Operating System from Microsoft where the availability or resources can change frequently as other applications request them.
The API can also optimize its performance of specific API requests. For example, the display device API can determine whether to use hardware or software capabilities in response to a blt request. Transparent (alpha) blting or Z (depth) blting may be supported in both hardware and software in a given system. In response to an API call for an alpha or z blt, the API can then decide whether to execute the API call using a software or hardware blter based on the available capabilities.
As another example, the sound device API can decide whether to use hardware or software capabilities to mix sound files. For instance, the API can accelerate a wave mixing function by using the underlying hardware to perform it. The API can also decide how to allocate memory for secondary mixing buffers. If the mixing buffers on the sound device are unavailable, the API can then decide to create the secondary mixing buffers in main memory of the CPU.
FIG. 4 is a flow diagram illustrating how a media API determines how to process an API request in one implementation. In response to API request (such as blt a pixel block, mix sounds in sound buffers, render the 3D graphics scene described in an execute buffer), the API identifies the requested capabilities for the request. As an optimization, the API determines whether the requested capabilities are in the intersection of available hardware and software capabilities, which it obtained as shown in FIG. 3. If the capabilities are in the intersection, then there is no need to perform further checking. The API then packages the parameters needed to process the request and instructs the hardware device to perform it (114). In this particular implementation, therefore, the API chooses the hardware over the software capabilities when both are available.
If the requested capabilities are not in the intersection, the API next determines whether they are available in the hardware (116). If so, the API proceeds to package the parameters and instruct the hardware to perform the request.
Finally, if the request cannot be performed in the hardware, the API determines whether the HEL has the requested capabilities (118). If so, the API invokes the function or functions in the HEL to process the API request.
FIGS. 5 and 6 illustrate examples of how multimedia APIs in FIG. 2 can be implemented in the Window 95 Operating System from Microsoft, Corp. FIG. 5 illustrates an example of the display architecture while FIG. 6 illustrates an example of the sound device architecture.
In FIG. 5, applications can control the display through the Graphics Device Interface (GDI) 142 and the display device API 144. GDI provides a set of high level 2D graphics functions, which are carried out in the DIB (Device Independent Bitmap) engine 146 and in the display hardware 148 through a Windows Device Driver Interface 150. The use and operation of these graphics components in the Windows Operating System from Microsoft Corp. are well known.
The display device API 144 includes functions to provide device independent, low level operations through the device driver 152. These functions include blting, flipping and lock and unlock functions used to synchronize access to surface memory. Surface memory is a region in video memory (or possibly main memory) where an array of image data such as a pixmap, alpha buffer or z buffer is stored.
The API also includes a function to return the capabilities of the underlying display device and HEL. If the display device does not have a capability required to process an API request, the display device API can use the HEL and the resources of the host to provide the capability.
The display device API provides access to hardware capabilities through the device driver, and supports software emulation in the HEL. In this particular implementation, the HEL can utilize GDI and the DIB engine to provide software emulation for graphics operations such as blts.
In FIG. 6, applications 160 can perform sound-related functions using the Wave functions 162 in the Windows API set or in the sound device API 164. The wave functions 162 access a sound device such as the sound card 166 shown in FIG. 6 through a Windows Audio device dependent interface (DDI) 168.
The sound device API 164 provides low level device independent access to the sound card 166 through a device specific driver 170. The device API includes functions to create and manipulate sound buffers used to store digital sound data. The API also includes a function to return the capabilities of the sound device 166 and HEL 172, a sound device emulation layer. If a function cannot be performed using the sound device, the sound API uses the HEL 172 to perform the function using the host CPU and main memory.
In one embodiment, the multimedia APIs shown in FIG. 2 are implemented as objects that represents the underlying device hardware. Each of the multimedia APIs include a function to create an instance of a device object representing the underlying media hardware. For example, the display device interface includes a function to create an instance of a display device object, the sound device API includes a function to create an instance of a sound device object, the 3D API includes a function to create an instance of a 3D device object, etc.
There can be one instance of a device object for every logical media device in operation. For example, there can be one instance of a display device object for every display device in operation. A software development environment may have two monitors, one running a game using the display device interface shown in FIG. 2, and another running the development environment using an alternative display device interface such as GDI (the graphics device interface), which is part of the Windows.RTM. 95 operating system from Microsoft Corporation.
The device object in this particular architecture owns the global attributes and capabilities for the device that it represents. FIG. 7 illustrates examples of different media device objects 180, 182, 184 along with examples of attributes and capabilities that they manage. As depicted in FIG. 7, the display device object 180 owns global attributes of the display device (e.g. video card) that it represents including, for example, the display mode, video memory, overlay capabilities, available palette capabilities and pixel formats, etc. The display device object 180, in this implementation, also controls default values for the global attributes such as the color key values, color depth, resolution and the hardware's display mode.
The sound device object 182 controls the global attributes and capabilities of the sound device that it represents including the master volume, sample rate capabilities, sound mixing capabilities, available device memory, etc.
Similarly, the 3D device object 184 controls the global attributes and capabilities of the 3D accelerator that it represents. This can include, for example, the vertex formats for geometric primitives, transform and lighting capabilities, the maximum size of an execute buffer, and maximum vertex count. The execute buffer refers to a region in memory for storing rendering commands and geometric primitives to be rendered according to those commands. For some devices, the execute buffer can be located on the device, while in others, the execute buffer is located in main memory of the host.
Each of the media objects include a member function to get the hardware and software capabilities of the respective media device and software emulators. To invoke these member functions, the application specifies the media device object and also provides a pointer to capabilities structure to be filled with the capabilities. To get software capabilities, the application provides a pointer to a capabilities structure to be filled with available software capabilities, and to get hardware capabilities, the application provides another pointer to a capabilities structure to be filled with available hardware capabilities. In response to a call to get the device or software capabilities, the media device object returns a capabilities structure filled with capabilities. Examples of these member functions for display, sound, and 3D device objects are provided with a description of corresponding capabilities structures in Appendix A, which is incorporated by reference. In Appendix A, these example functions are referred to as "GetCaps."
In addition to the function for returning capabilities, each of the media device objects include member functions to create objects that provide access to and control of the respective media hardware. The media device objects create instances of objects in the context of the hardware that they represent. Thus, when invoking these functions to create instances of these objects, an application can only create instances of objects compatible with the capabilities of the media objects capabilities. A few examples will help illustrate this concept.
The display device object includes member functions to create surface, palette, and clipper objects. The relationship between palette and surface objects and the display device object that corresponds to them is illustrated in FIG. 7 by the arrow from the display device object 180 to the palette object 186 and the surface objects 188 and 190.
A surface object represents a region in memory that holds a pixmap, an alpha buffer, or a Z buffer, for example. The member functions of the surface object provides services for managing and manipulating surfaces. These services include functions to flip surfaces, attach or detach a surface from another surface, perform a bit block transfer, list surfaces attached to a given surface, attach or return a palette object attached to a surface, return a clipper object attached to the surface, etc. The surface object also includes a member function to return the capabilities of a specified instance of a palette object.
A palette object is an object that represents a color table. The palette object includes member functions to get and set entries in a palette. It also include a function to return the capabilities of specified instance of a palette object.
To create these objects, the application invokes the member function specifying the display device object and the desired capabilities of the object. If the display device object cannot support a requested capability, the call will fail.
The display device object creates surface and palette objects in the context of the display hardware that it represents. Since the display device object controls the attributes of the display device, it ensures that all instances of surface objects that it creates are compatible with the display device hardware. For example, a display device object can only create a surface in YUV format on hardware that supports YUV color space conversion.
The sound device object includes a member function to create a sound buffer object. This object represents a region in memory for storing a sound or sequence of audio samples. The primary sound buffer represents the actual audio samples output to the sound device. These samples can be a single audio stream or the result of mixing several audio streams. The audio data in a primary sound buffer is typically not accessed directly by applications; however, the primary buffer can be used for control purposes, such as setting the output volume or wave format. Secondary sound buffers represent a single output stream or sound. An application can invoke member functions of the soundbuffer object to play these buffers into the primary sound buffer. Secondary sound buffers that play concurrently are mixed into the primary buffer, which is then sent to the sound device.
The sound object includes a number of member functions to control playback (play, stop, get current play position, lock or unlock a sound buffer, etc.), manage the sound environment (get and set frequency and volume), and get information about a sound buffer (return its capabilities, return the format of the sound data, etc.). To retrieve the capabilities of a sound buffer object, an application invokes a GetCaps member function specifying the instance of the object and structure to be filled with the sound buffer's capabilities.
To create an instance of a sound buffer, the application invokes a "CreateBuffer" member function of the sound device object and specifies the capabilities of the sound buffer to be created as well as the sound device object associated with it. If the sound device object does not support the capabilities of the requested sound buffer, the call will fail.
Like the display device object, the sound device object creates sound buffers in the context of the sound device that it represents. For example as shown in FIG. 7, sound buffer objects 200, 202, 204 can have specific attributes and capabilities based on the attributes and capabilities of its sound device object. The sound buffer has attributes such as volume, a specific location in memory, and a sample rate, all provided in the context of the sound device object.
The 3D device object includes a member function to create an instance of an execute buffer. An execute buffer object represents a memory region for storing a set of 3D rendering instructions, also referred to as a "display list." A display list comprises a vertex list and an instruction stream specifying how the vertex list should be lit and rendered. The execute buffer object includes member functions to get and set data in the underlying memory region holding the execute buffer and also includes lock and unlock functions to synchronize access to the execute buffer among multiple processes or threads running in the system.
To create an instance of an execute buffer object, an application invokes a CreateExecuteBuffer member function of the 3D device object and passes in the attributes of the execute buffer in a execute buffer description structure. FIG. 7 illustrates the relationship between the 3D device object 184 and an instance of an execute buffer object 210 relating to the device object. The execute buffer is created in the context of the 3D device and therefore must have attributes compatible with the 3D device object.
The media device object obtains the capabilities of the hardware it represents from the Hardware Abstraction Layer (HAL) that represents that piece of hardware. The media device object obtains its software emulation capabilities from the Hardware Emulation Layer (HEL). There can be more than one HEL and more than one HAL. In the case of multiple HALs and HELs the capabilities provided by them can be presented through the media API either separately or as a composite depending on the media type.
The HAL obtains the capabilities of the hardware through its intimate knowledge of the hardware it was written for. HALs are predominately implemented by the manufacturers of the hardware and as a result they are capable of determining the version and capabilities of the hardware down to the fabrication facility that created them. These HALs can be implemented by anyone skilled in the art of software development. They generally require a specialized software development kit from the provider of the media device API which defines the interface between the media device API and the HAL.
The HEL knows the software emulation it provides because the author of the HEL knows what emulation algorithms were incorporated in it. HELs are predominately implemented by the provider of the media device APIs. They can be implemented by anyone skilled in the art of software development. They generally require a specialized software development kit from the provider of the media device API which defines the interface between the media device API and the HEL.
Some hardware capabilities require software augmentation. When this type of functionality is being reported through the capabilities method (GetCaps in one implementation) of the media device API, it reports the capability requiring software preparation as HW ASSIST (hardware assist). This allows a client of the media device API to know that some software preparation of the media type wMll be done before the hardware can be used to perform the acceleration.
Blter Example
An example of exploiting a display specific media device API that has this capability from an application follows. The application calls the display specific media device API and asks for the hardware capabilities of the device. The API reports that the device can do source copy blts (one for one copies) from video memory to video memory but that it can not do transparent blts (copies that don't copy one specified color) from video memory to video memory. Video memory, in this context, refers to memory that the video display card can access with its hardware accelerators. Video memory is typically a scarce resource on current display devices, but using the display device blter to move the video memory around is also typically two or three times faster than the host processor. The application can modify its behavior with regard to the capability information returned by the display specific media device API by putting all of its pixmaps (media that can be blted) that don't require transparent blting into video memory, in order of their usage priority, until the video memory is gone. If the hardware had reported that it supported transparent blting the application may have loaded pixmaps that required transparent blting into video memory, over pixmaps that don't. Transparent blting is even more costly, when done on the host than source copy blting is. For the pixmaps that do not fit in the memory that is hardware accessible, the application can ask if the display specific media device API can perform the transparent blt operation without the hardware by emulating the feature in software. If the API can provide software emulation, then the application has a choice whether or not to use the software emulation provided by the API, or use the application's own algorithms. This is a very important option for real time applications and games. Blting is just one example of display functionality that may be provided by a display device to a host PC, other examples include overlays, 3D rasterization acceleration, decompression, etc. This is just one example of display functionality that may be exposed to an application through a media device API that is capable of reporting both hardware and software capabilities it offers.
Mixing Example
Another example, of exploiting a audio specific media device API that has this capability, from an application follows. The application calls the audio specific media device API and asks for the hardware capabilities of the device. The API reports that the device can do hardware mixing for sound buffers that are created in audio memory. Audio memory in this context refers to memory that the audio device can access with its mixer. The API may also report that it can mix a maximum of sixteen sounds at one time and that they must all be of the same format. Given this information, if the application wants to tune its performance to the hardware, it would load the sounds that it uses together most often into the audio memory of the device and use the audio devices mixer (a mixer merges multiple sounds into one). Mixing is a time consuming process for the host processor and the more sounds that are mixed together the more time it consumes. The application can determine from the software capabilities whether or not sound mixing is supported and then choose whether to use the software emulation provided by the audio specific media device API or its own custom solution. Audio mixing is just one example of audio functionality that may be provided by an audio device to a host PC, other examples include 3D positional audio, effects like reverb, format conversion, decompression, etc. This is just one example of audio functionality that may be exposed to an application through a media device API that is capable of reporting both the hardware and software capabilities it offers.
Applications which use media device APIs with the ability to report hardware and software capabilities do not have to use this capability. The media device API is useable without coding specific knowledge of where (software or hardware) the function will be performed into the application. This is important, because even real time applications and game generally only tune the critical pieces of their application. The other pieces just need to work, so the ability to use the media device API without regard to the specific source of the capabilities is an important feature of this type of interface as well.
The media device API is capable of modifying the storage of any media type if the storage is not good for the software emulation being provided. An example, building on the previous example Blter Example above, of this would be the creation of a pixmap in video memory (which can be slower to access from the host processor) followed by requests for operations that are not supported in hardware by the display device. Again, looking at the Blter Example, an example of a common feature that might not be provided in hardware would be transparent blting. The display specific media device API is able to tune its own performance, in this case by moving the pixmap from video memory into host memory, so the software emulation required for the operation the application requested, can be carried out at maximum speed. Host memory, in this context, is any memory that is quickly accessible to the host processor and is not allocated to the display device. This is only one example of the optimizations that the media device API can make, because it knows specifically what the hardware can and can't do, in order to provide the application with tuned performance.
One way to implement the media device object functions described above is to implement them as a COM interface. The objects in the media device (display, sound, 3D, input and multiplayer or "player" device) objects described above and in Appendix A are implemented as COM interfaces. The member functions of the object types include the standard member function of a COM interface in addition to the specific functions described above. The design and function of a COM interface is well understood in the field and is described in several publicly available documents including "Inside OLE" by Kraig Brockschmidt, MS Press.
Having described and illustrated the principles of our invention with reference to a preferred embodiment and several alternative embodiments, it should be apparent that the invention can be modified in arrangement and detail without departing from its principles. Accordingly, we claim all modifications as may come within the scope and spirit of the following claims.
APPENDIX
Forming a part of the present specification is the following:
Appendix A
(Copyright in the appendix is maintained by Microsoft Corporation).