Continuous Media Stream Control

This invention relates to computer and network systems that render continuous media information such as audio and video information from sources such as network servers and analog tuners.

There was a time when information presented by a computer was limited to numbers and/or text. With modern personal computers, however, a wide variety of information can be presented, including graphics, audio, and video.

The number of information sources has also multiplied. Multimedia information is now available from local digital storage such as CD-ROMs, from remote providers and online services, and even from local analog sources such as television and radio receivers.

While such variety allows application programs to provide much richer content, it also creates a degree of complexity for programmers. Such complexity is a result primarily of the different ways in which application programs must interact with different computer and network elements to render information of various kinds from different sources.

It is possible to relieve application programs of some of the details of information rendering by providing APIs (application programming interfaces) that take care of such details. Even with such APIs, however, application programs must be aware of the types of information they are rendering and must be able to determine the correct APIs for rendering such information. This approach makes it difficult to add new media types to a system. In order to take advantage of a new media type, each application program has to be rewritten so that it is aware of the proper APIs to use for rendering the media type.

The invention described below enables application programs to render continuous media streams of different types and from different sources without being aware of the details of the media streams such as format or location or how it is rendered. Each media stream is represented as an object with a plurality of interfaces. The interfaces have functions or methods for adjusting the volume of a media stream, for setting display parameters for video streams, and for performing other functions with regard to the rendering of media streams.

In the embodiment described below, an application program learns of a media stream by receiving a pointer to a media-specific object representing the media stream. The application program passes this pointer to an object creation function that implements a general-media programming object, resulting in a general-media object that has a similar or identical set of interfaces regardless of the type of media stream the general-media object represents. Using this scheme, the application program is never required to be aware of the details of the media stream. Rather, the general-media programming object performs any lower-level functions necessary to control the rendering of the media stream.

FIG. 1 shows a computer network system in accordance with one embodiment of the invention.

FIG. 2 shows a client computer in accordance with the invention.

FIG. 3 is a functional block diagram of the computer shown in FIG. 2.

FIG. 4 shows a conventional representation of a programming object and its interfaces.

FIG. 5 shows a general-media programming object and preferred interfaces.

FIG. 6 shows examples of media-specific programming objects.

FIG. 7 shows the interactions of various components in accordance with the invention.

FIG. 8 shows preferred steps in accordance with the invention.

FIG. 9 shows a media manager object in accordance with the invention.

FIG. 1 shows a computer network system in accordance with one embodiment of the invention. The system includes one or more network servers 12 and a plurality of network client computers 14. The client computers preferably communicate with the servers through local-area or wide-area networks 16, and/or are connected for communication with servers 12 by on-demand or dial-up connections such as conventional modem connections. For example, a particular client computer 14 might be connected to the Internet through a modem connection to a local Internet service provider. Servers 12 in this case would typically have dedicated, high-speed connections to the Internet. As another example, a client computer 14 might be connected to a server 12--such as an online service provider like Compuserve or the Microsoft Network--by a direct, dial-up, modem connection.

The network servers 12 have access to mass storage devices 18 on which are stored a plurality of continuous media data streams. In this case, the continuous media streams consist of sequences of digital data that are intended to be supplied serially to client computers and rendered by the client computers in a form that is useful to users of the client computers. For example, a continuous media stream might consist of a sequence of audio samples which are to be converted to analog format by the client computer and rendered as audible sound. As another example, a continuous media stream might consist of a sequence of video frames that are to be rendered sequentially by the client machine. In many cases, the data streams will be compressed.

These continuous media data streams are typically stored as files on mass storage devices 18, and are referenced by conventional file names. A client can specify a particular media stream by indicating its file name in combination with a unique name or ID that has been assigned to the server having access to the mass storage device on which the file resides.

FIG. 2 shows a client computer 20 in the form of a conventional desktop IBM-compatible computer. Client computer 20 has a central processing unit (CPU) 22, a display monitor 24, a keyboard 25, and a mouse 26. The computer 20 also has a floppy disk drive 28 for interfacing with a compatible floppy memory diskette 30 (e.g., 3.5" form factor), and an optical disk drive 32 for interfacing with a compatible CD-ROM 34. The computer 20 might also include another input device in addition to, or in lieu of, the keyboard 25 and mouse 26 including such devices as a track ball, stylus, or the like.

FIG. 3 shows a functional block diagram of the computer 20, revealing components that are not shown in FIG. 2. The computer 20 has a processor 40 and computer-readable storage media 42 including volatile memory (e.g., RAM) and non-volatile memory. The various components are interconnected by an internal bus 46. Non-volatile memory can be implemented as integrated circuit chips (e.g., ROM, EEPROM), disk drive(s) (e.g., floppy, optical, hard), or a combination of both. The volatile memory will typically comprise electronic DRAM (dynamic random access memory). Display 24 is connected to bus 46 through appropriate hardware interface drivers (not shown). Additionally, input devices 25 and 26 are connected to supply data to bus 46 via appropriate I/O ports, such as serial RS-232 ports.

Computer 20 includes a network interface 47 for connection to a remote network server. As discussed above, the connection to the server can be through a local-area or wide-area network. Network interface 47 therefore comprises a conventional Ethernet network card, a modem, or another device that provides communications with a server.

The computer also includes one or more local analog tuners 48. The tuners are connected to receive analog broadcast signals from one or more sources. A tuner might be a radio tuner for receiving radio broadcasts or paging messages. Alternatively, such a tuner might be a television receiver for receiving television broadcasts that include both video and sound. An example of such a tuner is manufactured by Zenith Electronics Corp. of Glenview, Ill., as model number #ST5612.

The analog data received from the tuner is considered a continuous media stream for purposes of this description.

Computer 20 runs an operating system 50 which supports multiple applications. Operating system 50 is stored in memory 42 and executed on processor 40. The operating system is preferably a multitasking operating system which allows simultaneous execution of multiple applications, although aspects of this invention may be implemented using a single-tasking operating system. The operating system preferably employs a graphical user interface windowing environment which presents the applications or documents in specially delineated areas of the display screen called "windows." Each window has its own adjustable boundaries which enable the user to enlarge or shrink the application or document relative to the display screen. Each window can act independently, optionally including its own menu, toolbar, pointers, and other controls, as if it were a virtual display device. One preferred operating system is the Windows.RTM. operating system sold by Microsoft Corporation, such as Windows.RTM. 95 or Windows NT.TM. or other derivative versions of Windows.RTM.. However, other operating systems which provide windowing environments may be employed, such as the Macintosh Finder from Apple Corporation and the OS/2 Presentation Manager from IBM.

An application program 52 is stored in memory 42. When activated, the application program 52 runs under operating system 50 while executing on processor 40. The application program 52 can be loaded into electronic memory from the floppy diskette 30 or CD-ROM 34 or, alternatively, downloaded from network 16 via network interface 47.

In addition, a media stream controller 53 is stored in memory 42 and executed by processor 40. Media stream controller 53 allows application program 52 to create programming objects representing the different media streams. Because of this, the application program does not need to be aware of the specifics of the media streams. In fact, the application program controls the rendering of each media stream using the same procedures regardless of the format or source of the media stream.

The programming objects created by media stream controller 53 have common interface functions that can be invoked by the application program to control playback of different media streams even though such media streams are of different media types. For example, each programming object has a volume control interface having identical methods or functions that can be invoked by the application program to determine and set the rendered volume level of a particular media stream--whether the media stream is a digital audio stream or an analog television broadcast.

The invention is described within the context of a familiar desktop computer that typically includes an Intel-based or Intel-compatible microprocessor, with four to sixteen megabytes of RAM. Such a computer is capable of running a multitasking operating system with a windowing environment. However, aspects of this invention might also be employed in other forms of computing devices, such as laptop computers, hand-held computers, portable personal information managers (PIMs), and the like. In these devices, the application program may be configured to run on a single-tasking operating system which does not support a windowing environment.

In addition, the invention will be useful in non-traditional network environments other than the wide-area network environment shown. For example, the invention will be useful in the interactive TV environment, where the server will consist of a cable headend and the client will consist of a set-top box or other device performing a similar function. In many proposed interactive TV systems, the headend will be capable of supplying media streams in a variety of different formats, such as file-based digital formats and broadcast digital and analog formats. Some media streams will be supplied on demand to individual clients, while others will be broadcast on a pre-determined schedule. In this environment, the client will include both a digital network interface and an analog tuner for receiving television or other audio/visual broadcasts.

The configuration of FIGS. 1-3 includes several media sources, which are capable of supplying media streams in a variety of different formats. For example, the local non-volatile storage of computer 20 is a potential source of media streams. Audio and/or video streams are available from local hard disks, floppy disks, and CD-ROMs. Similar streams might be available from Internet sources or other network sources such as a server 12. Tuners 48 also supply media streams. In some cases, the tuner itself will render the media streams, without participation by processor 40 of computer 20. In other cases, a tuner will supply a digital data stream for rendering by processor 40. In many cases, processor 40 or application program 52 can control the channel or frequency of the tuner, to select a particular broadcast signal.

Typical file-based digital media formats include ".wav" audio files and ".avi" video files commonly used in the Windows.RTM. operating environment. "MPEG" is another file or media stream format for video information. Available media formats also include analog formats including analog radio and TV broadcast formats.

In this document, the "type" of a media stream includes both its format and its source. In the prior art, even media streams having the same data format must often be treated differently by an application program if they originate from different sources. For example, the steps for obtaining a ".wav" media stream from a local source might be quite different from obtaining an identical media stream from a remote network server.

As mentioned above, media stream controller 53 allows application program 52 to create and use programming objects to represent different media streams. Such objects are implemented in the Microsoft Windows.RTM. programming environment using features of Object Linking and Embedding (OLE), also referred to as ActiveX technology. OLE is based on a protocol referred to as the Component Object Model (COM). COM specifies how objects interact with each other using interfaces. OLE and COM have been well documented and will not be explained in detail here. For more information regarding OLE and COM, refer to "OLE 2 Programmer's Reference" and "Inside OLE 2, Second Edition," by Kraig Brockschmidt, both published by Microsoft Press of Redmond, Wash., and both of which are hereby incorporated by reference.

Objects and their interfaces are conventionally illustrated as shown in FIG. 4. A circle is used to represent each interface.

The services or functions of an object can be used by both in-process clients and by out-of-process clients. In addition, an object interface can be used by a client executing on a different computer. The COM protocol covers all of these situations. Calling a remote interface (one that is in a different address space than the calling process) requires the use of "stubs" and "proxies," and involves topics such as "marshalling" and "unmarshalling" of procedure parameters. These mechanisms are well understood and are documented in the books mentioned above. Also refer to "X/Open DCE: Remote Procedure Call," published by X/Open Company Ltd., U.K. In addition, the Microsoft Development Library CD (April 1996), available from Microsoft.RTM. of Redmond, Wash., contains a Specification entitled "Component Object Model Specification 0.9" that discusses distributed COM programming. Both of these references are incorporated by reference.

FIG. 5 shows a general-media programming object 60 implemented by media stream controller 53. Such an object is created for every desired media stream and/or source, including both analog and digital sources. It includes the following interfaces, which are common to all general-media programming objects; regardless of whether they represent analog or digital media services, and regardless of whether they represent file-based sources or tuner-type sources:

IObject. This interface has functions for initializing, activating, and deactivating the media object.

IPlay. This interface has functions for controlling playback of the media object, including functions for starting, stopping, and pausing rendering of media streams. Additionally, for media sources that have the capability, this interface also supports fast-forward and rewind operations.

IVideo. The video control interface has functions for determining and controlling the size, position, and visibility of video output generated by a video media object. This interface is not available for media streams that do not include video.

IAudio. This interface, also referred to as the volume control interface, has functions for determining and controlling the volume levels and audio channel of audio produced by the media object.

IEvents. This interface allows an application program to receive and respond to events generated by media objects.

These interfaces, and the functions implemented by the interfaces, are the same for each instance of a general-media programming object 60, regardless of the type of media source or media stream represented by the object. For instance, a general-media programming object representing a remote network source has the same interfaces and functions as a general-media programming object representing an analog broadcast source. To control any aspect of the rendering from a particular media source, an application program invokes the proper interface function of the object representing the media source. The application program follows the same procedure regardless of the type of the media source.

Underlying each general-media programming object is a media-specific object that may be created under the direction of application program 52, by some other process in computer 20, or by a remote server. The media-specific object has interfaces that allow the general-media object to locate the corresponding media source. For instance, for a remote server source, the media-specific object allows the general-media object to determine the server ID and filename of the media stream that is to be rendered. For a local tuner source, the media-specific object indicates the channel number or frequency of the subject media stream.

FIG. 6 shows exemplary media-specific objects. Each of these objects has a "Data" interface that returns an identifier that allows the object to be uniquely identified, and a DataSource interface with the functions mentioned above that can be invoked to reveal source information. Object 70 corresponds to a server file source. Object 76 corresponds to an AVI (audio-video interleaved) file source. An AVI file uses an interleaved format in which video and audio that are to be played at the same time are located at the same place in the file.

Object 82 corresponds to an analog broadcast source such as an RF (radio frequency) tuner. Object 88 corresponds to a digital broadcast source such as might be used in some foreign countries like Japan

For the first two types of media-specific objects (server file source and AVI source), the DataSource interface has functions that return the server ID and filename for a particular data file. For the second two types of media-specific objects (analog and digital broadcast sources), the DataSource interface has functions that simply return a frequency or channel number for a broadcast media stream.

FIG. 7 shows how the various objects and components interact with each other. Application program 52 is shown at the top of the diagram. It interacts with one or more general-media objects 60, which in turn interact with media-specific objects 62. Objects 60 interact with the various media sources 64, which pass data or signals to appropriate rendering devices 66. FIG. 7 illustrates rendering audio and video from a remote network server, from a local file source, from an analog tuner source, and from a digitally-broadcast audio/visual source.

In the present embodiment, an application program follows the procedure of FIG. 8 to set up a general-media programming object. First, the application program performs a step 100 of receiving a source designation that specifies a media stream. The source designation indicates a filename, a server ID and filename, a channel number, or a frequency, depending on the type of media source from which the media stream will be supplied. In this embodiment of the invention, a source designation is literally a reference or pointer to an interface of a media-specific object. The media-specific object, which may be located locally or remotely (using OLE and COM), has one or more functions that can be invoked to reveal the source information (filename, server ID, channel number, frequency, etc.).

It should be noted that the application program itself will often not be concerned with the details of the media-specific objects. Rather, the application program will simply receive source designations in the form of references or pointers to media-specific objects. In some cases, the application program may be responsible for creating a media-specific object. In the preferred embodiment, APIs are provided for this purpose, for different types of media formats and sources. The application program can call such an API while providing required information about a desired media stream, and the API will return a pointer to a media-specific object representing the media stream.

A step 102, performed by an application program, comprises instantiating a media manager object. An exemplary media manager object 104 is shown in FIG. 9. In includes a "Manager" interface 106. The Manager interface is the central mechanism for accessing any type of media object. General-media objects are created using this interface.

A step 110 comprises passing the received source designation to an object creation function. The object creation function is implemented as a method of the Manager interface 106 of media manager object 104. The object creation function accepts a pointer to a media-specific object, and utilizes methods of that object to determine the source of a desired continuous media stream. Based on the source designation, the object creation function creates a general-media object such as shown in FIG. 5, representing a media source. The same object creation function is invoked by the application program regardless of the type of media source represented by any particular media-specific object. Regardless of the media type, a pointer to the media-specific object is passed to the object creation function, and the object creation function creates a general-media programming object having interfaces as shown in FIG. 5 for controlling various aspects of multimedia playback. The general-media programming object is responsible for dealing with the underlying specifics of media sources and rendering hardware, or it can utilize lower-level software components for handling these responsibilities. The application program simply receives a pointer to the general-media programming object from the object creation function.

Step 112 comprises invoking the interfaces of the general-media programming objects to control media rendering from the media sources represented by the programming objects. At this point, controlling these objects is totally independent of the media type being manipulated. Changing the volume of a file-based media stream is no different than adjusting the volume of an analog broadcast, even though different hardware may be manipulated at a lower level.

As a specific example, suppose that an application program will oversee rendering of two different media streams--one from a server- and file-based digital media source and another from a local analog media source such as a tuner. In this case, the application would first either create or receive media-specific objects representing the two media streams--one for the digital data stream and one for the analog broadcast stream. It would then pass pointers to these objects to a general-media object creation function. The object creation function would then create a general-media programming object for each of the two media streams. The general-media programming objects would have identical interfaces, and the application program would invoke them as necessary to determine or set parameters relating to rendering of the media streams.

Although the invention has been described primarily in terms of its computer hardware and methodology, the invention also includes computer readable storage media, such as electronic memory and non-volatile optical- or magnetic-based removable mass storage components, containing instructions for carrying out the steps described above.

Furthermore, it is to be understood that the invention is not limited to the specific features described, since the means herein disclosed comprise preferred forms of putting the invention into effect. The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended claims appropriately interpreted in accordance with the doctrine of equivalents.