Method and apparatus for efficiently processing an audio and video data stream

[0002] 1. Technical Field

[0003] The present invention relates generally to an improved data processing system and in particular to a method and apparatus for processing a data stream in a data processing system. Still more particularly, the present invention relates to a method and apparatus for processing a data stream including audio and video data.

[0004] 2. Description of the Related Art

[0005] Multimedia, the presentation or transfer of information through more than one medium at any time, is a fast growing segment of the computer industry with many applications being developed, which incorporate various features of multimedia. Additionally, many businesses are using multimedia to present information to consumers. Multimedia combines different forms of media in the communication of information to a user through a data processing system, such as a personal computer. A multimedia application is an application that uses different forms of communications within a single application. For example, multimedia applications may communicate data to a user through a computer via audio and video simultaneously. Such multimedia applications are usually bit intensive, real time, and very demanding, requiring ample processing power in the data processing system. Users may access in the multimedia, for example, in the form of video games or movies on a digital video disk (DVD) or through a communications link.

[0006] With MPEG decoding, a system stream containing video and audio packets is received at an input layer. The input layer is used to read data into an internal buffer that can be processed. The data can be retrieved either by file reads or through a network client passing data. The end result is a buffer that can be processed and parsed by the system layer. The system layer will take the data in the input buffer and parse the data until it hits packet starts. MPEG data is comprised of a system stream that contains multiple packets. These packets may be audio or video data packets. Each audio or video packet is designated by a packet start and followed immediately after by packet information such as the packet type, audio or video, length of the packet and possibly PTS values. Past this descriptive information is the data that will be sent to the respective audio and video decoders. The amount of data sent is based on the packet length that is decremented as the packet information is processed and skipped over.

[0007] For many MPEG applications, data rates are very high and video decoding is where most of the processing power is required. It is desirable to efficiently process audio as well as to continue to process video to improve frame rates. In some instances, audio underruns may occur causing undesirable effects in the multimedia presentation. Therefore, it would be advantageous to have an improved method and apparatus for processing multimedia data streams.

[0008] It is one objective of the present invention to provide an improved data processing system.

[0009] It is another objective of the present invention to provide a method and apparatus for processing a data stream.

[0010] It is yet another objective of the present invention to provide a method and apparatus for processing a data stream including audio and video data.

[0011] The foregoing objectives are achieved as is now described.

[0012] The present invention provides a method and apparatus for processing a data stream including audio and video data in which high data rates and throughput is required. Thresholds are employed to control the processing of video and audio data in a data stream. Video data is decoded in response to a comparison of audio and video data to threshold values. Additionally, another threshold value is employed to control buffers storing audio data in response to an underflow of audio data.

[0013] Additionally, these thresholds may be dynamically changed in response to configurations present on different processing systems.

[0014] The above as well as additional objectives, features, and advantages of the present invention will become apparent in the following detailed written description.

[0015] The novel features believed characteristic of the invention are set forth in the appended claims. The invention itself however, as well as a preferred mode of use, further objects and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

[0016] FIG. 1A is a block diagram of a data processing system in the present invention may be implemented;

[0017] FIG. 1B is a block diagram of a audio/video adapter in which the present invention may be implemented;

[0018] FIG. 1C is a functional block diagram of a system for processing multimedia data according to the present invention;

[0019] FIG. 2 is a flowchart of a process for initializing video and audio streaming with high data rates according to the present invention;

[0020] FIG. 3 is a flowchart of a process for initializing an audio device according to the present invention;

[0021] FIG. 4 is a diagram of an input buffer according to the present invention.

[0022] FIG. 5 is a flowchart of a process for starting playback according to the present invention;

[0023] FIG. 6 is a flowchart of a decoder loop according to the present invention;

[0024] FIG. 7 is a flowchart of a process for performing audio callback according to the present invention; and

[0025] FIG. 8 is a flowchart of a process for performing an audio right into one buffer according to the present invention.

[0026] FIG. 9 is a block diagram illustrating the processing of audio packets according to the present invention.

[0027] With reference now to the figures, and in particular with reference to FIG. 1A, a block diagram of a data processing system 100 in which the present invention may be implemented is illustrated. Data processing system 100 employs a peripheral component interconnect (PCI) local bus architecture. Although the depicted example employs a PCI bus, other bus architectures such as Micro Channel and ISA may be used. Processor 102 and main memory 104 are connected to PCI local bus 106 through PCI bridge 108. PCI bridge 108 also may include an integrated memory controller and cache memory for processor 102. Additional connections to PCI local bus 106 may be made through direct component interconnection or through add-in boards. In the depicted example, local area network (LAN) adapter 110, SCSI host bus adapter 112, and expansion bus interface 114 are connected to PCI local bus 106 by direct component connection. In contrast, audio adapter 116, graphics adapter 118, and audio/video adapter (A/V) 119 are connected to PCI local bus 106 by add-in boards inserted into expansion slots. Expansion bus interface 114 provides a connection for a keyboard and mouse adapter 120, modem 122, and additional memory 124. SCSI host bus adapter 112 provides a connection for hard disk drive 126, tape drive 128, and CD-ROM 130 in the depicted example. Typical PCI local bus implementations will support three or four PCI expansion slots or add-in connectors. The depicted example includes four loads on the mother board and three expansion slots. Those of ordinary skill in the art will appreciate that the hardware in FIG. 1A may vary. For example, other peripheral devices, such as optical disk drives and the light may be used in addition to or in place of the hardware depicted in FIG. 1A. The depicted example is not meant to imply architectural limitations with respect to the present invention.

[0028] In FIG. 1B, a block diagram of a audio/video adapter is depicted from FIG. 1A according to the present invention. A/V adapter 119 has an input connected to source 150 which may be a digital storage media, such as hard disk 126 or CD-ROM 130. Alternatively, source 150 may be a data stream from a remote source received through LAN adapter 110. Parser 152 separates video data from audio data with video data being sent to video buffer 154 and audio data being sent to audio buffer 156. Video decoder 158 is employed to synchronize and decode or drop video frames to produce a video output. Audio decoder 160 is employed to decode audio to create an audio output for the multimedia presentation. The decoding performed by A/V adapter 119 may be implemented using the MPEG standard. The processes of the present invention may be implemented within A/V adapter 119.

[0029] With reference now to FIG. 1C, a functional block diagram of a system for processing multimedia data is depicted according to the present invention. In particular, system 150 is suitable for processing MPEG-1 and MPEG-2 data streams. System 150 includes an input layer 152, which is employed to read new data into a buffer 154 for processing. The amount of new data read into buffer 154 is equal to the amount of free space in buffer 154. System layer 156 parses new data into audio packets and video packets. Thereafter, audio layer 158 and video layer 160 process the packets. Specifically, audio layer 158 decodes audio packets and sends the data to the hardware for presentation to a user. Video layer 160 decodes video packets up to one frame before presenting them to the user.

[0030] With decoding of audio and video, thresholds are employed to improve performance according to the present invention. To avoid audio underruns, an audio threshold is used. On audio callbacks, if no data is available to write to the hardware, then an underrun recovery would not start until a threshold buffer count had been reached where no data was written. To obtain better frame rates, other system thresholds are employed in the system parsing layer so that when the thresholds are reached, additional parsing is stopped so that more time can be spent in decoding the video. The combination of the audio threshold and the system thresholds reduces the problem of both audio breakup from underruns and improved video frame rates. The thresholds depicted in the examples are values that can be tuned and changed based on the configuration of the data processing system so that on faster configuration, the thresholds can be increased and on slower data processing system configurations, the thresholds can be decreased. The audio callback threshold and the system threholds (for video packets and audio packets) described herein provide a balance in processing of multimedia data streams so that no significant amount of time in any given layer (input, system, audio, or video) is spent in processing multimedia data for presentation to a user.

[0031] Turning now to FIG. 2, a flowchart of a process for initializing video and audio streaming with high data rates is depicted according to the present invention. The process begins with an input in which data is read into a buffer (step 200). Thereafter, the system parses data read into the buffer into audio and video packets (step 202). The audio device is initiated if an audio stream is present (step 204) and the video device is initiated if a video stream is present (step 206). Thereafter, the video threshold for the number of packets is initialized (step 208) and the audio threshold number of packets is initialized (step 210). Thereafter, the audio threshold for the number of buffers in the hardware for processing audio data packets is initialized (step 212) with process terminating thereafter.

[0032] With reference now to FIG. 3, a flowchart of a process for initializing an audio device is depicted according to the present invention. FIG. 3 is a more detailed description of step 204 in FIG. 2. The process begins by skipping over audio data until a header is found (step 300). The process then initializes the audio device with information from the header (step 302). Then, a determination is made as to whether software audio decoding is to occur (step 304). If software audio decoding is to occur, the process then initializes internal buffers with compressed data (step 306). Then, the audio decoder is initialized (step 308) and the audio data is decoded (step 310). Next, the audio buffers are set up with data decoded audio data (step 312). With reference again to step 304, if software audio decoding is not to occur, the process proceeds directly to step 312. From step 312, the process frees consumed packets and space in the input buffer (step 314) with process terminating thereafter.

[0033] With reference to FIG. 4, a diagram of an input buffer is illustrated according to the present invention. Data from a data stream is stored in input buffer 400 and divided into packets. Input buffer 400 contains data packets 402-416. Data packets 402, 406, 408, and 410 are video packets while data packets 404, 412, 414, and 416 are audio packets. When data packets 402-416 are released, data can be read up through data packet 416 and more packets can be created from new data.

[0034] With reference now to FIG. 5, a flowchart of a process for starting playback is depicted according to the present invention. The process begins by starting a decoder loop (step 500). Thereafter, pre-rolled audio data is written to the audio hardware device (step 502). Pre-rolled data is pre-decoded data that is set up in the buffers but not yet sent in the hardware for processing. Then, the audio buffer count in the hardware device is set equal to the number of buffers written (step 504) with the process terminating thereafter.

[0035] With reference now to FIG. 6, a flowchart of a decoder loop is depicted according to the present invention. FIG. 6 is a more detailed description of step 500 in FIG. 5. The process begins by determining whether the input can read more data (step 600). If the input is not at the end of stream, additional data is present for reading. If the input has more data to read, the process then reads more data into free space in the buffer (step 602). Otherwise, the process determines whether the system layer has more data to parse (step 604). The process also proceeds to step 604 from step 602. If the system has no more data to parse, a determination is then made as to whether enough data is present to parse into additional packets (step 606). If more data is available to parse into packets, a determination is made as to whether a system end of stream marker is present (step 608). If the system is at the end of stream marker, the process then sets the system to the end of the stream (step 610) with the process proceeding to step 618 as described below. The process also proceeds directly to step 610 if insufficient data is available to parse into additional packets. With reference again to step 608, if the system is not at the end of stream marker, the process then parses one packet from the data (step 612). The process also proceeds to step 612 directly from step 604, if the system is at the end of stream.

[0036] From step 612, a determination is made as to whether the number of audio packets is greater than the audio threshold (step 614). If the number of audio packets is not greater than the audio threshold, the process returns to step 604 as described above. Otherwise, the process determines whether the number of video packets is greater than the video threshold (step 616). If the number of video packets is not greater than the threshold, the process also returns to step 604. If, however, the number of video packets is greater than the video threshold, the process then decodes a single video frame (step 618). Thereafter, the video packets are removed for the decoded frame (step 620). Next, a determination is made as to whether the number of video packets is greater than zero (step 622). If the number of video packets is greater than zero, the process then returns to step 600, otherwise, a determination is made as to whether the system is at the end of stream (step 624). If the system is at the end of stream, the process returns to step 600, otherwise, the process waits for audio packets to be consumed (step 626) with the process then exiting the playback mode (step 628) with the process then terminating.

[0037] With reference now to FIG. 7, a flowchart of a process for performing audio callback is depicted according to the present invention. The process begins by returning a buffer from the hardware (step 700). Next, an audio write to one buffer is performed (step 702). Next, a determination is made as to whether an underflow is returned from the hardware (step 704). If an underflow is not returned, the process then terminates. Otherwise, a determination is made as to whether the number of buffers in the hardware is greater than the audio callback threshold (step 706). If the number of buffers in the hardware is not greater than the audio callback threshold, the process then flushes the input and system data after pausing the decoder loop (step 708). Then, input is read into the buffer (step 710) and the system parses the input buffer (step 712). Thereafter, the process resyncs the video to the current audio (step 714). In other words, the process skips video until it matches the current audio time stamps in the buffer. Then, an audio write is performed into a buffer (step 716) with the process terminating thereafter. Different audio callback thresholds may be used depending on whether software or hardware decoding. Less buffers are needed with hardware decoding of audio data. If underflow recovery is not desired, then silent data may be copied in audio buffers and sent to hardware to prevent audio breakup instead of flushing input and system data.

[0038] With reference now to FIG. 8, a flowchart of a process for performing an audio write into one buffer is depicted according to the present invention. The flowchart is a more detailed description of step 702 in FIG. 7. The process begins by determining whether software audio decode is present (step 800). If software audio decode is present, the process then fills compressed audio data into a buffer (step 802). Next, audio packets are removed for new data in the compressed buffer (step 804). The audio packets are freed as data is copied into the internal buffer as described below with reference to FIG. 9. Thereafter, input space is freed from the removed packets (step 806) and new data is decoded (step 808).

[0039] Then, a determination is made as to whether the amount of decoded data is sufficient to fill an audio buffer (step 810). If insufficient data is present to fill the audio buffer, an underflow is returned (step 812) with the process terminating thereafter. Otherwise, the process fills the buffer with the decoded audio data (step 814). Then, the data in the buffer is written to the hardware (step 816). Next, the buffer count in the hardware is incremented (step 818) and an indication of success is returned (step 820).

[0040] With reference again to step 800, if software audio decode is not present, the process then determines whether sufficient data in the audio packets are present to fill the buffer (step 822). If insufficient data is present, the process proceeds to step 812 as described above. Otherwise, the audio buffer is filled with compressed audio data from the audio packets (step 824). Then, the audio packets are removed and input space is freed (step 826). Then, the data in the audio buffer is written to the hardware (step 828) with the process then proceeding to step 818 as previously described.

[0041] Turning to FIG. 9, a block diagram illustrating the processing of audio packets is depicted according to the present invention. Audio packets 900, 902, 904, and 906 are copied into internal buffer 908 in section 910. Section 912 in internal 908 remains unused in the depicted example. The data in section 910 is decoded with the decoded data being stored in internal buffer 914 in section 916. Section 918 in internal buffer 914 remains unused. The decoded data is then sent to the hardware for presentation in buffers 920, 922, 924, and 926.

[0042] In the depicted example, the buffers used for input are circular buffers. Thus, if an audio packet is freed from the buffer before a video packet located before the audio packet, no free space is available unless the video packet before the audio packet also is freed. Thus, video decoding is not interrupted so that video packets can be pulled out of the buffer to free input space to read in more data and to hopefully parse more audio packets for processing.

[0043] In the depicted example, hardware MPEG support has improved playback specifically in MPEG-2 decoding which is very video intensive and where each video frame is usually greater than 50K bytes of data. For hardware audio support, fewer buffers are employed during playback and in the depicted example, only seventy-five percent of the buffers are needed to be filled with data to prevent audio breakup after playback has started. In the depicted example, the threshold for audio is seventy-five percent of the existing audio buffers and the threshold for system and read in the video decoding is seventy-five percent of the packets processed with at least one audio packet available at all times.

[0044] The present invention may be implemented to process different types of MPEG data streams. For example, the processes of the present invention may be implemented in an MPEG-1 data stream defined under ISO/IEC 11172 from the International Organization for Standardization and the International Electronics Commission. MPEG-2 data streams also may be processed by the present invention and are defined under ISO/IEC 13818. Additionally, the present invention may be employed to process other types of data streams containing audio and video data.

[0045] It is important to note that while the present invention has been described in the context of a fully functioning data processing system, those of ordinary skill in the art will appreciate that the processes of the present invention are capable of being distributed in a form of a computer readable medium of instructions and a variety of forms and that the present invention applies equally regardless of the particular type of signal bearing media actually used to carry out the distribution. Examples of computer readable media include: recordable-type media such a floppy discs and CD-ROMs and transmission-type media such as digital and analog communications links.

[0046] The description of the preferred embodiment of the present invention has been presented for purposes of illustration and description, but is not limited to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. That the embodiment was chosen and described in order to best explain the principles of the invention the practical application to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.