Facility for simultaneously outputting both a mixed digital audio signal and an unmixed digital audio signal multiple concurrently received streams of digital audio data

[0002] This application contains subject matter which is related to the subject matter of the following applications, each of which is assigned to the same assignee as this application and each of which is hereby incorporated herein by reference in its entirety:

[0003] “Method To Play Audio And Video Clips Through MPEG Decoders,” Ser. No. 09/046,283, filed Mar. 23, 1998; and

[0004] “System And Method For Merging Multiple Audio Streams,” Ser. No. 09/273,919, filed Mar. 22, 1999.

[0005] The present invention relates generally to digital data decoding systems, and more particularly, to a processing facility capable of receiving multiple concurrent streams of digital audio data and simultaneously outputting therefrom both a mixed digital audio signal and an unmixed digital audio signal.

[0006] Full motion video and audio displays based on digital signals have become widely available. While these displays have many advantages, they also often require a massive amount of raw digital data. Because the storage and transmission of digital video and audio signals is central to many applications, and because an uncompressed representation of a video and audio signal requires a large amount of storage, the use of digital compression techniques is vital to this advancing art.

[0007] Several international standards for the compression of digital video and audio signals have emerged over the past decade, with more currently under development. These standards apply to algorithms for the transmission and storage of compressed digital data in a variety of applications, including: video-telephony and teleconferencing; high quality digital television transmission on coaxial and fiberoptic networks; as well as broadcast terrestrially and other direct broadcast satellites; and in interactive multimedia products on CD-ROM, digital audio tape, and Winchester disk drives.

[0008] Several of these standards involve algorithms based on a common core of compression techniques, e.g., the CCITT (Consultative Committee on International Telegraphy and Telephony) Recommendation H.120, the CCITT Recommendation H.261, and the ISO/IEC MPEG-1 and MPEG-2 Standards. The MPEG algorithms have been developed by the Moving Picture Experts Group (MPEG), part of a joint technical committee of the International Standards Organization (ISO) and the International Electro-technical Commission (IEC). The MPEG Committee has been developing standards for the multiplex, compressed representation of video and associated audio signals.

[0009] Briefly summarized, the MPEG-2 standard describes an encoding method that results in substantial bandwidth reduction by a subjective lossy compression followed by a lossless compression. The encoded, compressed digital data is subsequently decompressed and decoded in an MPEG-2 compliant decoder.

[0010] Video and audio decoding in accordance with the MPEG-2 standard is described in greater detail in commonly assigned U.S. Pat. No. 5,576,765, entitled “Video Decoder”, which is hereby incorporated herein in its entirety, as well as the above-incorporated, commonly assigned pending application Ser. No. 09/046,283, entitled “Method To Play Audio And Video Clips Through MPEG Decoders”.

[0011] Audio/video decoders are typically embodied as general or special purpose processors and memory. Decoders that are used with television sets are often referred to in the industry as set-top box (STB) systems.

[0012] Stand-alone audio/video decoders currently used in STB systems generally utilize a dedicated interface to a specific transport chip and/or network interface module (NIM). As the required utility of these STB systems expands, it is becoming desirable to adopt input from various sources in addition to the base network (e.g., cable/satellite) connection.

[0013] For example, in addition to a main audio stream, such as a soundtrack for a movie, it may be desirable for the transport stream carrying the program to also contain a secondary audio stream or channel that provides, e.g., additional narrative for disabled viewers who are unable to hear subtle sounds in the original soundtrack. Alternatively, it may be desirable to allow a user to run other functions in the background and request audio queues when certain events have occurred, such as receiving mail, etc. Further, it may be desirable to allow an internet browser or other interactive application to run in the foreground, with primary audio and video programming (e.g., a movie) running in a window on the display. In such cases, it is desirable that the secondary or supplementary audio sound be mixed with the original audio stream so that both can be heard at a single time through a common set of speakers, as opposed to mixing the two streams where only one or the other is heard.

[0014] As an additional consideration, in order to accomplish decoding of multiple audio streams, the base audio decoder would conventionally be replicated for as many digital audio streams as output is desired. However, this could be costly and unnecessary in many potential audio mixing applications such as those noted above.

[0015] Therefore, in order to establish commercial advantage, there is a need for an integrated device capable of decoding and merging multiple independent digital audio streams using a single audio decode facility. The present invention is directed to meeting this need, as well as others described herein.

[0016] In set top boxes, there is an increasing need to be able to play multiple audio streams concurrently by mixing two streams together to a single composite output. As a further example, message overlays can be used when web browsing, while simultaneously watching either a program or listening to other audio content. Service providers are requiring this support for their designs.

[0017] Existing solutions have focused on combining the audio streams in the analog domain after decoding and digital to analog conversion. One technique for analog mixing is described in the above-referenced patent application entitled “System And Method For Merging Multiple Audio Streams.” However, recognized herein is a need for a facility for digitally processing multiple streams of audio data, and for simultaneously outputting of both a mixed digital audio signal and an unmixed digital audio signal, for example, for simultaneous listening and taping, respectively.

[0018] Briefly summarized, therefore, the present invention meets these needs by providing a method for processing digital audio data which includes: receiving a first stream of digital audio data and a second stream of digital audio data; outputting the first stream of digital audio data as an unmixed digital audio signal; and digitally mixing the first stream of digital audio data and the second stream of digital audio data and outputting based thereon a mixed digital audio signal, wherein the mixed digital audio signal is output concurrent with output of the unmixed digital audio signal.

[0019] In enhanced aspects, the method includes concurrently receiving into an audio decoder the first and second streams of digital audio data through dedicated real time interfaces, wherein the audio decoder decodes at least one of the first stream of digital audio data and the second stream of digital audio data. Further, if digitized at different rates, resampling of the second stream of audio data can occur based on a sample frequency of the first stream of digital audio data. Also, n additional streams of digital audio data, wherein n≧1, can be received and digitally mixed for output as the mixed digital audio signal.

[0020] Systems, articles of manufacture and program storage devices corresponding to the above-summarized processing facilities are also described and claimed herein.

[0021] Advantageously, in one embodiment the present invention provides an ability to send a second stream of digital audio data to an audio decoder for processing by its internal digital signal processor, e.g., using a second “clip mode” type interface such as that described in the above-incorporated application entitled “Method To Play Audio And Video Clips Through MPEG Decoders.” As a result, the audio decoder can simultaneously receive a stream through either its transport interface or a first clip mode interface and also through a second clip mode interface. One application might be to send pulse code modulated (PCM) uncompressed data through the second interface for resampling and weighted mixing with the audio stream being decoded and played through the primary interface. Note, however, there are no direct restrictions on the type of data that is input through the first or second interface. Any streaming content could be sent that is within the processing power of the available digital signal processor (DSP). This can be extended to other compressed data types, and even non-audio content for DSP processing (e.g., software modem) in parallel with the primary audio stream if there are enough cycles available to handle two real time tasks.

[0022] As a further advantage, an audio decoder in accordance with the present invention provides multichannel output. This might entail a front left-right pair, a rear left-right pair, and a center or surround channel. Often, audio applications only require a subset of the possible channels (simple stereo). Since the output channels are under the DSP control, in the case of mixed audio streams one set of stereo channels might be used for the mixed, or composite stream, for example, for direct playback to a user, while the other stereo channel can be used for output of the original unmixed digital audio data, for example, for recording on a VCR or other device. In this manner, message overlays can be heard on the live broadcast, but not recorded for future listening.

[0023] To restate, provided herein is an audio device, for example, for an integrated set top box design, that is able to receive multiple concurrent input streams for processing simultaneously into a single mixed digital output signal. Further, the audio device, after mixing, for example, a primary digital input stream with one or more secondary input streams, outputs both the original primary stream and the mixed output stream for concurrent storage and playback by separate devices. The input can be received through a transport and/or clip mode interface, decompression of one or more streams of data can occur, as well as resampling of the secondary stream based on a frequency of the primary stream. All of these functions can be accomplished by the digital signal processor.

[0024] Prior solutions are implemented either in the analog domain, which requires additional components and may not provide the mixing granularity or the audio output quality of the present invention, or if in the digital domain require additional component parts to implement, which may make the solution more costly to build and more complex to control. A facility in accordance with the present invention can be readily implemented by one skilled in the art within a single chip set top box controller without the addition of any external components.

[0025] Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered part of the claimed invention.

[0026] The above-described objects, advantages and features of the present invention, as well as others, will be more readily understood from the following detailed description of certain preferred embodiments of the invention, when considered in conjunction with the accompanying drawings in which:

[0027] FIG. 1 is a block diagram of a prior art audio decode system;

[0028] FIG. 2 is a block diagram of one embodiment of an audio decode system implementing a digital processing facility in accordance with the principles of the present invention;

[0029] FIG. 3 is a block diagram of an alternate embodiment of a digital decode system implementing a digital processing facility in accordance with the principles of the present invention;

[0030] FIG. 4 is a block diagram of one embodiment of a set top box system to implement a digital processing facility in accordance with the present invention, wherein multiple digital audio inputs are received for digital mixing and output as a mixed digital signal simultaneous with outputting of a primary, unmixed audio signal; and

[0031] FIG. 5 is a block diagram of one embodiment of an audio decode system employing multiple clip mode interfaces for receiving multiple streams of digital audio data for processing in accordance with the principles of the present invention.

[0032] FIG. 1 depicts one embodiment of a conventional audio decode system which employs an audio decoder 10 that receives either compressed data (for example, a stream of MPEG encoded data) or uncompressed, pulse code modulated (PCM) data. Depending on the data type, audio decoder 10 outputs either a decoded stream of data or the uncompressed PCM data to a digital-to-analog converter 12, the output of which drives one or more speakers 14.

[0033] As the integration of audio/video decoders and other components of, for example, STB systems onto common silicon progresses, new options become viable. Integration allows greater flexibility in the interconnection of functional units and access to memory, such as a common memory subsystem architecture.

[0034] The above-incorporated patent application entitled “Method To Play Audio And Video Clips Through MPEG Decoders” describes a technique for operating a digital data encoder wherein a first stream of encoded data is transmitted to the decoder, and a second stream of encoded data is stored in a memory device. One of the first and second streams of encoded data is selected, and the decoder is used to decode that selected stream of encoded data. The second stream of data could have been placed in the memory device by other devices or processes present in a set top box (STB) system. In such a case, all the decoder needs to process the data is a pointer to the data and some additional information about it, for example, its size. Since the processor has access to all memory, it can do any necessary parsing/manipulation required by the stream format. This provides a large degree of flexibility. The processor can then pass location and attributes of data to the decoder. This also minimizes data movement to/from memory, reducing bandwidth requirements.

[0035] As a further extension of the teachings of this patent application, the above-incorporated application entitled “System And Method For Merging Multiple Audio Streams” described a device capable of decoding, as needed, and merging two or more independent digital audio streams using a single base audio decoder circuit. As one example, the application describes an integrated device which employs a direct memory access (DMA)-like control interface that can be used to setup the continuous reading of input files or streams when run dynamically. As data is received in the PCM channel macro, it is buffered, formatted for a digital, serial interface as common in the industry and output using a high speed sample clock that can be provided by one of multiple sources. Since the input format of at least the secondary stream of audio data is assumed to comprise an audio PCM stream, it can be provided as a stored clip, a running PCM stream, or the result of decoding by the host controller, particularly in the case of relatively low complexity compression techniques used for voice-grade audio. The actual mixing of the signals occurs in analog format and digital-to-analog converters are needed to convert the received streams of digital data.

[0036] Described and claimed herein are further enhancements on the teachings of the above-incorporated patent applications. More particularly, as shown in FIG. 2, a system in accordance with the principles of the present invention comprises and audio processing system, which in one embodiment may be implemented within an audio decoder, generally denoted 20, for a set top box. In this embodiment, a primary stream of data is received through a first interface 21 into decode system 20 and is assumed to comprise an encoded stream of data. This encoded primary stream of data is decoded by a stream decode capability 24 of the system and the resultant decoded data is output as an unmixed digital audio signal. The stream decode capability is a function well known in the art of audio digital signal processing. For example, the capability resides with an IBM digital set top box integrated controller, part no. STB 01000.

[0037] A secondary stream of audio digital data is received through a second interface 22 and in this embodiment, undergoes resampling 26. The resampling logic 26 also receives as input a frequency control signal derived from the sample frequency of the primary stream of digital data. This redigitization function, wherein a second stream of audio digital data is resampled to a sample rate of a first stream of audio digital data, is well known in the art. For example, reference a MacMillian Publishing textbook by J. Proakis and D. Manolakis entitled Digital Signal Processing Principles, Algorithms and Applications, second edition, pp. 761-768 (1992) (ISBN 0-02-396815-x). Note that if the secondary stream of audio digital data has the same sampling rate as the primary stream of digital audio data, then no resampling is necessary.

[0038] In this embodiment, the processing system further includes host registers 27 for holding, for example, user programmable volume levels for use in outputting the unmixed digital audio signal and weighting the components of the mixed digital audio signal. These volume levels, labeled V0, V1 and V2, may be user controlled through a host processor, not shown.

[0039] In one embodiment, the audio processing system has no visibility to stream parameters (i.e., sample rate, PCM sample width, number of audio channels, etc.) defining the secondary stream of digital data to be mixed with the primary stream of data being processed. The user therefore specifies the desired mixing volume associated with the secondary stream to be mixed. The desired effect may be for the primary stream to have a higher amplitude than the secondary stream, or be equal in volume, or for the secondary stream to have a higher amplitude than the primary stream. Controls and support are preferably in place to allow and define these requirements to the audio processing system. The following are input controls which may be employed to specify to the audio digital signal processor how to mix the secondary stream with a primary stream:

[0040] MIX_VOL—mixing volume. Controls attenuation of mixed PCM data.

[0041] MIX_CH—number of channels in the secondary mixing stream (usually stereo or mono).

[0042] MIX_WD—secondary stream PCM sample width.

[0043] MIX_FS—secondary stream PCM sampling frequency.

[0044] Once the desired parameters are specified to the audio decode system, the system can accept the two separate audio streams of digital data, process them simultaneously, and present both unmixed and mixed outputs, for example, to digital-to-analog converters for driving one or more sets of speakers. The mixing function, labeled 28 in FIG. 2, thus receives as input the secondary stream of data, resampled if necessary and having a user specified volume V2, and the decoded primary stream of data, again having a specified weighting volume V1. The volume levels are compared, and possibly ramped (e.g., through host software), to obtain a desired output volume level for the mixed digital audio signal. Ramping allows for the mean average power to remain constant notwithstanding variations in the weighted volume levels of the primary and secondary audio streams. Mixing of digital audio signals is well established in the art, for example, U.S. Pat. No. 5,963,153, the entirety of which is incorporated herein by reference.

[0045] Since many of today's audio digital decoders are capable of multichannel output, i.e., more than two channel (left/right) output, it is possible to output both the mixed digital audio signal and one of the non-mixed digital audio signals simultaneously on separate channel outputs. For example, both the decoded primary stream of digital audio data and the mixed digital audio signal can be output simultaneously in accordance with the present invention. The advantage of being able to present both mixed and non-mixed output simultaneously is that the mixed audio output, for example, may be played to a display screen or television, while the non-mixed audio output is presented to a recording device, such as a video cassette recorder. Thus, message overlays can be heard on a live broadcast, but not recorded for future listening.

[0046] FIG. 3 presents an alternate embodiment of the present invention wherein a decode processing system 30 receives N streams of digital audio data for processing. A primary stream of data is received through a first interface 31, a secondary stream through a second interface 32 and an N^thstream of digital audio data through an N^thinterface 35. In this embodiment, each stream of digital audio data is assumed to be encoded and therefore first undergoes stream decoding 51, 52 . . . 55. If one or more of the secondary stream of digital audio data through the N^thstream of digital audio data is sampled at a different frequency than the primary stream of data, then resampling 62 . . . 65 occurs based on a sample frequency derived from the primary stream of data. The resultant signals are then mixed together to produce the mixed digital audio signal to be output from the processing system. Volume for each signal can be independently weighted through appropriate fields in host registers 67. Again, both an unmixed output, for example, comprising the decoded primary stream of digital audio data, and a mixed output are provided. In one embodiment, these different outputs might drive different channels of a multichannel audio system.

[0047] FIG. 4 depicts one embodiment of a set top box (STB) 100 to employ audio processing in accordance with the present invention. Since multiple input streams of digital audio data may be received, STB 100 may receive input from programs with one or more audio signals, for example, through satellite, cable or terrestrial modes, a microphone input 102, a telephone input 104 and/or internet derived audio data 106. These multiple inputs are then employed as described above to generate for concurrent output both an unmixed audio signal, for example for forwarding to a video cassette recorder (VCR) 110, and a mixed audio signal, for output, e.g., through a set of speakers or a television set 120.

[0048] FIG. 5 presents an extension of the above-incorporated patent application entitled “Method To Play Audio And Video Clips Through MPEG Decoders.” In this embodiment, the primary stream of digital audio data and the secondary stream of digital audio data are retrieved from memory as clips of data by audio decoder 200. The memory includes a primary clip mode data region 210 and a secondary clip mode data region 310 which are interfaced with audio decoder 200 through dedicated clip mode interfaces 205 & 305, respectively. Clip controls, such as described in the above-referenced application, enable retrieval of the desired clips of primary stream and secondary stream digital audio data. The primary clip controls 230 and secondary clip controls 320 can be generated by the digital signal processor within audio decoder 200. Primary clip controls 230 and secondary clip controls 320 are externally loaded into audio decoder 200 for use in retrieving the primary stream of digital audio data and the secondary stream of digital audio data from the memory described above. The primary clip controls present data to a stream decoder 220 as needed. Again, the primary stream of digital audio data is assumed to be encoded in this example. Further, resampling 330 of the secondary stream of digital audio data can occur based upon a sample frequency derived from the primary stream as described above. The resample function pre-processes the incoming secondary stream, for example, PCM audio data, to align the PCM data with the decoded output PCM data of the primary stream. This again involves creating or deleting audio samples in the incoming secondary stream and smoothing the result to eliminate potential audio artifacts. The secondary stream must contain the same number of PCM samples per second as the primary decoded PCM audio data, i.e., both must be at the same audio sample rate, in order to allow digital mixing.

[0049] In addition to being fed to an unmixed output buffer 410 the output of stream decoder 220 is also forwarded to a digital mixer 400 for combining with the output of resample logic 330.

[0050] A set of host registers will control the PCM mixing feature, much like the interface for the primary and secondary clip modes. Control register support for PCM mixing can include the following facilities: QAR2, QLR2, WAR2, WLR2, Block Valid2, Stream Busy2, and End of Clip2 bits. These facilities are similar to the controls for the primary clip mode, with the exception of the End of Clip2 bit.

[0051] QAR2—Contains the memory address of the start of the next audio clip containing data to be mixed.

[0052] QLR2—Contains the length of the next audio clip to be processed.

[0053] WAR2—Contains the current memory address of the current audio clip being played. This address will be somewhere between the beginning and the end of the address region of the current clip being processed.

[0054] WLR2—Contains the current length remaining of the current audio clip being processed by the audio decoder.

[0055] Block Valid2—Indicates if a queued clip is waiting to be processed in the QAR2 and QLR2 fields as soon as the current audio clip has completed.

[0056] Stream Busy2—Indicates a secondary audio clip is being processed by the audio decoder.

[0057] End of Clip2—Used to indicate that the secondary audio clip stream is coming to an end and the audio decoder should begin to ramp down the intensity of the secondary audio clip to avoid audio artifacts and provide a smooth transition to the primary stream playback only.

[0058] Using the above information, digital mixer 400 averages together the PCM samples of each audio stream using an appropriate weighting to create the desired mixed audio effect, which is then output to a mixed output buffer 420. Weighting of the PCM samples can be adjusted using volume values V1 & V2 which may be input, for example, by a host processor or a user. Essentially, volume levels V1 & V2 control ramping of the PCM samples relative to each other for output as the mixed digital audio signal. (V1 and V2 are user controlled parameters which determine the appropriate weight to be applied to each audio component.) The unmixed and mixed output buffers present the audio data to output digital-to-analog converters (not shown), and subsequently to the output speakers. Again, the unmixed digital audio signal is output simultaneous with the mixed digital audio signal as described above.

[0059] To summarize, provided herein is an audio device, for example, for an integrated set top box design, that is able to receive multiple concurrent input streams for processing simultaneously into a single mixed digital output signal. Further, the audio device, after mixing, for example, a primary digital input stream with one or more secondary input streams, outputs both the original primary stream and the mixed output stream for concurrent storage and playback by separate devices. The input can be received through a transport and/or clip mode interface, decompression of one or more streams of data can occur, as well as resampling of the secondary stream based on a frequency of the primary stream. All of these functions can be accomplished by the digital signal processor of, for example, a digital audio decoder.

[0060] The present invention can be included, for example, in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. This media has embodied therein, for instance, computer readable program code means for providing and facilitating the capabilities of the present invention. The articles of manufacture can be included as part of the computer system or sold separately.

[0061] Additionally, at least one program storage device readable by machine, tangibly embodying at least one program of instructions executable by the machine, to perform the capabilities of the present invention, can be provided.

[0062] The drawings depicted herein are provided by way of example. There may be variations to these diagrams or the steps (or operations) described herein without departing from the spirit of the invention. For instance, in certain cases, the steps may be performed in differing order, or steps may be added, deleted or modified. All of these variations are considered to comprise part of the present invention as recited in the appended claims.

[0063] While the invention has been described in detail herein in accordance with certain preferred embodiments thereof, many modifications and changes therein may be effected by those skilled in the art. Accordingly, it is intended by the appended claims to cover all such modifications and changes as fall within the true spirit and scope of the invention.