Process for Synchronizing Computer Video With Independent Audio

The present invention relates generally to audiovisual presentation methods and, more specifically, to a method for synchronizing computer-generated video displays with an independent audio source of unknown rate and commencement.

The market acceptance of low-cost personal computers with advanced graphics capabilities has resulted in the placement of versatile video display systems in millions of homes in the United States. Such systems typically offer a video signal generation system which includes redefinable character shapes, high resolution graphics, superimposed moveable objects, or a combination of these widely-recognized and well-understood techniques. For instance, the model VIC-20 computer manufactured by Commodore includes both redefinable character shapes and high-resolution color graphics. This model has sold over a million units in the United States and is currently available for less than $100.00. Other popular systems which include advanced graphics capabilities for home users include the Apple II family, the Radio Shack Color Computer family and the Atari family, all of which are widely available and have found broad acceptance.

These personal computers do not incorporate sound generation capabilities which can produce sounds commonly available from high-fidelity sound reproduction systems. Many of the computers identified above incorporate a tone generator with a frequency control and, in some cases, a selection of predetermined waveforms. However, these devices can only approximate the tone of a musical instrument, and fail to provide satisfactory imitation of a human voice. The ability to imitate multiple voices or instruments simultaneously is wholely absent. Some personal computers additionally include a digital-to-analog convertor which can be utilized to reproduce a digitized human voice, but the cost of random-access memory limits the vocabulary available through this method and prohibits simultaneous graphics presentations. External voice-generation modules are available at a cost which, in some cases, exceeds the cost of the personal computer itself. Thus external modules have found only limited market acceptance.

Because consumer audio equipment has found a very wide acceptance in the past several years, and because a large majority of personal computer owners also own consumer audio equipment, it has also been suggested that the audio portion of an audio visual presentation be provided from an independent audio source such as a tape or disk, and that the instigation and pace of computer-generated video be synchronized with the independent audio track. However, it is well-known that the transport speed of consumer tape and disk equipment, while constant for a given source of constant power, is not precisely the same as the standard speed for such equipment. It can be appreciated that a tape deck which operates at only two percent faster or slower than the standard speed will lead or lag behind a standard-speed video presentation by an interval in excess of two seconds after less than two minutes of operation. Thus it can be appreciated that the common technique of commencing audio presentation, and then manually commencing video presentation at the sound of a cue tone, does not provide sufficient capability for continuous synchronization of computer-generated video with an independent audio track.

Other systems have in the past presented computer-generated video as a sequence of still images, with a cue tone provided at intervals in the audio track to prompt manual advancement of the video images. This is analogous to the well-known method of operating photographic slide projectors, and incorporates similar deficiencies of continuous manual operation and lack of the illusion of animation.

Atari had provided a stereo tape drive for its model 400 and 800 personal computers. On one track of this tape, voice and other human understandable audio could be recorded in synchrony with timing pulses, software or data information on a second track. The tape drive was coupled to the computer so that the computer supplied regulated power to the recorder, and sensed the data track. The sound track was coupled through the radio-frequency modulator of the computer system so that a television receiver could receive both the computer visual presentation and the independent audio track. This system essentially permitted elimination of the human in the loop but required a purchase of the Atari recorder unit, which duplicated the function and capital commitment represented by consumer audio gear already in the hands of the computer owner in most cases.

Other distinct problems arise where the visual presentation is made through means other than a personal computer. In the field of motion picture projection, many methods have been set forth for controlling the film transport and shutter mechanism to coordinate an independent sound source with a film projector. Where the image source is a television camera system, a variety of methods have been set forth for delaying an independent audio track to match video delayed through a frame storage unit or other means for synchronization with other video sources.

In this latter category, one example is U.S. Pat. No. 4,218,705 which discloses an audio synchronization system in a television relay arrangement along a line of television stations. Each station delays the video signal by a time less than or equal to the known frame interval and sufficient to synchronize the same with its own internal video operations, and thereafter relays the delayed signal to the next station in the chain. While the delay at an individual station may be less than or equal to the known interval, adjustment of an audio delay interval to equal a multiple of the known interval nearest to the cumulative video delay provides an error in synchronization which as inperceptable to the viewer. A similar system is revealed in U.S. Pat. No. 4,313,135 in which the input video arrives in synchrony with input audio at a station in the chain. A video processing device delays the output video by a fluctuating delay interval at the station. The system compares input video and output video to determine on a continuing basis the fluctuating delay interval, and then varies the performance of an audio delay system to delay output audio by an equal interval. Thus output video and output audio remain synchronized.

U.S. Pat. No. 4,040,088 discloses a system for audio-visual presentations based on a single audio track carrying both audio and video information. While the audio information is a typical analog recording, the video information comprises a plurality of message packets, each of which has a recognizable identifier preamble, a screen address and a character shape or code recorded in binary frequency shift key form. The length and pattern of the preamble is chosen so that it exceeds the maximum likely length of any similar tone in the analog audio information. A preamble detector recognizes each incoming message packet and decouples a audio amplifier from the audio information until the message packet reception is completed. A memory mapped video graphic subsystem is updated according to each received message packet, so that information comprising text or graphics can be displayed in synchrony with accompanying audio information. In a single-channel simultaneous signal mode, the system provides a demultiplexer which separates the message packets from audio information so that the audio amplifier may remain constantly coupled and the resulting presentation is more closely synchronized. However, this system is clearly limited by the bandwidth of audio gear so that normal screen image updating is too slow for detailed animation of the kind contemplated by the present invention.

All of these prior art systems pose a serious financial disadvantage for the typical consumer who already owns both audio gear and a personal computer, because all of the prior art require the acquisition of additional hardware at substantial expense. In many instances, the prior art deals only with correction of an interval delay, while presuming that the audio signal speed is precisely synchronized with an independent video frame rate. As stated above, this presumption does not hold for consumer audio equipment which is already in the hands of personal computer users. Therefore, it can readily be appreciated that there exists a need for an alternative low-cost method of synchronizing computer-generated video displays with an independent audio track of unknown speed and commencement.

The present invention provides a low-cost process for synchronizing the commencement and rate of a computer-generated video display with an independent audio sound track. The sound track may be delivered from any well-known conventional audio source such as, for example, phonograph disks, compact disks, cassette tape or open-reel tape. It has been found that the present inventive process works acceptably well with inexpensive portable tape players available from sources such as Radio Shack, General Electric and others. While gear which exhibits lower transport speed variation is somewhat more expensive, it is nonetheless very commonly found in the hands of personal computer owners and provides excellent results according to the present inventive process. A variety of personal computers may be utilized according to the present invention, such as those mentioned above, all of which are widely available and have found broad acceptance in the market place.

According to the present inventive process, a general-purpose programmable digital computer with video output is utilized to prompt the viewer to initiate an independent audio sound track which is not under the control of the computer. The user is then prompted to strike a key of the computer keyboard when a cue tone is provided by the audio sound track. This action in response to a cue tone is repeated at least once and preferably more than once, at intervals in excess of thirty seconds. The computer measures the time interval between each keystroke and thereby determines the operating speed of the audio track, while the occurence of each stroke allows determination by the computer of the commencement of the track. The computer then updates the video presentation to provide the illusion of animation at a rate corresponding to that of the audio track beginning at a time period corresponding to an interval between a final cue tone and commencement of presentation audio. To improve the physiological response accuracy of the viewer, it is preferable to record a packet of three cue tones at intervals of one-half second, and instruct the viewer to strike a key at the third tone; the process then measures the interval between packets. Physiological response accuracy of one-sixtieth second is commonly achieved in this manner. The present method provides an acceptable synchronization over a period in excess of seven minutes for lowest-cost portable cassette tape players, and in excess of fifteen minutes for higher-cost consumer audio equipment. The present invention clearly accomplishes its aims without the need for any additional hardware or the corresponding capital outlay.

These and other objects and advantages of the present inventive process will be made more readily apparent by reference to the following detailed description thereof, together with the drawings appended hereto.

FIG. 1 is a flow chart of steps conducted according to the present inventive process.

The best mode and preferred embodiment of the present invention is illustrated in the single FIGURE, and a listing of computer software embodying the present inventive process is attached hereto. Referring to the FIGURE, the process is begun at step 50 by providing an audio recording to an audio playback mechanism such as a phonographic disk player, a compact disk player, an open-reel tape player, a cassette tape player or any independent audio playback device. The audio recording begins with a series of cue tones recorded at known intervals, and preferably packets of cue tones recorded at known intervals, each interval in excess of one second and preferably thirty seconds in length. The final cue tone is followed by a commencement interval of known length, after which audio program material is recorded. Because the term "program" is used by some to indicate computer instructions and by others to indicate audio-visual information, within this disclosure the term "program" will take the latter meaning while the term "software" will take the meaning of computer instructions.

In step 100, several items of information are provided to a general purpose programmable digital computer with video graphics capabilities. These items include the number of intervals between cue tones or packets recorded on the audio recording, the lengths of the intervals between the cue tones, the commencement interval between the final cue tone and audio program, and the time intervals I.sub.n between consecutive video events displayed by the computer, all at the transport speed at which the recording was produced. The intervals I.sub.n are those which would be utilized in presentation of video program by the computer if the audio playback device operates at precisely the standard speed at which the audio recording was recorded.

In step 150, the audio playback device is started, so that the first of the cue tones or packets on the audio recording is heard. In step 200, the user is prompted to strike a key on the typewriter-style keyboard of the computer, precisely when a cue tone is heard. If cue tone packets are used, then the user is instructed to strike a key when the final tone in each packet is played. This instructional information may be provided either by the audio program, or by the computer video program, or by both. Alternately, any type of human-controlled input could be prompted, although only the keyboard is universally supplied with personal computers at present. A specific key may be designated, or any group of keys. In step 250, the computer determines that a key has been struck by the user. This step may be implemented by either of the well-known techniques of scanning the keyboard at frequent intervals, or interrupting a program loop when a key has been struck.

In step 300, when an appropriate key has been pressed to indicate that an initial cue tone or packet has been played, the computer begins measuring a time interval. In step 350, the computer again waits for an appropriate keystroke, and in step 400 when an appropriate key is pressed the computer recognizes completion of a time interval between consecutive cue tones and records the length of that interval. If more than two cue tones or packets are placed on the audio recording, then the keystroke which indicated the end of the actual interval also indicates the beginning of a subsequent actual interval, and so steps 300 through 400 are repeated for each subsequent cue tone or packet, up to a limit indicated by the cue tone interval count provided previously in step 100. Preferably, each actual interval between cue tones or packets is measured and committed to memory by the computer separately.

Following the final repetition of step 400, the computer begins measurement of the actual commencement interval in step 500. In step 550, the computer calculates the sum of the intervals recorded in step 400 and divides that sum by the number of intervals to calculate the average actual interval length. The advantage of using the total interval from first cue tone to final cue tone is that the total error introduced by the user in striking the key early or late, as a fraction of the total interval, is minimized. In step 600, the computer calculates a ratio of the average actual interval length divided by the standard interval length previously provided in step 100. This ratio, indicated herein as R, will be less than 1 if the playback device is operating faster than the standard, and greater than 1 if the playback device is operating slower than the standard.

In step 650, the standard commencement interval is multiplied by R to calculate the actual commencement interval. In step 700, the computer waits until the actual commencement interval as calculated in step 650 is completed. In step 750, the computer initiates the first of the sequence of video events I.sub.n in the video program at the termination of the actual commencement interval calculated in step 650. Finally, in step 800, each standard interval I.sub.n is multiplied by R to calculate the actual event interval, and the events are executed by the computer at the actual event interval so calculated by repetition of steps 750 and 800 a number of times equal to the number N of video events.

Additionally, the present inventive process can evaluate the quality of the viewer's response in striking the appropriate key at the cue tones. To this end, step 100 may additionally comprise providing upper and lower response time limits, which may be expressed as ratios, and which exceed the typical playback speed variance from the standard speed at which the audio recording was recorded. Step 400 then additionally comprises calculating a ratio between the actual measured interval and the standard interval, and prompting the user for better performance if the ratio exceeds the variance limits provided in step 100. The standard audio interval provided in step 100 may then be equal to the full interval from the first cue tone to the final cue tone, and the ratio R is computed in step 600 by dividing the total actual interval by the standard interval from first cue tone to final cue tone.

The present inventive process has been implemented on a Model 64 Computer from Commodore, in conjunction with a Model 1541 Diskette Drive from the same manufacturer. This particular computer incorporates a radio frequency modulator which may operate to transmit a video signal on Channel 3 or 4 in the United States, selectable by the user. Thus any television receiver may be utilized together with the computer in order to perceive the result of the present invention.

The implementation was prepared through use of Commodore Basic, which is available on a turnkey basis in read-only memory with the Model 64, and with machine-language routines written in assembly language for the Type 6510 Central Processing Unit of the Model 64. The type 6510 uses the same instruction set as the widely-utilized and well-understood type 6502 Central Process Unit. The particulars of Basic and machine-language as implemented on the Model 64 may be better understood by referring to the publications from Commodore entitled Commodore 64 Users Guide and Commodore 64 Programmers Reference Guide.

The Commodore 64 incorporates high-resolution graphics capability, superimposed moveable objects, and redefinable character graphics. These factors may be better understood by referring to the Programmers Reference Guide noted above. The Commodore 64 also incorporates two type 6522 complex interface adapter integrated circuits. Each 6522 circuit provides a setable time-of-day clock with one tenth second resolution and alarm option, as well as a 16-bit countdown timer with a one-microsecond resolution. With the time-of-day clock with alarm or the countdown timer in use, a processor interupt may be optionally generated when the alarm time occurs or the countdown timer reaches zero. Alternatively, the system processor may repeatedly scan either function to determine if a particular value has been reached.

In normal turnkey operation of the Commodore 64, one of the countdown timers is utilized to generate an interrupt every one-sixtieth second in order to scan the keyboard and certain other asynchronous input and output functions. The interrupt service routine provided by Commodore in read-only memory of the computer additionally updates a section of random-access memory utilized as a time-of-day clock known as a "jiffy" clock. Many of these clocks may be utilized in accordance with the process of the present invention.

In situations where a particular computer model is not delivered with independent internal clocks of sufficient resolution for use according to the present invention, it is a well-known and widely recognized techique to create processor delay loops of known precise time length, based on numbered iterations multiplied by the number of system clock cycles utilized per loop iteration. At the end of each loop, a random-memory location is incremented or decremented until a predetermined value is reached. Such a timing technique is equally useful for implementation of the present invention as the hardware or interrupt-driven techniques decribed above which are available in the Commodore 64.

A listing of a computer program package implementing the present inventive system on the Commodore 64 is attached hereto following this description and preceding the claims. The first Basic program loads several data files and program files from diskette, instructs the viewer to start the audio playback device, measures actual cue tone intervals as indicated by the viewer, and reports when the viewer response exceeds limits of audio media transport speed variability. This provides training experience for the neophyte viewer. The second Basic program is then chain-loaded and executed. It also loads several data files and program files from diskette, measures actual tone intervals as indicated by the viewer, and executes a sequence of video graphics events at actual intervals calculated as described above.

However, the present description, listing, and drawings are only exemplary of the present inventive system and do not serve to limit the same. In particular, the steps of the present inventive process may be interchanged and are equivalent. Those familiar with the art will recognize that the invention may be carried out in other ways without departing from the true spirit and scope of the following claims. ##SPC1##