Audio-driven visual representations

[0002] The present invention relates to electronic transformation of images. More particularly, the present invention is directed to electronic transformations of images in response to time varying signals, such as audio signals.

[0003] Audio-visual entertainment is one of the most enduring forms of entertainment dating back to the days of yore when history was passed down through a combination of song and dance. Modern television and motion pictures are the progeny of the early days of song and dance, using a combination of audio information and video information to provide an entertaining experience for audiences world-wide. Traditionally, the video information and the audio information were created independent of each other. In the shadows of the motion picture industry evolved a profession in which sound-tracks were tailored for a pre-existing motion picture. In the early 1980's, this process was reversed when recording companies introduced a new marketing technique, the rock-video. In the rock-video, a short story, or collage of visual effects, would be associated with a pre-existing sound-track. In both of the aforementioned audio-video products, the audio and video information is synchronized to maximize the enjoyment of the audio-visual experience, requiring a great deal of human labor.

[0004] With the prevalence of the personal computer, audio-visual entertainment has been revolutionized with the introduction of interactive games, MPEG algorithms, and MP-3 algorithms and the like. More recently, the flexibility provided by the personal computer in creating audio-visual entertainment has been enhanced with development of computationally efficient algorithms, such as geometric transformation algorithms described in U.S. Pat. No. 5,204,944 to Wolberg et al. These algorithms have facilitated computer-generated animations that employ image transformations to enhance the enjoyment of personal computing as a portion of an interactive game or as a stand-alone application. Image transformations involve varying the visual representation of a two-dimensional (2-D) image using either 2-D or 3-D techniques. Transformations associated with 2-D images include image translation, scaling and rotation, and transformations associated with three-dimensional images include the aforementioned transformations, as well as bending, twisting and other more complicated modifications.

[0005] More recently, image transformations have been described as being desirable to synchronize with music in an automated fashion. To that end, a system was developed to generate movies in non-real-time. As a first step, a piece of music is analyzed in non-real-time by a computer program to extract a constant tempo and the energy of the associated beats within multiple frequency bands. Control signals are then triggered at these computed beat-times, and the amplitude of a control signal is proportional to the energy in the associated frequency band. These control signals vary smoothly over time and are used as inputs to a rendering program which varies the geometry of a pre-defined graphics scene. These geometry variations include changing an object's position, applying deformations to an object's surface, moving the camera viewpoint, and changing the lighting. For each frame of the animation, the rendering program generates a single image which is stored on the computer. After all of the images are generated, they are combined with the original piece of music into a movie. The final result is an animation sequence that is synchronized to the music in a smooth and visually appealing manner. The system does not, however, run in real-time, and it is not interactive.

[0006] Various programs (Winamp visualization plugins, for example) have been developed that respond to audio in real-time to deform an animation sequence. At a broad level, these programs perform the steps of: mapping different frequency ranges of the audio to different parts of the graphics scene and moving parts of the graphics scene in response to this audio. Real-time animation is achieved by a processing loop of:

[0007] 1. audio feature extraction

[0008] 2. geometry computation

[0009] 3. geometry rendering

[0010] performed individually for each animation frame.

[0011] However, if such a mapping is done directly, without any conditioning of these time-varying energy signals, the visual results are not perceptually very pleasing. Additionally, because the auditory feature extraction and geometry computation must be performed for each frame, the audio and video outputs are not synchronized.

[0012] Accordingly, improved audio-driven graphics techniques are continually being developed.

[0013] Provided is an improved method and system to drive transformations of a visual representation, in real-time, with characteristics extracted from an audio signal. In this manner, transformations of the visual representations are displayed concurrently with extraction of the audio signals characteristics, which facilitates pipeline processing and user interaction with the processing. According to one aspect of the invention, the method includes extracting characteristics of the audio signal, varying the representation in response to the characteristics, defining a modified representation; and periodically, providing a visual display of the modified representation synchronized to the audio signal, while extracting characteristics of the audio signal.

[0014] The system includes a processor, a buffer, in data communication with the processor, with the buffer holding digital data representing an audio signal, and a memory, in data communication with both the buffer and the processor, the memory storing a program to be operated on by the processor, the program including information corresponding to an object, a first process for capturing a frequency domain representation of a time segment of the audio signal, extracting characteristics therefrom, and forming a conditioned control signal based on extracted characteristics, and a second process to vary parameters associated with the object in response to the conditioned control signal while capturing an additional time segment of the audio signal, with the second process defining a modified representation having deformation magnitudes controlled by the conditioned control signal.

[0015] According to another aspect of the invention, the characteristics extracted from the audio signal yield information often correlated with the musical beat, including the beat energy, i.e., amplitude, and delay between beats. A smoothed conditioned control signal respond quickly to an increase in magnitude of the energy characteristic of the audio signal. However, if the magnitude of the energy characteristic then decreases suddenly the smoothed conditioned signal decreases slowly so that a controlled deformation appears more pleasing to a viewer. Specifically, one or more functions are stored in the aforementioned memory and remap vertices corresponding to the one or more objects in response to the magnitude and timing of the conditioned control signal, thereby varying the visual representation of the one or more objects. Characteristics other than the beats of the audio signals may be extracted, such as phase information, timbre information, frequency information and the like.

[0016] According to a further aspect, the system may include an audio system and a display. The sound is generated from a delayed audio signal, in this manner, the transformations of the objects are displayed in synchronization with the sound corresponding to the portion of the audio signal from which the characteristics were extracted to create the aforementioned transformations.

[0017] According to an additional aspect of the invention, an animation function control signal is triggered when the conditioned control signal increases rapidly in magnitude and is set equal to the output of a function generator when the conditioned control signal begins decreasing in magnitude.

[0018] According to a further aspect, if a minimum time period has not expired before the conditioned control signal again increases in magnitude a new animation control signal is not triggered.

[0019] Other features and advantages of the invention will be apparent in view of the following detailed description and appended drawings.

[0020] FIG. 1A is a schematic diagram of a basic process loop;

[0021] FIG. 1B is a schematic diagram of an improved processing loop utilized by a preferred embodiment of the invention;

[0022] FIG. 2 is a block diagram of a system for implementing the invention; and

[0023] FIG. 3 is a block diagram depicting the relationships between software blocks of a preferred embodiment; and

[0024] FIG. 4 is a graph depicting the triggering of animation function control signals.

[0025] The improved animation loop utilized in a preferred embodiment can best be explained through some simple diagrams. The basic loop is depicted in FIG. 1A. With the basic animation loop, the viewer hears the audio directly from the source. For each animation frame, the loop records a segment of audio, processes, and then displays the resulting geometry. Because the animation loop takes a finite amount of time, a delay is introduced between the audio that the user hears and the video that the user sees. For animations rendered at rates on the order of 30 frames per second, this delay results in a very noticeable loss of synchronization between the audio and video.

[0026] In order to better synchronize the audio and video, the presently described embodiment uses an improved processing loop, depicted in FIG. 1B, in which the audio is delayed. The loop first records the audio into a record buffer, inserts a delay equal to the time it takes to render a frame, and sends it to a playback buffer. It then takes the audio in the record buffer and proceeds as in FIG. 1A. By adding this delay, the audio and video are now properly synchronized.

[0027] An overview of the operation of the preferred embodiment is depicted in FIG. 2. Referring to FIG. 2, an audio data stream is selected using selector 301. Possible inputs to this selector include, but are not limited to, microphone 306 digitized by A/D converter 302, compressed audio files on disk 304 as decoded by decoder 303, and MIDI synthesizer output 305. A record buffer 300 for holding a frame's (a frame is defined as one animation frame, e.g. 4,410 samples if the frame rate is 10 Hz) worth of audio data is output to an FFT block 310 and simultaneously to playback buffer 380. The playback buffer is sent to a D/A converter 385 and speaker 390 for delayed playback of the input audio. The frequency bins output by the FFT block 310 are averaged within three bands (labeled Bass, Midrange, and Treble) and processed by a smoothing/normalizing filter block 320. The smoothed, normalized control signals, G_x[n] and S_x[n] (where x is B, M, or T, for Bass, Midrange, and Treble versions thereof) are applied to deformation modules D_n330 which then create animation function control signals C_n335 to deform objects O_n340 according to the magnitude of the animation function control signals. The deformed vertices from the objects 340 are then sent to a render block 350 which converts the vertices into scan-converted polygons which are in turn placed into frame buffer 360 before being output to the computer display 370. The specific routing of animation function control signals C_nto the objects O_nis just an example, showing that multiple deformation types can act on one object (as in O₃) and that each deformation can produce multiple deforming animation function control signals (as in outputs C₁, C₂, and C₃of deformation D₁).

[0028] In a preferred embodiment the FFT block 310, smoother blocks 320, deformation blocks 330 object blocks 340 and render block 350 are implemented as software modules.

[0029] As can be seen in FIG. 3, in a preferred embodiment, two APIs provide much of the remaining functionality: DirectSound™, and OpenGL™. DirectSound™ implements the AudioIn() and AudioOut() functions discussed below, while OpenGL™ implements the 3D graphics rendering (lighting, 3-d transformation, projection and scan-conversion). In a currently preferred embodiment, the application is single threaded, although multi-threaded embodiments have also been implemented, with audio processing in one thread and graphics processing in another.

[0030] At the beginning of each animation frame, one frame-time worth of audio samples are read in via the DirectSound Input port. These samples are then passed to the FFT-based filter bank 320. This filter bank can be set up to produce multiple bands of smoothed, AGC'd (Automatic Gain Control) control data. In a currently preferred embodiment, there are three independent bands: 0-300 Hz, 1500-2800 Hz, and 3700-6300 Hz. Smoothed versions of these signals, which tend to pass on salient transient information while suppressing low-level noise, are passed on to the deformation block. This block causes the vertices of the objects in the current scene to be perturbed according to the control information. Each object in the scene can have multiple deformations applied to it. Once all the deformations have been calculated for each object, rendering begins. These steps are illustrated in the following pseudo-code which represents the main processing loop:

[0031] The following is a discussion of the specific stages in this main animation loop.

[0032] The overall rendering procedure uses a double-buffered approach to improve perceived image quality. Double buffering is a standard technique in computer graphics, wherein the contents of a single animation frame are rendered into an off-screen buffer. Once all the objects in the frame have been rendered, a buffer swap is initiated, wherein the off-screen buffer is now selected for display, and the buffer that had been displayed now becomes the new off-screen buffer into which the subsequent frame will be rendered.

[0033] At the beginning of the loop, the number of audio samples corresponding to a single animation frame (OneFrame) at the current frame rate is computed. For example, if the audio sample rate is 44,100 Hz, and the animation frame rate is 20 Hz, OneFrame would be 2,205 samples (44,100 divided by 20). This many samples are then read into the record buffer from the AudioIn port. This read will block until enough samples are available to begin processing the next frame (i.e. OneFrame worth). Once these samples have been read in, they are immediately written to the AudioOut port (where the playback buffer 380 of FIG. 3 resides). The act of reading and subsequently writing out one frame of audio enforces one frame (in our example, 50 ms) of delay in the audio.

[0034] The analysis filter bank will then operate on the data in the record buffer, producing control information which will be used later in the loop to control camera position, lighting, object deformations, etc. A normalization procedure is applied, so that the animation responds equally well to both loud and soft passages in the input audio. This procedure is described below.

[0035] Before any objects are rendered, it is necessary to clear out the animation buffer so the new frame contains only new animation data. InitFrame() accomplishes this, along with any other tasks that need to be reset (for example, the 3D transformation matrix).

[0036] Then, the camera position is computed, based on the smoothed output from the analysis filterbank and user inputs (e.g., from a mouse). The user could reposition using mouse x and y coordinates. This position is then used in the Apply() procedure to apply a perspective transformation reflecting the new camera position. Next, the currently active lighting model is applied. Here, various parameters such as light intensity, position, and hue may be applied, as well as any shading method (e.g., flat-shading vs. Gouraud shading) that may be active. In the preferred embodiment, these parameters may be affected by the analysis filterbank.

[0037] Next, any static objects in the scene (those which are not deformed by the output of the audio analysis) are rendered into the (off-screen) frame buffer. Following this, all of the deformed objects are rendered. For each such object, three steps are performed. First the deformations (described below) are applied to the vertex set representing the object. Next, the normal vectors are computed from this deformed vertex set (in order to perform accurate lighting computations). Finally, the object is rendered into the (off-screen) frame buffer, based on the deformed vertices and normal vectors. Note that while a preferred embodiment is described wherein the vertices are deformed from some initial state in response to the audio, it is also possible to derive the vertex set directly from some parameters in the audio analysis. Such vertex sets are computed “from scratch” at each animation frame, rather than being based on some prior vertex set.

[0038] During rendering user inputs from a keyboard 400 or mouse 410 can be utilized to implement mode switching. For example, the w-key on the keyboard 400 toggles wireframe mode on and off. Other examples, and their bindings, include:

[0039] T-texture on/off;

[0040] C-color effects on/off;

[0041] S-strobe light on/off;

[0042] L-colored lights on/off;

[0043] F11-flat/smooth shading;

[0044] F1-randomize object colors;

[0045] F5-change deformation type;

[0046] keys 1-8-change scenes;

[0047] p-pause camera;

[0048] left-mouse-button-drag-camera pan left-right, up-down;

[0049] right-mouse-button-drag-camera zoom in/out.

[0050] Once all the objects have been rendered, an EndFrame() procedure is invoked, which initiates the buffer-swapping operation described above. To prevent the buffer swapping from manifesting visual artifacts, it is typically performed during the vertical blanking interval. In normal operation, the animation process will take slightly less than one video frame time, so that the wait for vertical blanking is small.

[0051] A detailed operation of the smoothing/normalizing filter block 320 utilized in a preferred embodiment will now be described.

[0052] Let the variable n index the series of animation frames, and let the variable m index the series of half-overlapping DFT's computed from the audio stream during a single animation frame n. Let this series of DFT's be represented by H_n,m[k], where k is the DFT bin number. Associated with each control signal is a frequency range (sub-band) defined by the DFT bin boundaries bmin and bmax. The first step in computing a control signal is to compute the maximum energy in decibels within the specified frequency range over a single animation frame. This is given by $\begin{array}{cc}M\left[n\right]=\underset{m}{\max }\left\{10{\log }_{10}\left(\sum _{k=b\min }^{b\max }{\left|H_{n,m}\left[k\right]\right|}^2\right)\right\}.& \left(1\right)\end{array}$

[0053] From this maximum energy signal, two types of control signals may be generated: a smooth normalized signal, S[n], and a gain-normalized signal, G[n].

[0054] To compute the smooth normalized signal, a normalized saturating function is first applied to the maximum energy signal:

[0055] {circumflex over (M)}[n]=f{M[n],SdBmin,SdBmax},  (2)

[0056] where $\begin{array}{cc}f\left(x,\min ,\max \right)=\{\begin{array}{cc}0& x<\min \\ \frac{x-\min }{\max -\min }& \min <x<\max \\ 1& x>\max \end{array},& \left(3\right)\end{array}$

[0057] An adaptive single-pole smoothing filter is then applied to produce S[n]:

[0058] S[n]=α[n]S[n−1]+(1−α[n]){circumflex over (M)}[n].  (4)

[0059] The smoothing filter pole, α[n], is given by

[0060] α[n]=pow{0.5,τ({circumflex over (M)}[n],{circumflex over (M)}[n−1], Sthresh, Sdecaymin, Sdecaymax)⁻¹},  (5)

[0061] where τ(x,y,thresh,min,max) is a half-amplitude decay-time function computed according to $\begin{array}{cc}\tau \left(x,y,\mathrm{thresh},\min ,\max \right)=\{\begin{array}{cc}\frac{x-\mathrm{thresh}}{1-\mathrm{thresh}}\left(\min -\max \right)+\max & \mathrm{if}x>y\mathrm{and}x>\mathrm{thresh}\\ \max & \mathrm{otherwise}\end{array}.& \left(6\right)\end{array}$

[0062] Computation of the gain-normalized signal is similar to that of the smooth normalized signal, with the addition of a signal-adaptive normalization factor:

[0063] G[n]=f{M[n]/N[n],GdBmin,GdBmax},  (7)

[0064] where f(x,min,max) is the same as in (3) and N[n] is a normalizing signal computed as

[0065] N[n]=β[n]N[n−1]+(1−β[n])M[n−1].  (8)

[0066] The gain normalization filter pole, β[n], is given by

[0067] β[n]=pow{0.5,τ(G[n],G[n−1], Gthresh, Gdecaymin, Gdecaymax)⁻¹},  (9)

[0068] where τ(x,y,thresh,min,max) is the same as in (6).

[0069] When computing S[n], the parameters SdBmin and SdBmax define the range of audio intensity over which S[n] responds. The adaptive smoothing filter described in (4)-(6) is designed so that when the energy of the audio signal within the specified frequency range increases rapidly, the signal S[n] also increases rapidly. If the energy then decreases rapidly, however, S_i[n] decreases more slowly from its maximum. The exact rate at which S[n] increases and decreases is controlled by the parameters Sdecaymin and Sdecaymax. With these parameters properly tuned, use of the signal S[n] to directly control parameters in the graphics scene allows the graphics to respond immediately to the music while retaining a perceptually pleasing smoothness of motion.

[0070] The computation of G[n] adaptively compresses energy fluctuations about any average signal energy into a fixed dynamic range, providing a control signal that is independent of the average signal energy. The normalizing signal N[n] behaves similarly to S[n], growing rapidly if the audio signal energy grows rapidly, but always decreasing slowly and smoothly. The effect is that G[n] normalizes quickly to sudden bursts in energy and is slower to normalize back to lower energy levels. With GdBmin, GdBmax, Gdecaymin, and Gdecaymax chosen appropriately, the local maxima of G[n] are good indicators of transient events in the audio signal. These local maxima may be used to trigger the onset of functions that animate various parameters in the graphics scene. Typically, these functions decay smoothly from 1 to 0 over some finite number of animation frames—the raised cosine is a very useful example. Oftentimes a signal S[n] is used in conjunction with the animation functions triggered from a signal G[n]. The animation function controls a pattern of motion, and the signal S[n] is used to modulate the amplitude or degree of this motion. This way, independent control over the triggering of animation events and the amplitude of animation events is achieved.

[0071] The exact manner in which G[n] triggers the animation functions in the preferred embodiment can be described in pseudo-code. The signal G[n] serves as an input to an animation function generator (residing within each deformation module) which produces K simultaneous outputs, each of which control some aspect of the graphics scene. The outputs are triggered according to the following algorithm, which is repeated before every new animation frame is rendered.

[0072] if (time since last output was triggered)<(minimum allowable time interval),

[0073] i=index of the output that was last triggered else,

[0074] FIG. 4 depicts the results of this algorithm when K=3. Outputs 1-3 are triggered at times S₁-S₃, respectively. Each output tracks G[n] until G[n] reaches a local maximum. At these times, M₁-M₃, each output begins to decay according to a raised cosine function. We also note that after output 2 begins to decay at time M_2A, G[n] increases to a value greater than output 2 before the minimum allowable time between triggering of separate animation functions is reached. As a result, output 2 again tracks G[n] until G[n] reaches a new local maximum at time M_2B, after which output 2 again starts to decay. The animation function control signals, c_nof FIG. 2, are equal to the output_jvalues.

[0075] Before being used to control some aspect of the graphics scene, the animation function control signals, c_n, are optionally multiplied by S[n]. In so doing, loud passages in the audio will manifest a large series of deformations (whose fine structure is defined by G(n)) while soft passages will manifest a small series. These modulated outputs are particularly useful for controlling bump-like deformations that appear on the surface of objects in the graphics scene. Each time an output of the animation function control signal is triggered and increases from zero, a new bump is formed. The time evolution of the bump is then controlled by the associated output of the animation function generator. If the animation function generator generates more than one animation function control signal output, then multiple instances of a given bump type can co-exist on an object surface with each bump at a different state of decay.

[0076] The use of G[n] and [Sin] to audio-drive the deformation of a shape will now be described in the context of causing a cosine-based deformation on the surface of a sphere. Each point on the sphere (or other object) is represented by an (x, y, z) coordinate in a 2-d array of points (for example, Sphere[i][j]). Each element also has a normal vector (n[i][j]) specifying the normal to the sphere (or other object) at point (i,j). Having the normal vector makes it easier to do the deformation.

[0077] The 2-d array is indexed with an (i,j) coordinate. A deformation is computed as a function of i and j. For example, the deformation amount may be determined by:

[0078] deformation (i,j)=amplitude*cos(f₁* i+f₂*j)

[0079] where amplitude specifies how much the sphere will be deformed, f₁specifies frequency in the i (increasing longitude) direction and f₂specifies frequency in the j (increasing latitude) direction.

[0080] To initialize the (undeformed) shape, each vertex is assigned x, y, and z coordinates given by the prototype shape (sphere in our example).

[0081] DefShape[i][j][x]=Sphere[i][j][x]

[0082] DefShape[i][j][y]=Sphere[i][j][y]

[0083] DefShape[i][j][z]=Sphere[i][j][z]

[0084] DefShape is then deformed in the direction of normal vector n.

[0085] This becomes three equations in code, i.e.,

[0086] DefShape[i][j][x]+=deformation[i][j]*n[i][j][x]

[0087] DefShape[i][j][y]+=deformation[i][j]*n[i][j][y]

[0088] DefShape[i][j][z]+=deformation[i][j]*n[i][j][z]

[0089] where n[i][j][x] is the x-component of the normal vector at point (i,j) (and so on, for the y- and z- components). In this way, the bump deformations are always applied in a manner which is normal to the surface. Viewing the earth as a special case of a sphere, deformations are like elevations above sea level.

[0090] Using the above-described deformation as an example, the gain normalized signal, G[n], is utilized to trigger the initiation of a bump deformation when a transient (sudden increase in energy) is detected in the audio signal. The animation function control signal, described above, is typically supplied as amplitude in the deformation equation. If S[n] is used to modulate the animation function control signal, then the rate at which amplitude increases and decreases is modulated by controlling the parameters Sdecaymin and Sdecaymax to provide a visually pleasing audio-driven visual display.

[0091] Another type of deformation which is particularly suitable for use with the invention is the crawl-type deformation. It is well-known that textures can be mapped onto the surfaces of shapes. In this case, the control signal is utilized to translate the mapping so that the image mapped appears to “crawl” over the surface of the object. Other transformations can be utilized to distort the mapped image in visually interesting ways. Each vertex on the shape's surface has a corresponding texture coordinate (u,v) which determines which pixel in the texture array will be used to color this point on the surface. By modifying these texture coordinates based on the control signals, crawling, swirling, and other visual effects can be manifested in the texture.

[0092] The invention has now been described with reference to the preferred embodiments. Alternatives and substitutions will now be apparent to persons of ordinary skill in the art. For example, means other than the FFT can be used to extract frequency dependent energy information from the audio signal. Also, particular filter embodiments have been described to condition the gain-normalized and smooth normalized conditioned control signals. Alternative filters structures and normalization strategies could be utilized within the scope of the invention. Further, deformations other than those described here are suitable for use with the invention. Additionally, user inputs may come from other sources, e.g., a joystick, and the particular mode switches and binding are only examples. Accordingly, it is not intended to limit the invention except as provided by the appended claims.