Complex-Valued Phase-Based Eulerian Motion Modulation

[0001] This application is a continuation-in-part of U.S. application Ser. No. 13/607,173, filed Sep. 7, 2012.

[0002] Phase-Based Optical Flow.

[0003] Phase-based optical flow can track constant phase contours by computing the phase gradient of a spatio-temporally bandpassed video. This can provide an approximation to the motion field, and it was shown that phase is more robust than amplitude in detecting image changes due to contrast and scale.

[0004] Complex Steerable Pyramids.

[0005] Steerable pyramids are over complete representations of images that decompose the images along scale and orientation. The basis functions are similar to Gabor wavelets, which are sinusoids multiplied by a smooth spatial envelope.

[0006] A steerable pyramid can be real-valued by having all real coefficients, but it can be extended to complex-valued coefficients where the sinusoidal basis functions are replaced by a complex exponential. In other words, the real part represents an even-symmetric filter (cosine phase), while its imaginary counterpart represents an odd-symmetric filter (sine phase). While the real and imaginary parts increase the over completeness of the representation, the complex-valued steerable pyramid is a richer representation that separates the amplitude of the local wavelet from its phase, allowing for the convenient measurement of local phase information.

[0007] The present method builds on the link between phase and motion, but avoids the explicit computation of flow vectors, and instead directly manipulates phase variations in videos. The present method further employs the phase component to analyze motion.

[0008] In one embodiment, a method of amplifying temporal variation in at least two images includes examining pixel values of the at least two images. The temporal variation of the pixel values between the at least two images is below a particular threshold. The method then applies signal processing to the pixel values.

[0009] In another embodiment, a method of amplifying temporal variation in at least two images includes converting two or more images to a transform representation. The method further includes, for each spatial position within the two or more images, examining a plurality of coefficient values. The method additionally includes calculating a first vector based on the plurality of coefficient values. The first vector can represent change from a first image to a second image of the at least two images describing deformation. The method also includes modifying the first vector to create a second vector. The method further includes calculating a second plurality of coefficients based on the second vector.

[0010] In another embodiment, the transform representation is a pyramid. The pyramid can be a complex steerable value pyramid. The first and second pluralities of coefficient values can include real and imaginary parts of the coefficients.

[0011] In one embodiment, the first vector can be a scalar. Modifying the first vector to create the second vector can multiply the scalar by a constant. The first vector represents a change in phase from a first image of the at least two images and the second image of the at least two images.

[0012] In another embodiment, the method can also include generating at least one image based on the second plurality of coefficients.

[0013] In another embodiment, the two or more images may be a set of two or more images.

[0014] In a further embodiment, the method may include temporally bandpassing the second vector.

[0015] In one embodiment, a system for amplifying temporal variation in at least two images can include a transform module configured to convert two or more images to a transform representation. The system can further include a coefficient examination module configured to, for each spatial position within the two or more images, examine a plurality of coefficient values. The system can further include a vector calculation module configured to calculate a first vector based on the plurality of coefficient values. The first vector can represent change from a first image to a second image of the at least two images describing deformation. The system can additionally include a vector modification module configured to modify the first vector to create a second vector. The system can also include a calculation module configured to calculate a second plurality of coefficients based on the second vector.

[0016] The embodiments of the system and method described above can be referred to as a “phase-based” system and method.

[0033] A description of example embodiments of the invention follows.

[0034] FIGS. 1A-B are images 100 and 140 illustrating motion magnification of a source video of a baby breathing. FIG. 1A shows a frame 100 from an input sequence (baby), with a spatiotemporal vertical-line-versus-time (YT) slice 110 taken at the location marked by vertical line 102. FIG. 1B shows a motion magnified frame 140 based on the input sequence with spatiotemporal YT slices 150 and a zoom-in of motion 144 on a patch around the baby's chest in the insets. The magnification method illustrated by FIG. 1B is a phase-based magnification. The magnification method illustrated by FIG. 1B supports larger magnification factors with less artifacts and noise compared to other methods.

[0035] The method and apparatus described herein magnifies and reveals small movements in videos based on a novel analysis of motion employing complex-valued image pyramids. Instead of employing computation of optical flow or motion vectors that Lagrangian methods require, the method shows that the variation of the phase of the coefficients of a complex-valued steerable pyramid over time corresponds to motion and can be temporally enhanced or modulated. The method is fast, and is less sensitive to noise than previous Eulerian approaches.

[0036] Many phenomena exhibit motions that are too small to be well perceived by the naked eye, and require computational amplification to be revealed. Lagrangian approaches compute velocity and warp frames of the video according to magnified velocity vectors, which additionally requires background inpainting. One Eulerian approach alleviates the need for costly flow computation, and performs processing in a separable manner in space and time. Eulerian video processing can also be employed to dampen temporal aliasing of motion in videos. Unfortunately, typical Eulerian video magnification supports relatively small magnification factors for high spatial frequencies, and tends to significantly amplify noise when the magnification factor is increased.

[0037] To solve these problems, the method presents a new Eulerian approach to motion magnification based on complex-valued steerable pyramids, inspired by phase-based optical flow and motion without movement. The local phase variation over time of the complex-valued steerable pyramid coefficients corresponds to motion. After computing this phase variation, temporal processing can amplify motion in selected temporal frequency bands and reconstruct a modified video.

[0038] The link between motion and phase in steerable pyramids can be analyzed to determine limits of the method, which are linked to the spatial support of the steerable basis functions. The method improves on the previous Eulerian magnification method in two ways:

[0039] 1) The method achieves twice the magnification of prior methods in theory and is at least as good in practice; and

[0040] 2) The method has substantially better noise performance. Prior methods amplify noise linearly because they amplify temporal brightness changes. In contrast, the present method modifies phase, not amplitude, which translates noise without increasing its magnitude. The present method can achieve a greater degree of motion magnification while producing fewer artifacts in similar running times. Moreover, as the method supports larger degrees of motion modulation, it can further enable new applications for video manipulation such as motion fast forward, and motion pause.

[0041] FIG. 2A illustrates an input frame 200 from a video showing a membrane 204 oscillating in response to a 110Hz sound wave played by a speaker 202.

[0042] FIG. 2B illustrates ideal transfer functions 210 of some different filters in a complex steerable pyramid with three scales and two orientation bands, drawn on the spatial frequency plane (high and low pass residuals 212 in medium gray; black pixels 214 indicate spatial frequencies not modulated by the transfer functions of the steerable pyramid).

[0043] FIG. 2C illustrates phase videos 220 corresponding to different scales and orientations in FIG. 2B. FIG. 2D illustrates observations of phase over time for a single location 222 (FIG. 2C). Scale and orientation 230 reveals motions at the frequency of interest 240 (e.g., 110 Hz). By temporally filtering and amplifying the phase data, the method magnifies the motions in a required frequency range.

[0044] The steerable pyramid has high and low pass residuals that do not have orientation and are real-valued. The high and low pass residuals are identical to the corresponding levels in a real steerable pyramid. The transfer functions in the oriented bands of the steerable pyramid Ψ_ω,θ are scaled, rotated copies indexed by scale ω and orientation θ. The steerable pyramid is built by applying these transfer functions to the Fourier transform {hacek over (I)} of an image I to decompose it into different spatial frequency bands S_ω,θ={hacek over (I)}Ψ_ω,θ. That is, the Fourier transform of each pyramid sub-band is obtained by multiplying the FFT of the image by a “mask” similar to those in FIG. 2B, which selects a range of frequencies and orientation. FIG. 2B shows that the positive frequency values ω are retained and that their symmetric −ω values are zeroed out. This is why, when the masked FFT is inverse-transformed back to the image domain, the corresponding sub-band has complex values. Then, we utilize the real component of the inverse-transformed images.

[0045] In the frequency domain, the processing to build and then collapse the pyramid yields the reconstructed image, Ĩ_r, which is given by

Ĩ_R=ΣS_ω,θΨ_ω,θ=ΣĨΨ_ω,θ²   (1)

[0046] where the sums are over all of the scales and orientations in the pyramid. The processing is done in the frequency domain, but the method stores and uses the complex-valued results in the primal domain.

[0047] The transfer functions of a complex steerable pyramid only contain the positive frequencies of a real steerable pyramid's filter. That is, the response of 2 cos(ωx)=e^iωx+e^−iωx is e^iωx (see FIG. 2). The transfer functions mask regions, so that they are capable of isolating a range of frequencies as opposed to a single localized frequency.

[0048] The present method computes the local phase at every scale and orientation. Then, the method temporally bandpasses these phases to isolate specific temporal frequencies and in particular to remove the DC component. These temporally bandpassed phases correspond to motion in different scales and orientations. To synthesize magnified motion, the method multiplies the temporally bandpassed phases by an amplification factor α. The method then uses these amplified phase differences to magnify (or decrease) the motion in the sequence by modifying the phase of each frame by this amount.

[0049] In one example, the method analyzes a one-dimensional (1D) image intensity profile f under pure translation over time, f(x+δ(t)), for some displacement function δ(t). The method attempts to synthesize a motion-magnified sequence, f(x+(1+α)δ(t)), for the amplification factor α.

[0050] First, a linear magnifying a video applies a temporal filter to the intensity sequence. Assuming a DC balanced broadband temporal filter, this results in

$\begin{array}{cc}B\left(x,t\right)=\delta \left(t\right)\frac{\partial f}{\partial x}& \left(2\right)\end{array}$

[0051] which is then amplified and added back to the original frames. This leads to the following first-order (linear) approximation of the motion-magnified signal:

$\begin{array}{cc}f\left(x\right)+\left(1+\alpha \right)\delta \left(t\right)\frac{\partial f}{\partial x}\approx f\left(x+\left(1+\alpha \right)\delta \left(t\right)\right)& \left(3\right)\end{array}$

[0052] This approximation holds for small values of α and δ(t) and low spatial frequencies, but breaks down quickly and causes artifacts (e.g., noise).

[0053] An embodiment of the phase-based method relies on complex-valued steerable pyramids because they afford a local notion of phase that allows direct shifting of the local Gabor-like wavelets that represent the image. The phase-based method employs a Fourier expression of the displaced image profile f(x+δ(t)) as a sum of complex sinusoids.

$\begin{array}{cc}f\left(x+\delta \left(t\right)\right)=\sum _{\omega =-\infty }^{\infty }A_{\omega }{}^{\mathrm{\omega }\left(x+\delta \left(t\right)\right)}& \left(4\right)\end{array}$

[0054] For this analysis, the phase-based method approximates the pyramid construction and assumes that the Fourier masks in FIG. 2B select a single frequency ω and angle θ per pyramid orientation band. Starting from Eq. 4, the primal-domain idealized pyramid level for frequency ω is

S_ω,θ(x,t)=A_ωe^iω(x+δ(t))   (5)

[0055] FIG. 3 illustrates phase-based motion magnification for the case of sinusoidal functions. In these plots, the initial displacement is δ(t)=1. True amplification 300 is illustrated compared to amplification using a linear method 310 and the phase-based embodiment 320. While the errors for the technique of the linear method 310 of the method are dependent on wavelength for sinusoids, any errors in graph 330 from the phase-based embodiment 320 of the method are not dependent on wavelength and the error is uniformly small. The phase-based errors curves 338 are superimposed about zero error in graph 330, along the x-axis. On the other hand, linear-based error curves 332, 334, and 336 are shown to be greater for lower wavelengths, and only become close to zero as the wavelength becomes larger.

[0056] The pixels of sub-band S_ω,θ have complex values and their magnitudes encode the amplitude of the sine wave, while their complex phases depends on the pixel and vary according to the translating motion.

[0057] The phase-based method manipulates the phase of the bands of the pyramid. The phase-based method makes derivations simple by applying a zero-DC broadband temporal filter to the phase components, ω(x+δ(t)), for which the response for every scale and orientation is

B_ω,θ(x,t)=ωδ(t)   (6)

[0058] The phase-based method multiplies the bandpassed phase by an amplification factor α, takes a complex exponential, and multiplies the result by the corresponding scale and orientation band, to get

$\hspace{1em}\begin{array}{cc}\begin{array}{c}{\hat{S}}_{\omega ,\theta }\left(x,t\right)=S_{\omega ,\theta }\left(x,t\right){}^{\mathrm{\alpha }B_{\omega ,\theta }}\\ =A_{\omega }{}^{\mathrm{\omega }\left(x+\left(1+\alpha \right)\delta \left(t\right)\right)}\end{array}& \left(7\right)\end{array}$

[0059] The phase-based method receives the analytic signal at each scale and orientation, phase-shifted by (1+α)δ(t). The phase-based embodiment obtains the output motion-magnified sequence by taking the real part of Eq. 7 recovering the negative spatial frequencies, and collapsing the steerable pyramid, which approximates f(x+(1+α)δ(t)). FIG. 3 illustrates that the phase-based method is nearly exact for sinusoidal waves, as such signals contain a single spatial frequency.

[0060] A linear method is exact in the case of linear ramps while the phase-based embodiment of the method is nearly exact for sinusoidal waves (FIG. 3). However, both the linear method and phase-based method rely on spatial pyramids, where each level is bandlimited. Such spatially bandpassed images are better approximated by sinusoidal waves than linear ramps.

[0061] FIG. 4 illustrates diagrams 400 and 410 showing the phase-based method achieving performance being at least twice that of a linear method without suffering from clipping artifacts for general non-periodic structures. Wrap-around artifacts may become noticeable when structures are phase-shifted through 2π for large amplification (e.g., diagram 410).

[0062] As a rule of thumb, if the spatial wavelength lambda, the magnification alpha, and the original motion displacement delta(t) satisfy this relationship:

$\begin{array}{cc}\mathrm{\alpha \delta }\left(t\right)<\frac{\lambda }{4}& \left(8\right)\end{array}$

[0063] then the motion magnification gives accurate and visually appealing results.

[0064] Phase-based motion magnification has excellent noise characteristics. As the amplification factor is increased, noise is translated rather than amplified. At a particular scale and orientation band, the response for a noisy image I+σn might be

S_ω,θ=e^iω(x+δ(t))+σN_ω,θ(x,t)   (9)

[0065] σ is much lower in magnitude than the noiseless signal, so that temporal filtering of the phase is approximately ωδ(t) as in Eq. 5. To magnify the motion, the response is shifted by e^iαωδ(t), so that the motion magnified band is

Ŝ_ω,θ=e^{iω(x+(1+α)δ(t))}+σe^iαωδ(t)N_ω,θ(x,t)   (15)

[0066] The only change to the noise after processing is a phase shift. When the pyramid is collapsed, this phase shift corresponds to a translation of the noise. In contrast, a linear method amplifies the noise linearly in proportion to α.

[0067] In the phase-based method, all processing is performed using a complex steerable pyramid with scales that are an octave apart and four orientations. Complex steerable pyramid code computes the filter responses in the frequency domain. Processing is performed in YIQ color space, so that the amplification factor could be attenuated in the chrominance channels. It took an order of a few minutes to process a 512×512 pixel video with 300 frames using non-optimized MATLAB code on a machine with 12 cores and 64 GB of RAM. It can be efficiently implemented to run in real time similar to the linear embodiment as computing a steerable—rather than Laplacian—decomposition introduces only a minor performance overhead. The user has control over the amplification factor and the temporal bandpass filter.

[0068] FIGS. 5A-D illustrate comparisons between linear and phase-based Eulerian motion magnification in handling noise. FIG. 5A illustrates a frame 600 in a sequence of IID noise. FIGS. 5B-C illustrate motion 610 and 620 amplified by a factor of 50, where FIG. 5B employs the linear method and FIG. 5C employs the phase-based method. FIG. 5D illustrates a plot 630 of the by showing root-mean-square deviation (RMSD) as as function of noise for each method, using several magnification factors, showing the advantages of the proposed phase based approach versus the linear based approach. For example, the RMSE 632, 634, and 636 from linear methods all increase at a rate faster than the RMSE 638 from the

[0069] The phase-based method can be applied to a set of natural and synthetic videos, and the results can be compared with the some of them ones by the linear embodiment. The results on baby (FIG. 1), and guitar and camera (FIG. 6) confirm our noise analysis and derived bounds on amplification. In particular, the magnified motions in the results of the phase-based method (e.g. the respiratory motions of the baby and vibrating guitar strings) looks crispier, and contain significantly less artifacts and noise compared to a linear method. At the cost of moderate artifacts, the phase-based embodiment can magnify the motion of camera dramatically more than previous Eulerian techniques.

[0070] The phase-based embodiment can magnify microsaccades, which are subtle, involuntary, low amplitude (10-400 micron) movements in the human eye. The sample video of the microsaccades is taken with a high speed camera at 500 Hz. The phase-based embodiment processed a one second (500 frame) sequence with an ideal bandpass signal with passband between 30 Hz and 50 Hz. The phase-based embodiment applies a spatial mask to the phase shifts to emphasize the motion around the iris. Such a detection system can have clinical applications, as the frequency content of ocular microtremor can have clinical significance.

[0071] FIG. 6 illustrates representative frames 700, 710, 720, 730, 740 and 750 from videos used to demonstrate the methods.

[0072] In another demonstration, a tense membrane is made of a sheet of rubber mounted on a section of PVC pipe using a rubber band. A loudspeaker vibrates air that in turn vibrates the membrane. A high speed camera captures the result. The membrane has two modes when waveforms at 76 Hz and 110 Hz are sent through the loudspeaker. A video of the membrane is captured when a composite waveform of these two frequencies are played through the loudspeaker. The phase-based method separates and amplifies these two modes.

[0073] As the embodiments support large amplification factors, a user can increase the amplification, α, with time to simulate motion fast forwarding. This creates the illusion of speeding up time. For example, the phase-based embodiment can speed up motion of a shadow moving over time and show what would happen if the sequence had been recorded for a longer period of time.

[0074] The phase-based method also allows pausing motions in videos. The phase-based method can remove low amplitude short term motions from videos while larger amplitude motions continue to pass through. This is similar to motion denoising and video de-animation, but can be done in real-time in the phase-based method. To pause the motion in a sequence, the phase-based embodiment computes the phases for a single reference frame and sets the phases in the entire video equal to that of the reference frame. The result is not the same as a constant reference frame as the coefficient amplitudes are still evolving over time. For example, the phase-based method can remove the motion of the subway car while the car passes through the tunnel. In another example, the phase-based method can remove the rustling of the leaves while the illumination in the scene changes.

[0075] The method can also amplify color variations by temporally bandpassing the amplitude (Eg. 4) and the low-pass residual of the steerable pyramid. This yields similar results to the linear method for extracting and visualizing the human pulse signal, since in both cases the same processing is applied to the low-pass residual band of the image pyramid (e.g., a Laplacian pyramid in the case of the linear method, and a steerable pyramid in the case of the phase-based method).

[0076] Lagrangian approaches to motion magnification can amplify the motion in a video arbitrarily, but rely on accurate optical flow estimates, image segmentation, and inpainting. Like the Lagrangian approach, the phase-based method translates structures within the image. However, it does not involve lengthy computation and it can run in real time. The approach is limited by the fact that structures cannot be translated beyond the spatial envelopes of the steerable pyramid. High frequencies cannot be translated as far as low frequencies and this leads structure to break up as the low frequency components translate beyond the high frequency ones.

[0077] FIG. 7 is a flow diagram 900 illustrating an example of embodiment of the phase-based method described in this application. The phase-based method begins by converting two or more images to a transform representation (902). Then, the method examines coefficient values for each spatial position in the two or more images (904). The method then calculates a first vector based on the coefficient values (906). Then, the method modifies the first vector to create a second vector (908). The second vector represents amplified movement between the two images. The method then calculates a second set of coefficient values based on the second vector (910). Then, the method generates an image based on the second set of coefficients (912). This generated image should have the motion amplified according to the second vector.

[0078] FIG. 8 illustrates a computer network or similar digital processing environment in which the present invention may be implemented.

[0079] Client computer(s)/devices 1050 and server computer(s) 1060 provide processing, storage, and input/output devices executing application programs and the like. Client computer(s)/devices 1050 can also be linked through communications network 1075 to other computing devices, including other client devices/processes 1050 and server computer(s) 1060. Communications network 1075 can be part of a remote access network, a global network (e.g., the Internet), a worldwide collection of computers, Local area or Wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth, etc.) to communicate with one another. Other electronic device/computer network architectures are suitable.

[0080] FIG. 9 is a diagram of the internal structure of a computer (e.g., client processor/device 1050 or server computers 1060) in the computer system of FIG. 8. Each computer 1050, 1060 contains system bus 1179, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. Bus 1179 is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements. Attached to system bus 1179 is I/O device interface 1182 for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer 1050, 1060. Network interface 1186 allows the computer to connect to various other devices attached to a network (e.g., network 1075 of FIG. 10). Memory 1190 provides volatile storage for computer software instructions 1192 and data 1194 used to implement an embodiment of the present invention (e.g., motion magnification code detailed above). Disk storage 1195 provides non-volatile storage for computer software instructions 1192 and data 1194 used to implement an embodiment of the present invention. Central processor unit 1184 is also attached to system bus 1179 and provides for the execution of computer instructions.

[0081] In one embodiment, the processor routines 1192 and data 1194 are a computer program product (generally referenced 1192), including a computer readable medium (e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the invention system. Computer program product 1192 can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication and/or wireless connection. In other embodiments, the invention programs are a computer program propagated signal product 1070 embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)). Such carrier medium or signals provide at least a portion of the software instructions for the present invention routines/program 92.

[0082] In alternate embodiments, the propagated signal is an analog carrier wave or digital signal carried on the propagated medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other network. In one embodiment, the propagated signal is a signal that is transmitted over the propagation medium over a period of time, such as the instructions for a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of computer program product 1192 is a propagation medium that the computer system 1050 may receive and read, such as by receiving the propagation medium and identifying a propagated signal embodied in the propagation medium, as described above for computer program propagated signal product.

[0083] Generally speaking, the term “carrier medium” or transient carrier encompasses the foregoing transient signals, propagated signals, propagated medium, storage medium and the like.

[0084] While this invention has been particularly shown and described with references to example embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.