Method and system for preserving the creative intent within a motion picture production chain

[0001] This is a continuation of U.S. application Ser. No. 10/740,324, filed 18 Dec. 2003 now U.S. Pat. No. 6,972,828 by Mitchell J. Bogdanowicz et al., and entitled “A Method and System for Communicating and Preserving the Creative Intent Within a Motion Picture Production Chain”, wherein this prior Application is incorporated by reference herein.

[0002] The invention relates generally to the field of motion picture production, and in particular to effectively providing a communication tool for usage throughout the imaging chain that relates to transferring creative intent. More specifically, the invention relates to a creative intent recipe for maintaining a cinematographer's look and feel for a motion picture.

[0003] A motion picture production, either a commercial, television show or a theatrical movie starts with a creative intent by a cinematographer (and/or director and/or other people involved in the creative intent) for an overall “look” for the project and its incorporation throughout each scene.

[0004] This “look” can be the creation of a single individual, such as the cinematographer or it can be a collaborative creation of a few or several people. If there is more than one person, the task of communicating the creative intent to everyone becomes a problem. Most often, the best way to communicate the creative intent is by creating an image for sharing and discussing. This enables actual viewing of the image without relying solely on ambiguous words to describe the image. U.S. Pat. No. 6,122,006 issued to Bogdanowicz et al. on Sep. 19, 2000 discloses the creation of a “look” and creation of a printed image to communicate the creative intent.

[0005] However, the task of maintaining the “look” does not stop with the cinematographer. In a modern motion picture the film element is exposed, developed by a laboratory, and sometimes printed onto a motion picture print stock for projection. In the laboratory there are usually many choices in processing and various techniques to alter the photographic image. A communication link must be established between the cinematographer and the laboratory to continue the evolution of the creative intent of the image.

[0006] Currently, most motion pictures have special effects added in at a post-production facility. These special effects can be quite elaborate and intricate However, more and more motion picture productions are scanning in an entire film project to a digital intermediate (also known as a digital master), which can be altered in a digital-to-digital suite where global, and local color and tone scale changes are introduced. If the originators/creators of the “look” are not present in the post-production house, the “look” may be compromised by other individuals taste or ideas. Further, if an overall “look” is to be applied to the project on a scene-by-scene basis, a colorist (i.e., the person running the digital equipment) needs to try to apply the look consistently across the project.

[0007] Communication of the “look” throughout the motion picture production chain is advantageous for cinematographic consistency. The cinematographer is able to assume that the “look” will be correct and consistent from scene-to-scene. And yet, in post-production there are numerous degrees of freedom to change the image. A cinematographer cannot anticipate or influence the look that may be applied in a laboratory or postproduction facility. Additionally, in the postproduction facility, the application of the “look” can be a laborious process on a scene-by-scene basis. Furthermore, specific digital manipulation controls may not be robust throughout the entire dynamic range and may introduce artifacts or be unattainable with their instrumentation.

[0008] Also, an image can be altered by a 3D LUT (3-dimensional look-up table) to provide a “look”, but the normal process for producing a 3D LUT is to empirically measure a small set of image patches and use the resulting data to interpolate a 3D LUT. This method is inherently error prone and its effectiveness is limited to the chosen measured data. In the laboratory, alternative processes can introduce a time consuming and costly testing phase before principle camera work starts. Usually the “look” that these alternative processes impart on the image is revealed only after physical testing.

[0009] What is needed is a method and system for communicating the creative intent of the “look” to directors, producers, wardrobe, writers, laboratory personnel, colorists, clients and executives associated with the motion picture production.

[0010] The present invention is directed to overcoming one or more of the problems set forth above. Briefly summarized, one aspect of the present invention provides a method for communicating and preserving creative intent within a motion picture production chain, including a production stage, and at least one of a subsequent laboratory or post production stage, including capturing an original image or sequence of images of a scene during the production stage; generating a look representative of a creative intent, wherein the look is a digital representation of a creative alteration of the captured original scene image; and applying the look to the captured original scene image and confirming approval of the look during the production stage. The digital representation of the look is stored as a recipe for generating the creative alteration, and communicated from the production stage to at least one of the subsequent laboratory and postproduction stages for applying the look to the captured motion picture footage during subsequent laboratory and postproduction stages.

[0011] The present invention generates an image and produces a 3D LUT to apply a creative “look” to an image or stream of images (i.e., a scene). The present invention allows a cinematographer to capture (or input) an image either with a digital camera, scanned film or CGI (Computer Generated Image) and further alter the image by choosing alternative film stocks, processing, filters, color timing, telecine controls and other changes to the image. The representation of the changes can be saved and distributed as a “recipe” which can be sent to remote sites and loaded into the system to display the “look” the originators intended at an office (i.e., director, producer, etc.), laboratory or post production house. The digital display devices are also dually calibrated, first to a standard position (with regards to the brightness, black level, tone scale and white color balance), next to emulate a photographic motion picture print material projected in a darkened surround. This display device calibration ensures that people viewing the images remotely are seeing the same image with the “look” incorporated therein. The recipe can be global to the production or on a scene-to-scene basis.

[0012] Furthermore, the present invention enables a post-production person to generate, from the recipe, an entity, such as a three dimensional lookup table (3D LUT), which can be applied to a digitally scanned product, either in software or hardware. This 3D LUT will embody the “look”. The second part of the display calibration is a unique 3D LUT that emulates a particular print stock on a particular display device, such as a HD Monitor or a digital projector. The present invention uses this unique 3D LUT for the device to display the image, which includes the characteristics of the device, tone scale, color primaries or dye set (if the display device is a film print). For immediate display the recipe-created look 3D LUT (“look 3D LUT”) is combined with the device-dependent 3D LUT (“device 3D LUT”) to create a new 3D LUT (“display 3D LUT”), which will display the “look” on a specific device without altering the original data. The original data is maintained and transformations are applied in a forward manner, thus ensuring that there is no loss to the original data.

[0023] In the following description, at least one embodiment of the present invention will be described as a software program. Those skilled in the art will readily recognize that the equivalent of such software may also be constructed in hardware. Because image manipulation algorithms and systems are well known, the present description will be directed in particular to algorithms and systems forming part of, or cooperating more directly with, the method in accordance with the present invention. Other aspects of such algorithms and systems, and hardware and/or software for producing and otherwise processing the image signals involved therewith, not specifically shown or described herein may be selected from such systems, algorithms, components, and elements known in the art. Given the description as set forth in the following specification, all software implementation thereof is conventional and within the ordinary skill in such arts.

[0024] The computer program may be stored in a computer readable storage medium, which may comprise, for example, magnetic storage media such as a magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as an optical disc, optical tape, or machine readable bar code; solid state electronic storage devices such as random access memory (RAM), or read only memory (ROM); or any other physical device or medium employed to store a computer program. The present invention can be performed on any well-known computer system, such as a personal computer.

[0025] The present invention can generate an image and produce a 3D LUT to apply a creative “look” to an image or stream of images (i.e., a scene). The present invention allows a cinematographer to capture (or input) an image, either with a digital camera, scanned film or a computer generated image (CGI); and alter the image by choosing alternative film stocks, processing, filters, color timing, telecine controls or other image modifying processes. A representation of the changes can be saved and distributed as a “recipe” which can be sent to remote sites and loaded into a look management system to display the “look” the originators/creators (e.g., a director, a producer, etc.) intended at an office, laboratory or post-production house. To maintain the look, display devices are dually calibrated. A first calibration calibrates the display device to a standard position (with regards to brightness, black levels, tone scale and white color balance). A second calibration enables the look management system to emulate a photographic motion picture print material, as it would be projected in darkened surroundings or a dim surround if the final project is intended to be delivered for standard or high definition television. This display device calibration ensures that viewers remotely viewing the images are “seeing” the same image. The recipe, itself, can be global to the production or on a scene-by-scene basis.

[0026] Moreover, the look management system at the post-production house can generate from the recipe an entity such as a three-dimensional lookup table (3D LUT), which can be applied either in software or hardware to the motion picture project. This 3D LUT will embody the “look”, and is herein referred to as “look 3D LUT”. The second part of the display calibration is a second 3D LUT that uniquely emulates a particular print stock on a particular display device, such as a high definition monitor or a digital projector. The present invention uses this unique second 3D LUT for the device, which includes the characteristics of the device, tone scale, color primaries or dye set (if the display device is a film print), to display the image. For immediate display, the recipe created look 3D LUT, (“look 3D LUT”), is combined with the device dependent 3D LUT, referred to as (“device 3D LUT”) to create a third and different 3D LUT, referred to as (“display 3D LUT”), which will display the “look” on a specific device without altering the original data. The original data should be maintained and transformations be applied in a forward manner. This ensures that there is no loss to the original data.

Herein the following definitions apply:

[0027] A “Look” is the creative alteration of the captured original scene image (reality) to invoke an emotion or to emphasize an idea in the story telling process.

[0028] A “Recipe” is a computer file that includes all of the user's input choices (and some default values) that were used to create the “look” for the scene. It can be sent to another computer with similar software, which can then apply the recipe to an image and (on a colorimetrically calibrated output device) see the same image as the creative author. In the post-production facility it can be used to produce a 3D lookup table to process the whole or sequence of scenes.

[0029] For example, the recipe includes information that setup the look, such as emulation parameters, configurations, human readable information that facilitates text searching, and information expressed in a cinematographer's language (e.g., film stages chain, timing, processes, and principle photography). Other information may be included in the recipe such as binary information that enables tracking a user's information (e.g., author, user's computer information, license information, and user's system configuration). A user's monitor calibration information and a user's license level for recipe sharing, such as original author, and last author, may be included. The present invention may incorporate read only recipe management that necessitate inputting a password; recipe integrity checks that verify whether the recipe has been modified outside of the application; and/or recipe authenticity checks that verify whether the original intent has been modified by other users.

[0030] Referring to FIG. 1, a Look Management System (LMS) 100 communicates the creative intent of the originator/creator to persons in the motion picture production chain that need to understand and react to the “look” of the project. In one exemplary embodiment, an industry standard scanner 110 scans a film element 105. The original scene could also be captured by a digital camera device (115) or it can be constructed of computer-generated images (CGI) (120). The image is introduced into a Look Management Engine (LME) (130) where the creative changes can be made. LME 130 is driven by one of several originators/creators 125; for example, a Director of Photography (DoP), a producer, a director, a colorist, etc. The output images can be displayed on a display device 140, such as a digital Projector or CRT; alternatively it can be stored as a digital master 145, which can be recorded by a film recorder (not shown) and printed as film 150 for projection display.

[0031] One illustrative software overview for LMS 100 is shown in FIG. 2. Users of the software interact with a graphical user interface (GUI) (230) for all their choices. Alternatively, other possibly remote users can communicate by sending a “recipe” 225, which is a file that contains the instructions to the software about the changes to an image. Recipe 225 is used as an input to a script generator 235, which interprets the recipe 225 and generates a script (245) that is a logical sequence of transformation that will result in the perturbation of the image into the desired “look”. A processing engine (260) takes the script as input and processes image pixels (255), which are input (250) into software 250 that includes a plurality of image source types and metadata (e.g., DPX files, digital images). Each image source type has an associated source reader (255) to properly input the image. The Processing Engine (260) produces a math model in color science language and shown in FIG. 4.

[0032] Referring to FIG. 4, a math model (410) that includes color science language is created from the recipe (405). Math model 410 executes in a number of ways. For example, a three-dimensional look-up table, 3D LUT, (415) performs calculations and transforms upon the image. This process speeds up the transformation since only the points in 3D LUT 415 need to be calculated through the complete math model 410. The image is processed through 3D LUT 415 by either a software interpolation algorithm or an interpolation algorithm used as hardware code 430. One example of hardware code 430 is FPGA code, which is an algorithm used to interpolate a 3D LUT in hardware. The 3D LUT is created from the color science, found in math model 410. The implementation processing is not limited to using 3D LUT 415; each pixel can be processed either in script processing software (420) or wired in hardware code (425) for enhanced processing speed. The script processing software 420 enables platform independent processing.

[0033] Referring again to FIG. 2, the processing engine (260) generates images that incorporate the desired look and also transforms the image into the proper digital code values for displaying on properly calibrated display devices (265). Output device manager (240) communicates an image transform to the script generator 235, which transmits metadata about the display device 265 such that the script generated will faithfully render the image on the display device 265. Alternatively, a digital film recorder (not shown in Fig.2) writes the data, without the display transform, to film for subsequent printing and display. LMS simulation engine (220) generates recipe 225. LMS simulation engine (220) takes as input the user choices from GUI (230) and data from LMS data reader (215) to produce a recipe 225 that associates the user's choices with LMS color science data (210) generated by LMS data creator (205).

[0034] The total of the possible inputs and outputs for any subsystem and the method for moving image data through the processing chain is the Look Management System. The software that takes the users input and interprets that input into a recipe and then a mathematical model of the desired image chain is the look management engine.

[0035] Referring to FIG. 3, there are several possible 3D LUTs in the processing pathway. In FIG. 3, an example of one possible scenario is shown. Multiple input source color spaces (305), including DPX files 301, digital still camera 302, computer generated images (CGI) 303, and other scanned film data 304 are associated with their corresponding metadata (350), which enables selection of a proper 3D LUT(1)(310) to process the image. Several 3D LUTs(1) correspond to the input source color spaces 305, including 3D LUT(1) for DPX 311, 3D LUT(1) for digital still camera 312, 3D LUT(1) for CGI 313, and 3D LUT(1) for scanned film 314.

[0036] A LMS user (325) generates a recipe (320) which is used to create a “look” 3D LUT(2) (315) in the script generator (235) and processing engine (260), discussed earlier and shown in FIG. 2. A display device (345) is chosen and with the use of display calibration tools (330), a display device 3D LUT(3) (335 and/or 340) is created. 3D LUT(3) 335 corresponds to display device's color space exhibited by a monitor. 3D LUT(3) 340 corresponds to display device's color space exhibited by a digital projector. The total rendering of the images from the various input source color spaces (305) can be done in one 3D LUT by combining an input source 3D LUT(1) with a “look” 3D LUT(2) and a display device 3D LUT(3). Such a combined 3D LUT (350) should preferably be done in proper sequential order to maintain the accuracy of the math model 410, shown in FIG. 4.

[0037] The LME 100 can be used at different phases of the production cycle. First, in the pre-production subsystem, the LMS can communicate the creative intent even before the project filming begins. Referring to FIG. 5, a block diagram shows one embodiment of the use of LME 100 in pre-production. A scene (505) is captured on a photographic medium or device (510) by a cinematographer, director of production or other creator/originator. If the medium is film, a standard calibrated scan or preferably a “one light scan” (520) (a scan where the scanner is specifically set up in a calibrated mode to capture the largest range of film density that the characteristics of the scanner allow) is input to LME (530). Alternatively, a still image from digital camera or scanned film can be used as an input source 525. The user can produce many different scenarios of the film chain (535) and display them on a display device (555 and/or 560). Different scenarios can be choices of film types, exposure conditions, use of filters and process changes and other user inputs that alter the look of the image. Exposure conditions can be shutter or aperture control to alter the exposure to the film or digital camera, also light altering entities such as camera filters and lighting gels. Iteratively, the user can narrow down the choices until he is satisfied with the image's look. This look can then be distributed to the major principles in the project to communicate the creative intent of the upcoming production.

[0038] In a production subsystem, where film is shot on a day-to-day basis, the same scenario, as shown in FIG. 5 pre-production exists. However, some of the film from the daily production shoot can be scanned and input into the LME 530 to monitor and communicate to the production team. A recipe 532 or a rendered image can also be transmitted to a colorist at a film laboratory or to a telecine colorist in a digital post-production house for a reference while producing a daily image set. In production, where a daily image sequence is output not to a film print, but to a video medium (such as, tape or DVD, etc.) the present invention enables the creative “look” to be put into the first images coming back from the telecine scans, by rendering the images through a “look” 3D LUT with the corresponding display device 3D LUT. Consequently, with the present invention one can easily add the creative “look” using a 3D LUT (or a pixel-by-pixel software calculation), and deliver daily images with the desired look added.

[0039] Referring to FIG. 6, a film laboratory post-production subsystem is shown as alternative embodiment. LME 625 is intended for projects that only use film products in the imaging chain (as opposed to a hybrid of film and digital post-production or digital only post-production). LME 625 can show a cinematographer (or other creative) what look can be expected from the film exposure and processing choices selected. In this manner, the initial testing time is greatly decreased since many of the scenarios can be emulated in software to arrive at a “look.” Communication with the laboratory can be with printed images rendered with a recipe or electronic images rendered with a recipe. In FIG. 6, a scene (605) is photographed on film (as either a still image (606) or motion film image (635)) and processed in the laboratory. Alternatively, a digital image (610) from a digital camera 603 can be used. The scan from a still film scanner (615) is input (620) into the LME (625). The motion film images (635), exposed with their criteria (630), are processed at laboratory (640) and scanned (645) to generate an industry standard file. A cinematographer (650) oversees the creative “look” by initially creating the “look,” then sharing this “look” with the laboratory 640. The sharing can occur via a thermal print, ink jet print or an electronic display. Collaborative issues such as film choice (655), processing conditions (660), printing conditions, exposure and filter conditions (630) and selected display type (665,670) are communicated to persons or automated devices in the laboratory 640.

[0040] Referring to an illustrative digital post-production subsystem shown in FIG. 7, the present invention will have some degree of digital image rendering in the imaging chain. As in previously discussed subsystems for the present invention, an initial image for input into a LME 730 can be images derived from scanned film, CGI files, digital still cameras or other images having the following formats: JPEG, BMP, TIFF etc. Once a film is scanned (710) and stored as an industry standard DPX file (or other current and future industry standard formats), it can be loaded into the LME (730), such that a creator/originator can generate a “look” for the image (745). A proofing print can be generated (755) to help visually communicate the “look.” LME 730 creates a 3D LUT, which can be used to render the image stream in color correction software (750) or in hardware. The 3D LUT (or a recipe) can be transmitted to a data-to-data color grading system (725), such as DaVinci™ or Pandora™, which interpolates the data through the 3D LUT. The 3D LUT can be transmitted to an external (or internal) hardware interpolator (720) to render the image. The rendered image can be further fine-tuned in another color grading operation (780), or recorded to film (775) on a film recorder (770). In this manner, the original film scan can be left undisturbed on the path to the film recorder or other output digital formats (760), such as VCR, DVD etc. The main “look” is provided by the 3D LUT from the LME 730 and other color changes can be implemented through the data-to-data color correctors and other effects processors. These effects are stored as an edit decision list; this edit decision list, along with the 3D LUT, can completely render an original scan to the final output in a single pass without requiring storage of any intermediate data files.

[0041] A third alternative embodiment of a digital post-production subsystem is shown in FIG. 8. Film (805) is scanned via scanner 810 to produce digital files that contain the filmed image. Alternatively, an original digital image (815) can be used. In this embodiment for the present invention, there are at least two paths. In path A, a recipe or 3D LUT from LME (825) is passed to a hardware digital rendering box (840) (or software) which only has the “look” 3D LUT for rendering to film recorder output, or the “look” 3D LUT is combined with an output display 3D LUT which includes display device characteristics as well as characteristics of emulated print material color and tone scales. Utilizing path A, no further color correction is required to output a desired image with the “look” to film, or any digital display device.

[0042] Utilizing path B, an output of a hardware render box (830) is directed to a data-to-data color corrector (835). A 3D LUT in the hardware render box (830) only needs to be the “look” 3D LUT to impart the look before further color correction. After the data-to-data color grader (835), the image is rendered in a second hardware render box (840) which contains only the 3D LUT that takes the image data and renders it to a display device (850, 855) having associated display device characteristics as well as characteristics of emulated print material color and tone scales. In this embodiment, if the output is to the film recorder, the second hardware render box (840) can be skipped, or it could contain a second “look” 3D LUT for special looks.

[0043] The invention has been described with reference to a preferred embodiment. However, it will be appreciated that a person of ordinary skill in the art can effect variations and modifications without departing from the scope of the invention.

• [0044] 100 look management system
• [0045] 105 film
• [0046] 110 scanned film
• [0047] 115 digital still image
• [0048] 120 computer generated image (CGI)
• [0049] 125 originators/creators
• [0050] 130 look management engine
• [0051] 135 color correctors
• [0052] 140 digital projector
• [0053] 145 digital master
• [0054] 150 film output
• [0055] 205 LMS Data Creator
• [0056] 210 LMS Data
• [0057] 215 LMS Data Reader
• [0058] 220 LMS Simulation Engine
• [0059] 225 LMS Recipe
• [0060] 230 Graphical User Interface (GUI)
• [0061] 235 LMS Script Generator
• [0062] 240 Output device manager
• [0063] 245 LMS Script
• [0064] 250 Input Images and Metadata
• [0065] 255 Image source reader
• [0066] 260 LMS Processing Engine
• [0067] 265 Output Device (CRT, Digital Projector, Film recorder etc.)
• [0068] 305 Input Image source color space
• [0069] 310 3D-Lut(1) to support input image source type
• [0070] 315 3D-Lut(2) created from LMS “look”
• [0071] 320 LMS recipe
• [0072] 325 LMS user input choices
• [0073] 330 Display Device calibration tools
• [0074] 335 3D-Lut(3) for a specific calibrated monitor display device
• [0075] 340 3D-Lut(3) for a specific calibrated projector display device
• [0076] 345 Output Display Devices
• [0077] 350 Metadata for the Input Image source color space
• [0078] 355 Combined 3D LUT (3D LUT(1)+(2)+(3))
• [0079] 405 LMS recipe
• [0080] 410 Math Model from LMS recipe
• [0081] 415 3D-Lut implementation of LMS recipe
• [0082] 420 Math operations in script of LMS recipe
• [0083] 425 Math operations wire in hardware of LMS recipe
• [0084] 430 Hardware Code (i.e., FPGA code)
• [0085] 505 Original Scene
• [0086] 510 Test shoot (film or digital capture)
• [0087] 515 Cinematographer (or creative user)
• [0088] 520 Motion picture film scan
• [0089] 525 Still Picture film scan
• [0090] 530 Look Management Engine
• [0091] 535 User choices in the photographic image chain
• [0092] 540 Film Projector
• [0093] 545 3D-Lut to render data
• [0094] 550 Hardware digital rendering box
• [0095] 555 Calibrated monitor
• [0096] 560 Digital projector
• [0097] 605 Original Scene
• [0098] 606 Still film capture
• [0099] 610 Digital camera capture
• [0100] 615 Still film scanner
• [0101] 620 Image input to Look Management Engine
• [0102] 625 Look Management Engine
• [0103] 630 User specified film exposure conditions
• [0104] 635 Motion picture camera and film
• [0105] 640 Look Management System used in the Laboratory
• [0106] 645 Post Production film scan
• [0107] 650 Director of Photography (DoP) or creative user
• [0108] 655 LMS process database
• [0109] 660 Process details
• [0110] 665 “Look” on a calibrated display
• [0111] 670 Fine tuned “Look” on a calibrated display
• [0112] 705 Motion image origination images
• [0113] 710 Film scanner
• [0114] 715 Industry standard DPX file of image
• [0115] 720 3D LUT hardware interpolator
• [0116] 725 Industry standard color grading unit
• [0117] 730 Look management engine
• [0118] 735 Still digital camera input
• [0119] 740 Other still image input
• [0120] 745 User look generation (GUI choices)
• [0121] 750 Industry standard color corrector
• [0122] 755 Hardcopy proofing prints
• [0123] 760 Electronic distribution media
• [0124] 765 Digital Projector
• [0125] 770 Film recorder
• [0126] 775 Film output
• [0127] 780 Alternative color grading
• [0128] 805 Film input
• [0129] 810 Film scanner
• [0130] 815 Digital Image input
• [0131] 820 Industry standard color corrector
• [0132] 825 Look Management Engine
• [0133] 830 3D LUT hardware interpolator
• [0134] 835 Industry standard color grading unit
• [0135] 840 3D LUT hardware interpolator
• [0136] 845 Film recorder
• [0137] 850 Electronic distribution output masters
• [0138] 855 Digital Projector
• [0139] 860 Output film record