US 4,862,256 AGrant
Method Of, and Apparatus For, Coloring a Black and White Video Signal
Issue Date:1989-08-29
•13 Claims
•2 Drawing Sheets
Abstract
A method of coloring a black and white video signal is provided. At least one area of a frame is defined with reference to luminance values in that frame, and luminance levels are assigned to that area. A color is then assigned to that area, and the area is colored with it. In a subsequent frame, luminance levels are used to determine which part of the preceeding frame has been carried over into the subsequent frame and which portions of the subsequent frame adjacent that part have the same luminance values. That part and those portions are then colored with the respective assigned color.
Metadata
Assignee
- Colorization Inc.
Inventors
- Wilson Markle
- Christopher Mitchell
Application Information
Application Number:US 2756323
Filing Date:1988-11-16
Priority Date:1983-05-05
Art Unit:263
Classifications
IPC:
H04N 943
Field of Search:
35827;28;81;82;169;22
Patent Drawings (2 sheets)
Description
This invention relates to a method of, and an apparatus for, coloring a black and white video signal.
At the present time, nearly all films and video recordings are produced in colour. Consequently, the public are used to colour productions, and expect such products to be in full colour. However, there is also a large stock of old black and white productions, which were made before the advent of colour technology. In order to make these old black and white productions more acceptable to the public, and to enhance their quality and appearance, it is desirable to colour these old black and white products. Theoretically, it is possible to colour an old black and white film by hand, but since there are 24 frames of film per second this would take an impractical amount of effort and time.
Accordingly, the present invention is directed to a method and apparatus which enable a black and white video signal to be modified or coloured, without requiring each individual frame to be coloured by hand.
According to the present invention, there is provided a method of modifying a black and white video signal comprising successive frames, the method comprising:
(a) Defining at least one area of a frame with reference to luminance values in that frame;
(b) Assigning luminance levels to each such area;
(c) Assigning a colour to each such area;
(d) Colouring each such area with its assigned colour by adding a colour signal to the black and white video signal;
(e) Comparing a subsequent frame to a preceding coloured frame and determining from the luminance values in the subsequent frame and the respective assigned luminance levels which parts of each coloured area from the preceding coloured frame are present in the corresponding area of the subsequent frame;
(f) Colouring those parts of the subsequent frame with the respective assigned colour by adding a colour signal to the black and white video signal;
(g) Determining which portions of the subsequent frame adjacent a coloured part of that frame have a luminance value within the assigned luminance levels for that coloured part;
(h) Colouring each such portion with the respective assigned colour by adding a colour signal to the black and white video signal;
The basic technique of the present invention described relies on the principle of assigning colour on the basis of luminance levels. However many old black and white films show poor contrast, and some gray or luminance levels are crushed making it difficult to differentiate by luminance areas which should not be coloured the same. In other words, where one should see a certain range of luminance levels, one in fact finds only a narrow band of luminance levels, thus making it difficult to distinguish different areas of each frame by means of the luminance levels. To overcome this it is desirable to provide some means of modifying a video signal in order to expand some luminance levels and to compress others.
Accordingly, our issued U.S. Pat. 4,710,805 provides a method and apparatus for the compression and expansion of luminance levels.
For a better understanding of the present invention, and to show more clearly how it may be carried into effect, a description of the method of the present invention will now be given by way of example, with reference to the accompanying drawings, in which:
FIG. 1 shows a circuit diagram of a circuit for adjusting a video signal;
FIG. 2 shows a graph of Voltage against Time for an input video signal;
FIG. 3 shows a graph of the output Voltage against Time for a first clipper amplifier;
FIG. 4 shows a graph of the output Voltage against Time for a second clipper amplifier;
FIG. 5 shows a graph of the D.C. control Voltage against Time;
FIG. 6 shows a graph of Voltage against Time for a modified video signal; and
FIG. 7 shows a block diagram of the circuit.
The first frame of a picture, or each separate sequence in the picture, generally has to be coloured by hand.
To colour the first frame, different parts of the frame are assigned a particular colour on the basis of luminance or gray levels. For example, the area of a persons face can be defined between two different luminance values, and this luminance range assigned an acceptable fleshtone colour. Then, each article of a person's clothing can similarly be defined by a pair of luminance levels, and coloured a chosen colour. Note, that the operator is not restricted to colouring each article with the colour that it originally had. He can choose any colour that he wishes.
For images or individual pictures which have a relatively small number of parts of different colour, which are defined by clearly spaced luminance levels, there would be little or no problem in colouring the image. Difficulties can occur if two adjacent parts which clearly should be different colours have similar luminance levels, for example due to poor contrast quality of the film. For example, a person with a pale complexion could appear against a light background and due to the low contrast value of the film both may read similar luminance levels. In this case, care is needed to sufficiently distinguish luminance levels for the person's face and the background, which ensure that the correct areas are coloured with a flesh tone and with a background colour.
Once this first frame has been coloured by hand, subsequent frames can be coloured automatically be a computer.
For subsequent frames, the computer compares the frame with the previously coloured frame. In the presence of motion, generally one will find that one part of the area in the previous frame is no longer present at the end of the subsequent frame, and in the subsequent frame the corresponding area will include a part not present in the first frame.
Firstly, parts of the area from a first frame which are still retained in the same area of a second frame are identified from their luminance values and coloured with the colour previously assigned to that area and those luminance values. This, in effect eliminates that part of the area in that first frame which has not been carried over into the subsequent frame. Then, it is determined which portions of the subsequent frame are adjacent the now coloured part and have a luminance value within the assigned luminance levels for that colour thereby indicating that these portions are part of the same area or object. These portions are then coloured with the colour assigned to that area and the corresponding luminance levels. This effectively colours the portions of the area or objects in the subsequent frame, which are not present in the first frame. To summarize, effective movement of an area or object from one frame to a subsequent frame is achieved by a two-step process comprising;
elimination of those parts of the area in the first frame not carried over into the subsequent frame;
and identification of new parts of the area in the subsequent frame not shown in the first frame.
This technique is repeated for each separately coloured area with its associated assigned luminance levels and colour, for the whole of the frame. For all frames, this technique is clearly not going to enable full colouring to be achieved. There are a variety of changes in the image, which are not encompassed by this technique. For example, a new person or object can enter the picture, for which there is no assigned colour and luminance value. Also, in the case of a street scene for example, one vehicle can pass behind another and then re-emerge. The re-emerging vehicle will effectively have to be treated as a new object.
In order to colour the whole of each frame, a number of techniques can be used. For small areas, which are only present for a short time or small number of frames, it is possible that one can ignore them and not colour them without a viewer realizing any detriment in the picture quality. This could occur, for example, in the case of an aircraft which passes across the corner of the background of the picture in the far distance. For each uncoloured area, the computer could make an educated guess from the colours of surrounding area, the former colour of that area and the background colour. For simplicity, a bland and unobtrusive background colour could always be used.
Another possibility is to interpolate backwards from later frames to earlier frames. The above described technique, enables a computer to follow moving objects from one frame to the next, later frame. Exactly the same principles could be supplied moving backwards through the frames. This could be particularly useful, in the case of an object which enters slowly from one side of the picture. Initially, the object might be so small that the operator would not be aware of it until it has been present for a considerable number of frames and occupies a sizable portion of the picture. It might then be apparent that the object, for example, a dog, is going to traverse across the full width of the picture, and consequently it will need to have a colour assigned to it. The operator can then find it a colour, and instruct the computer to use the technique outlined above to interpolate back through the frames to fully colour this object or area as it enters the picture from one side.
In any case, irrespective of the origin of any uncoloured area, to ensure the best quality picture, the operator himself can decide on the colour of any large uncoloured areas. Also, the operator will keep a check on the decisions made by the computer. The technique of instructing the computer to make educated guesses for uncoloured areas and then having them checked by the operator should enable rapid coloring to be effected. The computer can decide rapidly what colour to assign to an area, and if this is incorrect it should be immediately obvious to the operator who can then correct it. On the other hand, if the computer chooses the correct colour, the operator is not required to choose a colour and he can immediately go on to the next portion of the film or tape. It is quicker for the operator to check the colours and correct any miscolouring by the computer, than for the operator to consider each separate uncoloured area and choose a colour for it.
Modern video technology considers each picture to comprise numerous picture elements, so-called pixel. The use of a picture broken down into pixel is ideally suited to digital computer techniques.
In the present invention the computer can view each area in terms of certain adjacent pixels. When comparing subsequent frames, the computer compares the luminance value of each pixel in a given area with its luminance value in the previously coloured frame, in order to determine the changes in the shape and size of each coloured area. It will be understood that each pixel that has a different luminance level will indicate that the coloured area in question has moved past that pixel in some direction.
Turning now to the drawings, a description will be given of the luminance level or grey scale amplifier shown in FIG. 1, and its mode of operation.
The circuit has an input 1 for a video signal, and an output 2 for that signal. The input 1 is connected to the output 2 via a voltage controlled amplifier 3. Additionally, the input 1 is connected to inputs of a first clipper amplifier 5 and a second clipper amplifier 10. The first clipper amplifier 5 is provided with a variable resistance 6 for adjusting its gain. The first amplifier 5 also has an input connected to a device 7 for setting the clip level. The device sets an input voltage to the other input of the amplifier 5. The second clipper amplifier 10 is similarly provided with a variable resistance 11 for adjusting its gain and a device 12 for adjusting its clip level. The outputs of the two clipper amplifiers 5,10 are connected respectively to a positive input and a negative input of a summation amplifier 15. The output of the amplifier 15 is connected to a control input of the voltage controlled amplifier 3.
FIG. 2 shows an input video signal, which has two synchronizing pulses 20, and between these two pulses 209 a ramp wave form. The ramp wave form commences at T1 and finishes at T2. Now, this input video signal is supplied to both the clipper amplifiers 5 and 10. As shown in FIG. 3, in this example, the clip level of the first clipper amplifier 5, set by the device 7, is adjusted so that the output of the first clipper amplifier 5 does not commence until time T3. However, the gain set by the variable resistance 6 is adjusted so that at T2 the output of the first clipper amplifier 5 is equal to the input video signal at that time. Thus, the output of the first clipper amplifier 5 is a ramp wave form, which is shorter and steeper than the input video signal, and whose final amplitude is equal to the final amplitude of the input video signal.
As the output of the second clipper amplifier 10 is applied to the negative input of the summation amplifier 15, the output of the second clipper amplifier 10 is shown inverted in FIG. 4. Again, the clip level for this second amplifier 10 is so adjusted that the output does not commence until an even later time T4. The gain set by the device 12 is so adjusted that the final amplitude of the output of the second clipper amplifier 10 is equal to the final amplitude of the input video signal. Thus, the output of the second clipper amplifier 10 is an even shorter and steeper ramp wave form, whose final amplitude is equal to the final amplitude of the input video signal.
FIG. 5 shows the effect of combining the outputs of the two clipper amplifiers 5, 10 in the summation amplifier 15. In effect, the output of the second clipper amplifier 10 is subtracted from the output of the first clipper amplifier 5. FIG. 5 shows a graph of the output voltage from the summation amplifier 15. Also, shown in FIG. 5 by dotted lines are the outputs from the two clipper amplifiers 5, 10, these two dotted lines being marked 5 and 10 respectively. From time T1 to time T3, the output of the summation amplifier 15 remains constant, since the ramp outputs of both the clipper amplifiers 5 and 10 have not commenced. At time T3, the output of the summation amplifier 15 starts rising, in proportion to the output of the first clipper amplifier 5. This rise continues until time T4, when the ramp output of the second clipper amplifier 10 commences. Since the output of the second clipper amplifier 10 is steeper than the output from the first clipper amplifier 5, from time T4 the output of the summation amplifier 15 declines steadily. Finally, at time T2, as the final amplitudes of the ramp outputs of the two clipper amplifiers 5 and 10 are equal, the output of the summation amplifier 15 has returned to its initial value.
The output of the summation amplifier 15 is applied to the control input of the voltage controlled amplifier 3, and the effect of this peaked output on the original video signal is shown in FIG. 6. The original video signal is shown by a dotted line in FIG. 6. The two synchronizing pulses 20 are not effected by the voltage controlled amplifier 3. From time T1 to time T3, the control input is constant, and consequently the output of the voltage controlled amplifier 3 is the same as its input, i.e., the video signal remains unaffected. From time T3 to time T4, the rise in the voltage applied to the control input is accelerated steadily, and consequently the video signal is increased in proportion to this rising voltage. From time T4 to time T2, the rise in control voltage is decelerated until it reaches its original value at T2, causing the video signal to increase at a slower rate than the original input video signal, whereby its final amplitude is equal to the final amplitude of the original video signal.
Additionally, as shown in FIG. 2, the video signal has voltages V1,V2,V3 and V4 at the respective times T1,T2,T3 and T4. The output signal shown in FIG. 6 has effectively had part of the signal compressed and another part correspondingly expanded. The first part of the signal between voltages V1 and V3 remains unchanged. However, a second portion of the original input signal originally had voltages in the range V3-V4, but this has now been expanded to cover the range V3-V5. Similarly, the final portion of the original signal originally covered the band V4-V2, but in the output signal it only covers a narrower band V5-V2. Thus, the portion of the signal between T3-T4 has been expanded to cover a wider voltage range and hence a wider range of luminance levels. Also, the portion of the signals between T4 and T2 has been compressed, so now that it now only covers a narrower voltage band V5-V2 and hence a narrower range of luminous levels. This will enable areas which originally had luminance levels or voltages in the range V3-V4 to be more readily distinguished from one another.
Reference will now be made to FIG. 7, which shows a block diagram of the overall circuit for effecting the method. A video input 30 is provided for inputting the black and white video signal. A preferred method is to store the black and white video signal on tape, and in this case the video input 30 could simply be a video tape player. The video input 30 is connected to a video display unit 32, which can be a conventional colour monitor. This video display unit 32 is connected to various controls for controlling the apparatus, indicated as operator controls 34. The operator controls 34 enable the operator to control the video input 30 and other parts of the apparatus. A computer 36 is connected to the video display unit 32 and preferably is a digital computer. The computer 36 is also connected to a data input 38 and a video tape recorder 40. Via the operator controls 34 and the data input 38, the operator controls the computer 36 to add a colour signal to the original black and white video signal. The black and white video signal and the additional colour signal are stored separately by the computer 36, until a sufficient portion of the original black and white signal has been coloured and the operator is satisfied with the colouring. For example, the operator can colour a 10-minute sequence of the original film or video signal. The original black and white signal and the colour signal are then combined and recorded on a colour video tape by the video tape recorder 40.
Also, there are a number of ways in which the original luminance levels for each area in the first frame are determined. The operator can be provided with a device for tracing around part of an area on a screen. A computer can then analyze the luminance levels within that area, and assign those luminance levels to that area. If necessary, the operator can adjust the upper and lower limits of the luminance levels for that area. Also, for large areas, it may be found that the computer need only consider the luminance levels in a narrow strip around the outside of that area. This has been found to be particularly advantageous when a wide variety of luminance values occur within the area, but the strip around the edge of the area always has luminance levels within a fairly narrow band. The computer need then only look at the luminance levels in the strip around the edge, in order to follow the changing shape of the area from frame to frame, and it does not need to consider variations of luminance level within this outer strip.
Claims
The embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows:
1. A method of modifying a black and white video signal comprising successive frames, the method comprising: (a) Defining at least one area of a frame with reference to luminance values in that frame; (b) Assigning luminance levels to each such area; (c) Assigning a colour to each such area; (d) Colouring each such area with its assigned colour by adding a colour signal to the black and white video signal; (e) Comparing a subsequent frame to a preceding coloured frame and determining from the luminance values in the subsequent frame and the respective assigned luminance levels which parts of each coloured area from the preceding coloured frame are present in the corresponding area of the subsequent frame; (f) Colouring those parts of the subsequent frame with the respective assigned colour by adding a colour signal to the black and white video signal; (g) Determining which portions of the subsequent frame adjacent a coloured part of that frame have a luminance value within the assigned luminance levels for that coloured part; (h) Colouring each such portion with the respective assigned colour by adding a colour signal to the black and white video signal.
2. A method as claimed in claim 1, when applied to a video signal in which each frame comprises individual pixels, wherein each area and its adjacent parts are defined by a plurality of adjacent pixels, and the luminance values of the pixels are compared for assigning colour.
3. A method as claimed in claim 1, in which steps (e),(*f), (g), and (h) are carried out by means of a digital computer.
4. A method as claimed in claim 1, wherein the method is applied to individual sequences of frames, the colour signals are stored separately from the original black and white video signal, and the original black and white video signal and those colour signals are only combined after the entire sequence has been coloured.
5. A method as claimed in claim 1, wherein when a further area representative of an area not present in the said preceding frame appears in a sequence of frames, that area is treated in one of the following ways, by assigning appropriate luminance levels and a colour, and by not colouring the area.
6. A method as claimed in claim 5, in which steps (e)(f)(g) (h) are carried out by means of a digital computer.
7. A method as claimed in claim 6, wherein the digital computer is adapted to colour each new area with the colour assigned to the luminance levels of such new area.
8. A method as claimed in claim 6, wherein the computer is adapted to colour each new area a background colour.
9. A method as claimed in claim 6, wherein steps (a),(b),(c) and (d) are repeated for each new area.
10. A method as claimed in claim 1, wherein prior to step (a) of the method at least one band of luminance levels in said black and white video signal is expanded to occupy a wider range of luminance levels, and at least one other band of luminance levels is correspondingly compressed to occupy a smaller range of luminance levels.
11. A method as claimed in claim 10, wherein: the black and white video signal is fed to a voltage controlled amplifier and to a first clipper amplifier and a second clipper amplifier, each having means for adjusting the gain and means for adjusting the clip level of that clipper amplifier; the gain and the clip level of the first and second clipper amplifiers are set to produce desired output characteristics of the first and second clipper amplifiers; the respective output signals of said first and second clipper amplifiers are fed to a summation amplifier and summed to produce a control signal; said control signal is fed to said voltage controlled amplifier to alter its gain, whereby one band of luminance levels of the black and white video signal is expanded so that it occupies a wider range of luminance levels, and another band of luminance levels is compressed so that it occupies a narrower range of luminance levels.
12. A method of modifying a black and white video signal comprising successive frames, the method comprising: (a) defining at least one area of a frame with reference to luminance values in that frame; (b) assigning luminance levels to each such area: (c) assigning a colour to each such area; (d) adding a colour signal to the black and white video signal to colour each such area with its assigned colour; (e) comparing a subsequent frame to a preceding coloured frame and determining from the luminance values in the subsequent frame and the respective assigned luminance levels which parts of each coloured area from the preceding coloured frame are present in the corresponding area of the subsequent frame; (f) colouring those parts of the subsequent frame with the respective assigned colour by adding a colour signal to the black and white video signal; (g) determining which portions of the subsequent frame adjacent a coloured part of that frame have a luminance value within the assigned luminance levels for that coloured part; (h) Adding a further colour signal to the black and white video signal to colour each such portion with the respective assigned colour; (i) for each new area in a sequence of frames, when the presence of that new area is not determined until one frame after the area has appeared in a number of frames: (i) assigning a colour to that new area; (ii) colouring that new area with the assigned colour by adding a colour signal to the black and white video signal; and (iii) colouring parts of each new area present in frames immediately preceding said one frame by interpolating back through those frames.
13. A method of modifying a black and white video signal comprising successive frames, the method comprising: (a) expanding at least one band of luminance levels in said black and white video signal to occupy a wider range of luminance levels, and correspondingly compressing another band of luminance levels to occupy a smaller range of luminance levels; (b) defining at least one area of a frame with reference to luminance values in that frame; (c) assigning luminance levels to each such area; (d) assigning a colour to each such area; (e) colouring each such area with its assigned colour by adding a colour signal to the black and white video signal; (f) comparing a subsequent frame to a preceding coloured frame and determining from the luminance values in the subsequent frame and the respective assigned luminance levels which parts of each coloured area from the preceding coloured frame are present in the corresponding area of the subsequent frame; (g) colouring those parts of the subsequent frame with the respective assigned colour by adding a colour signal to the black and white video signal; (h) determining which portions of the subsequent frame adjacent a coloured part of that frame have a luminance value within the assigned luminance levels for that coloured part; (i) colouring each such portion with the respective assigned colour by adding a colour signal to the black and white video signal.
Patent Citations (15)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US3551589 | 1970-12-01 | Moskovitz | |
| US3647942 | 1972-03-01 | Siegel | |
| US3706841 | 1972-12-01 | Novak | |
| US3710011 | 1973-01-01 | Altemus et al. | |
| US3761607 | 1973-09-01 | Hanseman | |
| US3772465 | 1973-11-01 | Vlahos | |
| US3784736 | 1974-01-01 | Novak | |
| US3972067 | 1976-07-01 | Peters | |
| US4149185 | 1979-04-01 | Weinger | |
| US4395732 | 1983-07-01 | Upton | |
| US4403253 | 1983-09-01 | Morris | |
| US4574393 | 1986-03-01 | Blackwell | |
| US4606625 | 1986-08-01 | Geshwind | |
| US4608596 | 1986-08-01 | Williams | |
| GBX933938 | 1963-08-01 |