US 6,310,591 B1Grant
Spatial-temporal multiplexing for high bit-depth resolution displays
Issue Date:2001-10-30
•5 Claims
•6 Drawing Sheets
Abstract
A method and apparatus for spatially and temporally multiplexing display data. The use of this method results in a bit-depth resolution higher than that achievable by the system given a number of bits of resolution. The method includes the steps of determining the desired perceived resolution (26), establishing the number of bit-planes to be used to achieve that perceived resolution (28), using at least one of those bit-planes for spatial-temporal least significant bit values (STMLSBs) (30), referencing the developed values of the STMLSBs to fractional bit gray code levels (32), developing spatial patterns (34), determining whether the spatial patterns will start in a predetermined sequence or randomly from frame-to-frame (36), loading the data onto the modulator and displaying it (38). The apparatus includes a random number generator (48) and a look up table (50) to enable the choice between random and predetermined spatial patterns, and pattern logic (46), which produces the pattern to be used.
Metadata
Assignee
- Texas Instruments Incorporated
Inventors
- Daniel J. Morgan
- Gregory S. Pettitt
- Donald B. Doherty
Application Information
Application Number:US 09/370,419
Filing Date:1999-08-09
Priority Date:1998-08-18
Art Unit:7
Classifications
IPC:
G09G 334
Field of Search:
345 84345147345148345 88345 89345 87345 30345 55348771348760348761348755348753
Patent Drawings (6 sheets)
Description
[0002] This application claims priority under 35 USC § 119(e)(1) of provisional application No. 60/096,925 filed Aug. 18, 1998.
BACKGROUND OF THE INVENTION
[0003] 1. Field of the Invention
[0004] This invention relates to spatial light modulator display systems, more particularly to addressing schemes for these systems.
[0005] 2. Background of the Invention
[0006] Spatial light modulator display systems typically include a spatial light modulator comprised of an x-y array of individually controllable elements that are used to modulate picture elements (pixels) of an image. Examples of these modulators include Digital Micromirror Devices™ (DMD™), Actuated Mirror Arrays™, liquid crystal cells, grating light valves, and plasma display panels. Some of these examples operated in an analog fashion, where the amount of light transferred to any pixel is determined by how far the corresponding cell moves, or how much light is allowed through that cell. Others operate digitally, where the cell either transfers light to the image or not.
[0007] The digital mode of operation raises unique problems since the human eye has an analog response. This analog response requires that the digital cells use a technique called pulse-width modulation (PWM); In PWM techniques, the display signal undergoes digital sampling, resulting in a predetermined number of samples, each having the same number of bits. These bits are then used to address the individual cell in for time periods proportional to the bits' significance (i.e., the most significant bit receives the most time to display its data). Systems with a higher number of bits per sample provide better images.
[0008] Addressing the cells typically involves transferring the data for a given bit to the activation circuitry for a cell, causing the cell to respond to that data, then illuminating the cell to modulate the light in the manner required by that bit of data. The tasks of storing, transferring, activating and illuminating the cells must be repeated several times in a relatively short time to achieve high quality images. A typical display system operates at 60 Hz, so each frame of data has only 1/60 of a second (16.7 milliseconds) in which to be displayed. In a sequential color system, where the modulator is illuminated with each of the three colors, red, green and blue, in sequence, each color receives one-third of that time (5.57 milliseconds).
[0009] In current spatial light modulator display systems, the largest number of bits achievable in a sequential color system is typically 8 bits. Eight bits of data must divide the 5.57 milliseconds up between them, with the most significant bit receiving approximately one-half that time (2.79 milliseconds), and the least significant bit receiving roughly one-two hundred fifty fifth of that time (20 microseconds). The cells must have a switching time fast enough to display the data for the least significant bit (LSB) in 20 microseconds to achieve 8-bits of resolution. Higher numbers of bits require even faster switching speeds.
[0010] For larger display systems, such as digital cinema, resolution higher than 8 bits is necessary to achieve film quality images with spatial light modulator displays. In some examples, such as the DMD™, 10 bits can be achieved. Therefore, for digital cinema quality images, a method is needed that will allow spatial light modulators to display more than 10 bits of resolution without requiring an increase in switching speed.
SUMMARY OF THE INVENTION
[0011] One aspect of the invention is a method to spatially and temporally multiplex display data to achieve a higher bit-depth resolution. Generally, the invention includes the steps of determining the desired perceived resolution, establishing the number of bit-planes to be used to achieve that perceived resolution, using at least one of those bit-planes for spatial-temporal least significant bit values (STMLSBs), referencing the developed values of the STMLSBs to a bit weighting, developing spatial patterns, determining whether the spatial patterns will spatially start in each frame in a predetermined sequence or randomly, loading the data onto the modulator and displaying it.
[0012] In one embodiment of the invention, twelve bits of perceived resolution can be achieved from only 9 bit-planes. The 9 bit-planes are established, with 2 of those being used for spatial and temporal multiplexing. In another embodiment of the invention, fourteen bits can be referenced from only twelve. The twelve bit-planes are established, with 3 bit-planes for spatial and temporal multiplexing.
[0013] The spatial multiplexing is achieved by using patterns of percentages of active pixels within a frame, such as a 50% checkerboard pattern. These patterns are controlled temporally by starting the pattern spatially at a different starting point from one frame to the next. The determination of the pattern spatial start point can be either random or predetermined in each successive frame.
[0014] It is an advantage of the invention in that it allows higher bit-depth resolution to be achieved using a lower number of bits than would otherwise be necessary.
[0015] It is an advantage of the invention in that it can use LSB weightings that are non-binary, allowing longer bit on-times for LSBs and thus slower element switching speeds.
[0016] It is an advantage of the invention in that it can be used by any PWM system.
[0017] It is an advantage of the invention in that it uses fewer bit-planes, allowing more time to bit split higher order bits.
BRIEF DESCRIPTION OF THE DRAWINGS
[0018] For a more complete understanding of the present invention and for further advantages thereof, reference is now made to the following Detailed Description taken in conjunction with the accompanying Drawings in which:
[0019] FIG. 1 shows one example of a spatial light modulator display system.
[0020] FIG. 2 shows a flowchart of a method for spatial temporal multiplexing of display data.
[0021] FIG. 3 shows one embodiment of a 50% spatial temporal multiplexing pattern in accordance with the invention.
[0022] FIG. 4 shows one embodiment of a 25% spatial temporal multiplexing pattern in accordance with the invention.
[0023] FIG. 5 shows one embodiment of a 12.5% spatial temporal multiplexing pattern in accordance with the invention.
[0024] FIG. 6 shows one embodiment of a pattern selection block diagram in accordance with the invention.
[0025] FIG. 7 shows on embodiment of a pattern signal block diagram in accordance with the invention.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] FIG. 1 shows one example of a spatial light modulator display system 10. The display data, which can be from any display source, analog, digital, video, graphics, etc., is received at receiver 12. The data is also received at memory 14, if necessary for further processing. The processor block 16 could actually comprise several processors. It performs such tasks as analog-digital conversion, if necessary, color space conversion, and any other selected processing. The processor also controls the light source 18 and the color wheel 20.
[0027] The system uses a color wheel to perform color sequencing of the light for the spatial light modulator 22. This particular embodiment is for a one spatial light modulator reflective display system. However, the techniques set forth herein are not limited to such systems, and the discussion is in no way intended to limit it as such. The only requirement of a system to use this invention is that it employs pulse-width modulation. the processor in FIG. 1 also performs the necessary bit manipulations to format the data into the correct bit-plane formats for the correct colors.
[0028] A bit-plane in this instance is a set of display data, one bit for each element on the spatial light modulator, each bit having the same significance from the digital sample for that element. For example, the data either is already digital, or is converted to digital by the processor 14. Each element on the array has a sample of 8 bits that represents its data. All of the data for all of the elements is written into memory. The data is read out of the memory such that the most significant bit (MSB) from each sample is read out together. If the MSB is bit 7, for example, the resulting bit-plane of data would be all the bit 7sfor each element on the array. The bit itself will either be 1, representing that the element should be on for the MSB time, or 0, representing that the element should be off for the MSB time.
[0029] The times for each bit in a conventional pulse width modulation system are determined by the amount of time necessary for the least significant bit (LSB). Since the bits are in a binary system, each higher order bit will be some multiple of the LSB time. For an 8-bit system, for example, where bit 0 is the LSB, bit 1 is 2 LSBs, bit 2 is 4 LSBs, bit 3 is 8 LSBs, etc. Each element displays its data, 1 or 0, for each bit plane, which the human eye integrates into shades of gray. The addition of PWM for each color allows the eye to integrate colors as well.
[0030] Systems such as these produce color images typically in one of two ways. Color sequential systems, described above, sequence the data for each color to the device to coincide with that color illuminating the device. Another method is to provide one device for each color, and to converge the color images at the display surface. The invention as described herein can be used in either system, however, for initial discussion purposes, a color sequential system will be assumed, using one device.
[0031] In CRT systems, the phosphors on the back of the glass produce colors in response to excitation by the cathode ray. These phosphors have a non-linear response, which is compensated for in the video signal. This correction is referred to as gamma correction. Spatial light modulator systems have a linear response and therefore must remove this correction. The removal of this correction is referred to as degamma. The degamma function typically involves a look-up table (LUT) used to map the incoming gamma-corrected data to a non-gamma data value. An 8-bit source results in 256 codes input to the degamma process and 256 codes output from degamma processing. However, the degamma output codes have precision requiring more than 8-bit values. This is shown in the table below.
[0032] As can be seen by the above table, higher bit-depth resolution is available for lower pixel codes after degamma is performed. Many more codes are available after degamma for lower code levels than for high source codes when the degamma processing output is less than twelve bits. If the number of bits of resolution shown on the spatial light modulator simply matches the number of bit from the source prior to degamma processing, then information is lost for lower codes output from the degamma processing. Higher than 8-bit resolution becomes available for lower significance codes. However, two problems arise.
[0033] First as discussed previously, most spatial light modulators do not switch quickly enough to allow more than 8-10 bits of resolution. Secondly, a problem occurs because of the nature of the PWM process. PWM assigns the most time to the higher significance bits. To avoid artifacts caused by transitions between the larger significance bits to the smaller significance bits, called temporal contouring, this time is broken up into smaller pieces for these bits.
[0034] For example, instead of bit 7 in an 8-bit system being displayed for all 128 of its LSB times in one continuous period, they are divided up into smaller periods in a process referred to as bit splitting. Achieving higher bits of resolution requires more time for loading more bit planes at the expense of the bit-splitting process.
[0035] Unless a method is used to show more bits of resolution than the source was encoded with, significant quantization errors result. The errors appear, for example, in the darker scenes where they collapse into a single light level. These artifacts are referred to as spatial contouring. One advantage of the invention lies in its ability to change the resolution of these types of scenes, avoiding the artifacts and improving image quality.
[0036] Generally, a single device system using color sequencing will show a dramatic improvement from 8 to 12 bits. Multiple device systems, since each device images for one color and therefore has more time, can typically produce 10 bits of resolution. Using 14 bits of resolution for this system improves image quality such that spatial contouring artifacts virtually disappear. Fourteen bits of resolution most closely matches the maximum resolution of the eye.
[0037] As an overview of the invention, the letter N will refer to the number of bits of the desired resolution. For example, in the one device system above, the desired resolution is 12 bits, so N=12. However, the method actually only loads Q bits planes, where Q<N. The method will employ a combination of spatial and temporal multiplexing and will be referred to in some instances as spatial-temporal multiplexing or multiplexing. The N-bit resolution is achieved in some embodiments of the invention by loading Q bit planes and then applying spatial-temporal multiplexing Y bit planes. The process is shown as a flowchart in FIG. 2.
[0038] In step 26 of FIG. 2, the N-bits of resolution desired for display are determined. The decision will depend upon the speed of the system, the switching time of the modulator elements, the number of bit-planes available in memory, among other considerations specific to any given system. For the example to be discussed herein, N will be selected as 12. This is not intended to limit selection for a one device system to 12 bits, but will be used as a specific example for discussion purposes. In step 28, the number of bit planes, weights, and their significance will be established. For purposes of discussion, Q will be set at 9. This means that 9 bit-planes will be loaded onto the modulator.
[0039] Of that 9 bit-planes, a number of the LSBs will be reserved for the spatial-temporal multiplexing process. In this example, the reserved LSBs, Y, will be 2. Therefore, 7 bit planes (9-2) will be unaffected by the spatial-temporal process. The numerical results of these selections are shown in the below table. The Fractional Bits (FBIT) are referenced to the 8-bit LSB of the degamma output per the previous table (8-Bit Fractional Code). This is shown as step 32 in FIG. 2.
[0040] The weights of the STMLSBs (spatial-temporal multiplex LSBs) are 1.14 and 0.75 when referenced against the 8-bit LSBs. The on-times would be 17 microseconds and 11.18 microseconds, respectively. For any given pixel, FBITs less than 0.75 can be produced by turning the values of the STMLSBs on and off over a number of frames. For example, an FBIT value of 0.375 could be attained by displaying the STMLSB 0.75 in every other frame, 0.75/2=0.375.
[0041] This temporal multiplexing of the STMLSBs forms new grayscale codes. For a given pixel, none, either, or both STMLSBs can be used. For example, showing both STMLSBs in one frame and then not in the next produces the code (1.14+0.75)/2=0.945. One problem with using only the temporal multiplexing occurs in smaller FBITs. The lower FBIT values have correspondingly low update rates. For example, FBIT=0.0469 (0.75/16), the update rate is 3.75 Hz. This will cause noticeable flicker each time the STMLSB is used if a large area of the display screen contains FBIT=0.0469, degrading the image quality even though the resolution has been increased.
[0042] To overcome this problem, spatial multiplexing is also employed. Spatial multiplexing takes the form of patterns, shown in step 34 of FIG. 2. Each STMLSB is applied to each pixel out-of-phase in time, on a frame-by-frame basis, relative to neighboring pixels on the modulator. Within any given frame, the STMLSB is evenly dispersed over the screen area for a particular FBIT code during a frame.
[0043] The selection of patterns is virtually unlimited. The patterns that have achieved the best results thus far are shown in FIGS. 3-5. For each pattern, the starting point of the upper left-most active pixel is varied from frame to frame. Thus, the spatial patterns are temporally multiplexed from frame to frame in a complex manner.
[0044] The basic patterns are active pixel densities of 50%, 25% and 12.5%. A 50% pattern is shown in FIG. 3. A pixel having the label ‘CKs’, as opposed to ‘CK’, designates the upper left-most pixel where the pattern starts. Each CK indicates the occurrence of an STMLSB, where all CK in any pattern are the same STMLSB for a given pair of frames.
[0045] This checkerboard pattern is spatially dense, so no spatial artifacts are seen within one frame. For any given pixel, four of its neighbors update in each frame, and half the total screen is updated, preventing the viewer from perceiving any flicker. The checkerboard pattern is spatially in phase with any other active STMLSB. 50% of all pixels in each frame are reserved for this checkerboard, even if no STMLSBs are actively using the checkerboard in that frame. This will be seen in further patterns as the CK pixels.
[0046] FIG. 4 shows a 25% pattern. The P25spixel, the start of the pattern, is randomly assigned each frame, for one active STMLSB. Two limitation are applied to its assignment, it must be a non-CK pixel (non checkerboard) and it must be in column 0, line 0 (C0/L0), CO/L1, C1/L0, or C1/L1. For two active STMLSBs, the lowest weighted one is assigned as above. The highest weighted one is simply placed in the pattern with the opposite spatial phase in each frame provided that it is still out of phase with the checkerboard.
[0047] FIG. 5 shows a 12.5% pattern, which has the same first restriction as the P25spixel. However, the P12spixel can be randomly assigned within the locations of C0/L0, C0/L1, C1/L0, C1/L1, C2/L0, C2/L1, C3/L0, or C3/L1. It also must be in phase with the 25% pattern, regardless of whether the 25% pattern is active or not. The same spatial phase relationships apply for two active STMLSBs as did in the 25% pattern.
[0048] As indicated above, it is possible to combine the patterns for different percentages of the different STMLSBs to achieve several different patterns. In fact, the same overall active pixel percentage can be obtained by combining various patterns as shown by the following table, which uses 1 STMLSB. Use of more then one STMLSB could allow combinations of the patterns to expanse the possible fractional bits.
[0049] The application of these patterns to the previous example is shown in the following table
Spatial-Temporal Muxing with 2 Dedicated STMLSBs
[0050] Referring back to FIG. 2, the above discussion has focused on the spatial pattern development step 34. Once the patterns have been determined, a system designer must decide on how to temporally multiplex the patterns in step 36. The temporal sequencing of spatial patterns can be predetermined, where the spatial start of the spatial multiplexing pattern repeats in a planned sequence over some number of frames. Another temporal multiplexing option allows for varying levels of random number generation to reduce temporal noise artifacts generated by use of this process. Any dithering technique such as those discussed herein will create new noise artifacts. Depending upon the system and modulator, it is typically best to use some degree of randomization of the spatial start point for the spatial multiplexing patterns in actual systems.
[0051] If randomization is used, however, some further things need to be considered. It is believed that no randomization should be used for the checkerboard pattern. The checkerboard pattern, as discussed relative FIG. 3 above, just alternates its start point in each frame. This should generate no temporal artifacts since the light energy is evenly dispersed temporally over both frames and spatially within each frame. At the start of each frame, the start point for any non-checkerboard spatial pattern for a given FBIT is randomly selected.
[0052] The consideration of predetermined temporal sequencing of the starting pixel or random starting pixels apply to all aspects of this invention. This invention applies to any use of PWM in a display system, one device or multiple devices, color sequential or not. The above discussion has centered on a one device, color sequential system. The use of this invention in a multiple device system allows even higher levels of resolution.
[0053] As mentioned previously, multiple device systems typically have more bits of resolution since they have more time per color in each frame when compared to color sequential systems. In the below example, a 10-bit reference number will be used, and the desired resolution will be 14 bits (N=14). In the below discussion, 12 bit planes of data will be used (Q=12), with 3 STMLSBs (Y=3). The STMLSBs do not fit into the normal 2npattern. For example the STMLSBs weights are 0.75, 1.00 and 1.25, with on-times of 10.5 microseconds, 14 microseconds, and 17.5 microseconds, respectively.
[0054] Using 3 STMLSBs has some advantages over using 2. Checkerboards and other symmetrical patterns with a high spatial frequency of active pixels in the pattern generate the least temporal and spatial artifacts. Higher spatial frequency patterns can be used when using 3 STMLSBs instead of 2. Temporal duced because the very symmetrical pattern makes discerning the filling in of empty space from one frame to the next very hard to see. Spatial artifacts are reduced because the density of the patterns prevents any detection of the spatial contours at a normal viewing distance from the screen.
[0055] The table below shows an example of the values used to achieve the FBIT codes for 14 bits perceived resolution. The “Bit” columns show the weights of the bit planes used for each intensity combined with the pattern shown in the corresponding “S-T Paf” columns. These are referenced to a 10-bit fractional FBIT reference.
[0056] At this point it may be advisable to add a step to the process of mapping the degamma function into the STM fractional level space. Using the previous table'values, the degamma function is rounded to the nearest value which can be achieved by the STM fractional levels. An example of this rounding process is given the degamma function value, or reference number, to be represented, (relative to a 10 bit space), is 26.444444, then the upper MSBs (non-STM FBITs), would save the value 26, while the lower STM FBITs would utilize the code 0.4375 (location 7 in the above table). The entire degamma function over it's entire input range would be mapped into the STM space, mapping the reference numbers to the spatial patterns, although the mapping is not exactly 1.1. This allows for the use of non-binary increasing STM fractional levels to be used.
[0057] Referring back to FIG. 2, the remaining step is to load and display the date once all of the various values needed have been determined. This process would not be done all at once. Each new frame would have to have this process, with whichever predetermined values have been decided upon, applied to it. More than likely, this will be done somewhere in the processing flow of the system described with reference to FIG. 1.
[0058] FIGS. 6 and 7 show two different parts of one embodiment of this integration into the processing of the incoming video data. The embodiment shown if for a 3 device system with 3 STMLSBs. The pattern selection is based on the 5 LSBs out of the degamma table for a particular color. The multiplexes shown in FIG. 6, such as multiplexer 42, allow the needed patterns to be formed. For example, if a 75% pattern is needed for STMLSB2, then one multiplexer outputs 50% and the other outputs 25%. The OR gate 44 then combines them and outputs them as 75%.
[0059] In some instances non-symmetrical patterns are needed, such as 7/32% and 9/16%. These are generated using the programmable pattern input at multiplexer 42.
[0060] FIG. 7 shows one embodiment of circuitry to implement this type of pattern generation. The logic block 46 generates the patterns shown as an input to multiplexer 42 in FIG. 6. It received the horizontal sync (HSYNC), the vertical sync (VSYNC) and the active data (ACTDATA) signals that indicate the initiation of a row, a frame or a column, respectively. The random number generator 48 is used to produce the random pattern starting points for logic block 46 discussed above.
[0061] The logic block 46 provides signals such as those labeled 50%, 25%, etc., for multiplexer 42 and its counterparts in FIG. 7, as well as signals for the LUT 50 in FIG. 7. The LUT 50 stores 4×8 repeating patterns with four programmable phases for each pattern, which allows the use of predetermined patterns that are then programmed into the LUT 50. The VSYNC signal initiates a new random number at the start of each frame. In this way both the random start and the predetermined pattern options are enabled.
[0062] FIG. 6 also shows the degamma circuit, which can be a look up table, an adder or any other circuit that can produce 14 bits of output for 10 bits of input. In the previous example of 14 bits, 5 bits would be the LSBs used to generate the patterns, shown entering LUT 40, and 9 bits would pass directly along path 52 to the display device control circuitry not shown. The output of the functions of FIG. 6 is the STMLSBs referenced in the tables for the 3-device example shown above.
[0063] This invention allows greater bit-depth resolution than would otherwise be obtainable on spatial light modulator displays utilizing PWM. The above discussion is in no way intended to limit the systems to which it is applied. The invention can be applied to produce more or fewer FBITs than discussed above. The number of source bits can be other than 8 or 10 bits as discussed above. Similarly, the patterns used are infinite and varied, the example patterns used above are not exclusive.
[0064] Further extensions of the invention could include multiple STMLSBs other than 2 or 3. The number used can range from 1 to the number of bits in the system, restricted only by the capability of the modulator used. Additionally, when multiple STMLSBs are used, the checkerboard pattern for each STMLSB can be out of phase, rather than in phase as discussed above. Finally, the weighting of the STMLSBs can be any value that a particular application or system can support.
[0065] Thus, although there has been described to this point a particular embodiment for a method and structure for a spatially and temporally multiplexing display data to achieve higher bit-depth resolution, it is not intended that such specific references be considered as limitations upon the scope of this invention except in-so-far as set forth in the following claims.
Claims
What is claimed is:
1. A method of display high bit-depth resolution images on a spatial light modulator, comprising the steps of:
a. determining a number of bits of resolution to be displayed;
b. establishing a number of bit planes to be loaded in each frame for each color;
c. reserving a predetermined number of least significant bits (LSBs) of the number of bit planes to achieve the number of bits of resolution to be displayed;
d. defining reference numbers to provide fractional bits;
e. developing a set of spatial patterns to be used with each of said predetermined number of LSBs for each said fractional bits;
f. selecting one of either random spatial placement of the patterns or predetermined placement of the patterns to display said patterns at the start of each frame; and
g. determining one of said patterns for each pixel according to said pixel's intensity.
2. The method of claim 1 wherein said selecting step selects a random placement of the patterns.
3. The method of claim 1 wherein said selecting step selects a predetermined placement of the patterns.
4. The method of claim 1 wherein said method further includes mapping said reference numbers to combinations of said spatial patterns.
5. An apparatus operable to generate patterns for high bit-depth resolution displays, comprising:
a. pattern generation logic operable to receive signals that indicate initiation of pattern generation and to generate spatially and temporally multiplexed patterns;
b. a random number generator operable to provide said pattern generation logic with a random number to indicate the spatial start of said patterns;
c. a programmable look-up table operable to store spatial phases to be selected based upon said patterns received from said pattern generation logic;
d. a circuit operable to generate fractional codes; and
e. logic to select specific ones of said spatially and temporally multiplexed patterns based upon outputs from said pattern generation logic, outputs from said programmable look up table, and said fractional codes.