Methods and apparatus for transferring data from a display screen

[0002] 1. Field of the Invention

[0003] The present invention relates generally to the field of data communications. More particularly, the invention relates to methods and apparatus for wirelessly transferring data from a display screen to an external receiving device.

[0004] 2. Background

[0005] A number of techniques have been proposed for transferring data from the display screen of a computer system to an external device such as a personal digital assistant (PDA) or other portable information device. Most typically, the prior art techniques modulate the video signal applied to a cathode ray tube (CRT) display. The modulated video signal is then detected and decoded using a photo detector and microprocessor in a receiving device.

[0006] U.S. Pat. No. 4,999,617 issued to Uemura, et al. discloses a device for reading patterns displayed on a CRT display unit of a television receiver. A bar code pattern is displayed within a small region of the CRT display unit. A photodetector is attached to the surface of the CRT display unit with a suction cup over the region in which the bar code pattern is displayed.

[0007] U.S. Pat. No. 5,488,571 issued to Jacobs, et al. discloses a method and apparatus for downloading information from a CRT display to a portable information device. Binary coded data are formatted into blocks of serial data bits, which are displayed as a pattern of spaced horizontal lines. Essentially the same system is also the subject of U.S. Pat. No. 5,535,147 issued to Jacobs, et al. and U.S. Pat. No. 5,570,297 issued to Brzezinski, et al.

[0008] U.S. Pat. No. 5,231,488 issued to Mohrbacher, et al. discloses a system for displaying and reading patterns displayed on a CRT display unit. A photodetector is attached to the display screen with a suction cup in the manner disclosed by Uemura, et al. The receiving apparatus detects an increase in intensity of a synchronizing scan line and an increase in intensity of a data scan line and then utilizes the time differential between the two events to determine the location of the increasing intensity of the data scan line. This location is determinative of the value of the data scan line.

[0009] U.S. Pat. No. 5,652,602 issued to Fishman, et al. discloses another serial data transmission technique using a CRT display of a computer. The computer is programmed to display sequential frames containing illuminated line segments that represent individual binary data bits. Each bit is encoded as a different line segment length to produce an optical pulse having a duration dependent on the value of the data bit. Thus, a “0” bit is represented by a pulse with a duration relatively longer than that of a pulse representing a “1” bit. Essentially the same system is also the subject of U.S. Pat. No. 5,748,895 issued to Shiff, et al.

[0010] All of the above-mentioned patents are specifically adapted for transferring data from a CRT display. CRT displays use a scanning electron beam to excite the phosphor coating on the face of the screen. As the beam scans across the screen it causes the phosphor coating to emit light. Once the beam has passed by, the phosphor continues to emit light but the brightness quickly decays over time. A photodetector aimed at the face of a CRT display will detect the fast rise of the phosphor luminance as the electron beam scans across the screen. The phosphor luminance will then decay until the next scan of the screen. In the CRT, the beam basically paints the image one dot (pixel) at a time.

[0011] Flat panel displays are rapidly becoming popular alternatives to CRT displays, not only in “notebook” computer systems, but in desktop systems as well. A variety of flat panel displays have been developed and proposed, including LCD, electroluminescent, plasma and others. The flat panel displays in common use today typically comprise a liquid crystal material placed in front of a fluorescent tube. The fluorescent tube is the actual source of the light in these types of LCD panels. Light from the fluorescent tube passes through polarizing filters, color filters and liquid crystal materials before radiating from the front of the screen. The liquid crystal material acts as a shutter to selectively block or pass light from the fluorescent tube, on a pixel by pixel basis.

[0012] The liquid crystal material is relatively slow to respond in comparison to a scanning CRT tube. FIG. 1 illustrates the relative luminance signals detected from CRT and LCD displays. Flat panel displays produce an electrical field across the liquid crystal to keep the crystals aligned to allow light to pass through. These drive signals are produced a line at a time. This means that one entire line on the display is driven at a time, with each pixel in that line being driven either on or off. After driving one line for a period of time, the driving electronics advances to the next line until all lines have been driven. A photodetector aimed at the front of a LCD panel will not detect any significant luminance change due to this scanning. This is because the liquid crystal is slow to respond and designed to hold its on or off state until the next display field. Consequently, data transfer methods adapted for use with CRT displays, such as those disclosed in the prior art patents mentioned above, are unlikely to function properly with an LCD display. At the very least, the data transfer rate will be significantly reduced.

[0013] The two types of flat panel LCD displays most widely used today are passive and active matrix. Both of these types work in a similar manner, but the active matrix provides a brighter picture compared to the passive matrix. Passive matrix panels have a matrix of electrodes which can be selected to drive the liquid crystal for each video pixel. The matrix is driven with a multiplex signal which commonly activates one line of the display at a time. This line scanning provides a periodic electrical bias to keep the liquid crystals oriented to either pass or block light from the fluorescent tube. The liquid crystals can not switch on and off at the line multiplexing rate. In a passive display the crystals take a very long time to transition between ON and OFF states. Passive panels typically require up to 300 msec. to respond (⅓ of a second).

[0014] Passive panels can also be dual scan (DSTN) in which the screen is divided into two separate scanning sections. This means that at any given time, two separate areas of the screen are being driven by the drive electronics.

[0015] Active matrix displays are similar to passive displays except that they have a transistor located at each video pixel. The transistors are selected a line at a time and each transistor in the line is either turned on or off. Note that even though the transistors are selected on a line by line basis, they have a storage element that holds the drive signal on the liquid crystal, even after the line drive signal has passed. This provides a better bias signal for the liquid crystal and allows more light to pass through from the fluorescent tube. This active approach allows for a longer drive potential at each pixel and produces better brightness and display quality compared to the simpler passive matrix display. Active panels typically require up to 50 msec. to respond ({fraction (1/20)} of a second). While this is much faster than passive panels, it is slower than CRTs. CRT displays respond very quickly with frame rates as high as 13 msec. ({fraction (1/75)} of a second).

[0016] Most current approaches to video modulation data transfer use sequential pulsing of the video image to provide a series of binary 1's and 0's. These binary bits are used with framing bits (start and stop bits) to form complete data bytes. Some of the current approaches rely on the scanning CRT image to serialize the data bits by providing a luminance pulse for each data bit This approach will fail when applied to flat panel LCD screens because these screens do not have a scanning luminance response like that found with the CRT.

[0017] Other methods provide a binary bit stream where each bit is produced at the video field rate. For a typical CRT, this provides one binary data bit each 16 msec. (60 fields per second). While this approach is viable for the CRT, it will not work well for flat panel displays. The slow response time of LCD panels mean that only a small number of data bits could be transferred per second. For a passive display, 3 bits would be possible (assuming 300 msec. response time). For active panels, 20 bits could be transferred. Using conventional start and stop bits, a passive panel would then be capable of transmitting 0.3 bytes per second and active panels 2 bytes per second. This is too low a data rate for many applications.

[0018] U.S. Pat. No. 5,347,110 issued to Audebert, et al. discloses a system for transmitting information by the time modulation of a character display area of an LCD or similar display. Binary information is transmitted by the binary modulation of the reflectance of an area of the LCD display. A reader device includes one or more light sources arranged to illuminate the modulated area of the LCD display. A photosensor is used to detect the relative reflectance of the display area.

[0019] The methods of the present invention use luminance or color modulation to transfer data. These methods are compatible with both CRT and flat panel displays and also provide a relatively high data transfer rate. At least a portion of the display screen is modulated to represent a data digit. Various modulation schemes are within the scope of the invention, including intensity (luminance or chroma), color, phase, time interval and others. The modulation employs a plurality of predetermined steps, levels or intervals, each of which represents a corresponding data digit For example, in the case of intensity modulation, by using a plurality of intensity levels greater than 2, it is possible to represent data digits having a numerical base greater than 2. Providing modulation with ten different intensity levels allows each modulation “dwell” to represent a decimal digit (0-9). The modulation is sensed by a photodetector in the external receiving device and converted to the corresponding data digit.

[0020] FIG. 1 illustrates the relative photoresponse for CRT and LCD displays.

[0021] FIG. 2 illustrates luminance modulation for transferring decimal data digits.

[0022] FIG. 3 illustrates luminance modulation with dithering encoding.

[0023] FIG. 4 illustrates luminance modulation with irregular graphic patterns.

[0024] FIG. 5 illustrates bi-color phase modulation.

[0025] FIG. 6 illustrates video bar code modulation.

[0026] FIG. 7 illustrates a self-clocking data encoding scheme for use with the present invention.

[0027] FIG. 8 is a functional block diagram of a receiving device suitable for use with the present invention.

[0028] FIG. 9 illustrates a display screen having a portion thereof for data transfer.

[0029] FIG. 10 is a plot of CRT photoresponse of a prototype system constructed in accordance with the present invention.

[0030] FIG. 11 is a plot of a calibration sequence used in the prototype system.

[0031] In the following description, for purposes of explanation and not limitation, specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods and devices are omitted so as to not obscure the description of the present invention with unnecessary detail.

[0032] A first approach to data modulation of a display screen in accordance with the present invention employs luminance modulation. This method drives the display with varying levels of intensity. Each intensity level can represent an entire data digit. For example, FIG. 2 illustrates a method using 10 shades of gray to represent a decimal digit. The luminance level for each successive decimal digit or luminance “dwell” is generated as fast as the display can accommodate. For an active matrix panel, 20 dwells could be sent in one second. This allows 20 digits of information per second, which is substantially faster than sending binary data.

[0033] The luminance levels are detected by a photodetector in a receiving device. Discrete luminance levels can be generated using several different methods:

[0034] This method drives a spot on the display to one of several discrete shades of gray. The photodetector in the receiving device can detect the discrete levels and convert each level into a single digit value. Reference levels can also be sent periodically in the data stream to establish the black and white (highest/lowest) luminance levels. This allows the photodetector output to be scaled to more accurately detect each discrete gray level. One limitation to this method is that the gray scale response of the display may not be linear. In fact, gray scale levels are greatly affected by the monitor's contrast and brightness controls. These controls can be changed by the user and are not predictable or known constants.

[0035] It is possible to convey various luminance levels by selecting different color combinations. Each color has a luminance component combined with a chroma component. Selecting different colors also selects different luminance levels. For example, dark brown has a low luminance while cyan has a high luminance. Note that what is being detected with this method is luminance—not color. Accurate luminance detection depends on the color response of the display, the monitor contrast, brightness and color settings, and the color response characteristics of the photodetector. Accurate detection using this method typically requires some form of calibration to match the photodetector response to the display color response.

[0036] With reference to FIG. 3, the currently preferred method displays a regular pattern of black and white pixels within a region of the display to produce an average luminance level. This “dithering” average level is created by dividing the entire detection region into a matrix of smaller discrete cells comprising one or more pixels. Each cell is either driven full white or fill black. The ratio of black to white cells determines the overall, average luminance for the detection area. This method eliminates problems with unpredictable gray scale response in the display due to contrast or brightness settings.

[0037] The dithering approach illustrated in FIG. 3 uses a rectangular matrix to obtain an average luminance for a given area. It is also possible to display other graphic patterns or characters which have a distinctive appearance while also presenting an average overall luminance. Some examples are shown in FIG. 4. Each of these have a unique luminance level when the black areas are averaged with the white background. This allows the photodetector to discriminate between unique patterns or characters to convert to a corresponding data value.

[0038] Another method is to use two or more color channels to provide a means of data modulation. For this method two or more photodetectors are used, each responsive to different portions of the color spectrum. For example, separate red and green photodetectors could be used to detect varying shades of each color. Using two channels allows data encoding using the luminance level of each color channel, or the phase between two color signals. Phase modulation works by modulating the color channels at a given rate, but varying the phase relationship between the two channels as shown in FIG. 5.

[0039] To further increase the data density, it is possible to combine modulation of color luminance with color phase. Thus at any given sample interval, three parameters are available: red intensity, green intensity and phase relationship. If eight discrete values of each parameter are used, each sample point can represent 8³values or 1 of 512 discrete numerical values per sample. A disadvantage to this method is the requirement for two color-selective detectors. Also, color response can vary between displays, so some type of color calibration may be required.

[0040] FIG. 6 illustrates another method of data encoding using video bar code modulation. This approach is similar to printed bar codes, but uses a higher density data coding. With this method, a video bar code is displayed across the screen. The user swipes a receiving device across the bar code to read data from the screen. Conventional printed bar codes work by using different spaces between vertical lines. The spacing relationship is translated into binary data bits. Multiple bits are combined to form bytes of data.

[0041] Using a video image, data can be represented using luminance levels or color. This allows higher data density because each “bar” in the video bar code can represent an entire decimal digit instead of just a single binary bit. This increases data density by 8 to 10 times compared to conventional bar codes.

[0042] FIG. 6 illustrates a video bar code using luminance levels. Note that luminance levels can be generated using the same methods as previously described for spot modulation. Each bar represents one of many luminance levels, for example, with 10 luminance levels each bar can represent a digit value of 0 to 9.

[0043] Chromatic luminance modulation was described above as a form of intensity modulation. It is also possible to employ a true color modulation in which specific color hues are used to represent corresponding numerical values. Depending on the range of hues used, an array of two or three separate detectors sensitive to different spectral components, such as by using appropriate filters, is required. A beam splitter may be employed to direct light to the individual detectors of the array in the receiving device.

[0044] Regardless of the method of modulation employed, it is desirable that the data transmission be self-clocking. This means that individual data characters are detected by the receiving device without precise time spacing between characters. This self-clocking approach allows for pauses to occur during the transmission of data characters without creating transmission errors. Pauses can occur in PCs if the operating system performs another task while the transmission is active. For example, multitasking operating systems will commonly write data between memory cache storage and disk drives. This activity can preempt the operation of other software and cause short pauses in the operation of lower level applications. For internet based data transfers, varying delays are also common when moving data between servers and client PCs.

[0045] It is also important to accommodate different data rates depending on the type of display monitor being used. Prior to starting the data transfer, the user can make a selection to indicate the type of display being used. If the display is a CRT, a faster transfer rate may be used (up to 75 digits per second). If an active matrix display is being used the transfer rate will be slower (20 digits per second). While the selection of transfer rate is easily accomplished on the PC side, the receiving device will preferably be compatible with all available transfer rates. Using self clocking data allows the receiving device to receive data at the transmission rate, without the need for a data rate selection on the receiving device itself

[0046] An efficient self-clocking method using a non-binary data encoding is illustrated in FIG. 7. If luminance modulation is used the receiving device can detect each discrete luminance level change as a new digit. The length of time between successive digits is irrelevant. If the same digit value is sent twice in succession, a special “repeat” character can be used to indicate that the last digit value is repeating. As shown in FIG. 7, 11 indicates a repeating digit value. If the data stream contains three successive 4's, the encoded data will be 4-11-4. With this approach a single digit value is never repeated twice in succession. The detector can simply wait for each change in luminance level to indicate a new digit value has been sent. Timing relationships between characters is not significant.

[0047] In contrast to self-clocking methods, another modulation approach is based on the time spacing between changes in intensity level or color. With this approach, only a limited number of intensity levels or colors is required. The number of discrete intensity levels or colors may be as few as two. The time interval between changes in intensity level or color has a number of possible discrete values, each of which corresponds to a numerical value. A significant advantage of this approach is that it is not sensitive to variations in display intensity or color fidelity. However, due to the characteristic response times, this approach is better suited to CRT displays than to LCD displays.

[0048] FIG. 8 is a block diagram of a receiving device 100 suitable for use in connection with the present invention. Light emitted by (or reflected by) a display panel falls on photodetector 102. The output of the photodetector is amplified by amplifier 104 and asserted at the input of the analog-to-digital (A/D) converter 106. The digitized output, in this case comprising an 8-bit word, is presented as an input to microcontroller 108. The operation of microcontroller 108 is controlled by program instructions stored in read only memory (ROM) 110. These instructions govern the conversion of the raw digitized input from A/D converter 106 into a data digit. The data digits are further processed in accordance with the particular functions to be accomplished by receiving device 100. In some applications, receiving device 100 may further communicate the data digits or information derived therefrom to a host device via a wired or wireless interface. A random access memory (RAM) 112 is coupled to microcontroller 108 for use as a scratchpad memory, the use of which may include temporary storage of the data digits received from A/D converter 106 or information derived therefrom. In many applications, receiving device 100 will include a user interface 114 comprising a display and/or various controls, such as function selection buttons and the like. Receiving device 100 may also include a provision to allow for automatic calibration of the analog to digital converter. A peak detector 116 detects the peak white level in the received signal. This level is used to establish the upper range of A/D converter 106. This allows the full range of the A/D converter to be used over the receiver's data detection range.

[0049] Receiving device 100 may be configured in any convenient form. For example, receiving device 100 may have an elongated cylindrical shape similar to a pen or a wand. In such case, photodetector 102 may be conveniently located at one end of the device. However, it has been found that pen- or wand-shaped devices have disadvantages when used with LCD flat screen displays. If the device is pressed against the display, even with light pressure, the display may be distorted, thereby affecting the accuracy of the data transfer. For flat panel displays, a flat, card-shaped receiving device is preferred. Such a device may be held against the display screen without distorting the display.

[0050] To ensure proper registration of the receiving device with the display screen, a visual indication of the area of the display screen that will contain the data modulation is preferably provided. As shown in FIG. 9, a rectangular area of the display screen, generally corresponding in size and shape to the card-shaped receiving device, may be configured as a window and may be labeled with a legend such as “place card here”.

[0051] In the examples discussed above, a single photodetector (or paired detectors in the case of bi-color modulation) is used in combination with a single modulated region of the display screen. It will be appreciated that the data transfer rate can be multiplied by employing a suitable array of photodetectors in combination with a corresponding array of data transmission regions. Obviously, the array of detectors must be properly registered with the array of modulated regions on the display. This can present a challenge in the case of a hand held receiving device. One solution to this challenge is illustrated in FIG. 9. Here, the display is divided into four quadrants that are independently modulated. The receiving device includes an array of four independent photodetectors. By providing simple registration marks on both the display screen and the receiving device, the receiving device can be held against the display screen so that the photodetectors are in proper registration with the corresponding quadrants.

[0052] A prototype system has been constructed. The prototype receiving device is configured as a card having the same length and width as a standard credit card. A 9 mm round photodetector element is located in the center of one face of the card. Electronic circuitry within the card amplifies the output signal of the photodetector, which is then applied as an analog input to a conventional personal computer system where A/D conversion is performed. The photodetector element is designed to detect the average luminance over a 9 mm round area of the display screen. The detector consists of a translucent glass window and a photo-Darlington transistor photodetector mounted in a plastic enclosure.

[0053] The prototype system employs luminance modulation using the dithering approach discussed above. A total of 12 luminance levels are used to represent ten decimal values plus two additional values to indicate formatting and repeating characters. Using a CRT display, the prototype system has achieved data transfer rates of 20 characters per second.

[0054] The photodetector in the receiver detects the luminance change as the electron beam in the CRT passes over the detector. This screen phosphor glows with a brightness related to the average screen luminance. For a CRT display, the beam is constantly scanning the screen. This creates a pulse as the beam passes over the detector. Therefore, the signal detected is a pulse which repeats at the frame rate of the display (typically 13 to 17 msec. per field). FIG. 10 is an actual capture of the signal received by the detector using a CRT-based display.

[0055] Since the received signal is a pulse, a software algorithm processes the A/D conversion readings in order to establish the luminance level represented by the peaks of the detected pulses. The software algorithm is then capable of decoding these levels back into packets of data.

[0056] It is desirable for the system to automatically adapt to varying intensity levels on the display. Different luminance levels will result due to variances in the brightness response of the display, the sensitivity characteristics of the photodetector and also due to adjustment of the brightness and contrast settings of the monitor.

[0057] To automatically adjust for these differences, the system provides a calibration sequence at the start of each data transmission. As illustrated in FIG. 11, the calibration pattern consists of a staircase of each of the 12 luminance levels used. A full white pulse (level 12) is sent at the start of the sequence, followed by values of 0 to 12. This signal is detected by the receiver and used to establish the actual 12 discrete levels obtained from the monitor. In FIG. 11, the bottom signal is the actual pulse waveform received by the photodetector. The top signal is that obtained after processing by a software algorithm.

[0058] It will be recognized that the above described invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the disclosure. Thus, it is understood that the invention is not to be limited by the foregoing illustrative details, but rather is to be defied by the appended claims.