Intelligent display screen which interactively selects content to be displayed based on surroundings

[0001] Miniature cameras have become ubiquitous parts of many different items, including cellular telephones and laptops. Structure for making the cameras can be easily fabricated as part of any electronic device.

[0002] Image processing devices can be formed within the same circuitry that is used to carry out other functions such as displays.

[0003] The present application describes a camera that is integrated with a display device. The camera obtains an image indicative of its surroundings, and determines characteristics of a person or thing that is located in the viewing area of the screen. An embodiment automatically changes the screen content based on those automatically detected characteristics.

[0006] An embodiment is shown in FIG. 1. A display screen 100 displays information which can be viewed within an area 101: the viewing area of the display screen. Any person in area 101 can view the contents of the display screen.

[0007] The display may be based on stored information, here shown as a media storage unit 110. The display screen 100 includes a camera portion 120 associated therewith, e.g., located at or near a corner of the screen, or in some area where it will not interfere with the display. The camera 120 may also be located off the screen for example. The camera may image the entire viewing area 101. A lens 121 may be used to adjust the incoming radiation. Alternatively, the image processor may determine electronically, the portion of the image that corresponds to the viewing area.

[0008] The output of the camera 120 is coupled to an image processor 130, which can be a processor that operates according to the characteristics described herein. Image processor 130 can be a general purpose processor that makes a selection from the media storage 110 to form an output for display, which is coupled through a video driver 135, to drive the display on the screen 100.

[0009] The media storage 110 may be any storage unit that stores a number of different media items. This can be a computer, a digital video recorder, a connection to a network based server, a media server, or any other item that stores media items. The media storage can hence be any item that stores media and can produce an output indicative of the stored media based on a selection.

[0010] The image processor 130 may process the image received from camera 120 to determine characteristics of a person or thing that is located in the viewing area of the screen. The specific image or media is selected for display based on the specific characteristics of the thing in the viewing area. This can use any known machine recognition technique, e.g., neural networks, to recognize the content within the scene sensed by the camera 120. It can also use a simpler technique of correlating with a kernel. In the kernel correlation embodiment, a “kernel” image is a small image of a searched-for feature, e.g., a generic face of 50 years old, or some part of that face. The part may be a skin texture, the way the eyes are set, or other age-indicating feature. The kernel is compared with each area in the image, and a distance between the kernel and the areas in the image are determined. The “distance” may be, for example, a least mean squared distance between the kernel and the image. The lowest distance between the image and each kernel is stored. The kernel with the lowest score is taken as best representing the item in the image, and is taken as the winner.

[0011] More sophisticated techniques can alternatively be used.

[0012] Processor 130 may operate according to the flowchart of FIG. 2. At 200, an image is obtained. The pointing of the camera ensures that the image or video obtained by the camera represents the object in the viewing area of the screen.

[0013] A first embodiment, described herein, analyzes the image to look for a specific criteria, which may be part of any item within the image. Alternative embodiments may analyze multiple items—for example, it may analyze multiple items within the image, e.g., multiple faces. An embodiment establishes a hierarchy of criteria, and the first item that matches the highest ranked criteria wins. For example, “children's” may be ranked higher than “parents”, because parents are often happy to watch items that are interesting to the children, while the reverse is not necessarily true. Also, a store that specializes in teen girls may display teen girls items whenever any teen girl is present, no matter what else is present. Another embodiment establishes group criteria, e.g., a boy and a girl together receive different content than either a boy, or a girl, individually.

[0014] At 210, the thus-obtained image is compared to a criteria. Different image criteria can be chosen depending on the applications. Many different applications are described herein, and each of these applications may use a different criteria. For example, when the display is displaying an advertisement, the criteria being used may be demographics of the person(s) who are watching the advertisement. In this case, the screen may be changed much more often, for example every minute. A simple example of a demographic bases the change in display on whether the person(s) viewing the screen is male or female or mixed (male and female), and selects a video or image from the media storage, tailored to a male or female. Another alternative is that the system can use automated techniques to determine a probable age range of the person(s) near the side. For example, techniques may be used to determine whether the watcher is a child, a teenager, someone who is middle-aged, or the like. Other criteria may sense colors associated with the person wearing, such as their hair color, clothing color, shoe color, automobile color, or other. The criteria may be stored in the media storing unit associated with the video or display information that is stored. For example, the criteria may be age, sex, color, or any other criteria discussed herein. A priority may be associated with the criteria, so that higher priority criteria take precedent over lower priority criteria.

[0015] In addition to estimating age, sex and color of clothing, the criterion may estimate the brand or style of clothes that they are wearing. Brand names on the clothes may be guessed based on characteristics like looking for labels, styles, colors or the like. At 220, according to a pre-stored set of rules, the content to be displayed is loaded from the media storage, and is displayed at 230. As an example, the detection of a teenage girl may play an advertisement for teen girl clothing, or a scene with teen girls or that appeals to teen girls. As another example, detection of a teen girl wearing red causes the red color scheme to be used, where at least one item of clothing shown on the screen is similarly red.

[0016] The displayed item is automatically selected from the media storage based on the automatic determination of characteristics of item(s) in the viewing area of the screen. However, it may be disconcerting if the image(s) or media changes too often. Accordingly, the obtaining 200 in FIG. 2 may be carried out at intervals that are appropriate for changing the viewing. For example, the viewing may be carried out every 1, 2, 3, 4 or 5 minutes, or every 15 minutes, every hour, or once a day.

[0017] The first played part of a many different items from the media storage may be cached locally, to reduce lag time. For example, the first 10 seconds of the 8 most commonly played videos may be cached locally, giving the media storage and any network connection 10 seconds to respond to any request.

[0018] Another embodiment may sense changes in the obtained image greater than a certain amount and automatically change the image when the image so changes. When the image changes only a little, or the content of the image moves around without a drastic change, the display might stay the same. However, when the image changes completely, a new analysis and display is made.

[0019] Different techniques can be used for different applications. For example, in an advertising technique, the media storage 110 may store eight different advertisements: e.g, for a young child male; young child female; teenager male, teenager female; parent age male; parent age females; 50+male, 50+female. Each of these different people may receive a different kind of display. The display may be advertising or may be an image or a video, e.g., one with embedded advertising or suggestive advertising. The advertising may all be for the same advertiser, or up to eight different advertisers (for this embodiment) may each share the same display to get their advertisements displayed to every person of the specified demographic, who comes near the screen.

[0020] This system can be used anywhere where any media of any type might be displayed; for example in an elevator, gas station, bathroom, taxi, or a billboard.

[0021] In the gas station environment, the sensing changes embodiment may use the camera to analyze the photo once every time a person comes up to the gas pump or when a car pulls up. The camera may also look for other information, such as the color of the car, the kinds of stickers on the car, the make of their car, and other information. Any of these things can be used to select the proper display content. For example, certain kinds of cars such as muscle cars may be highly correlated with people who are sports fans. Pulling up to a gas tank with a muscle car would indicate that it might make sense to provide a sports type advertising to the person at the gas pump. The video display on the gas pump may display an informational video or an advertisement, based on the kind of person who is pulling up.

[0022] In the bathroom or waiting room environment, a display might display a newspaper section tailored to the likely interests of the person close to the screen. This could be displayed on a wall, where it can be read by people in the room.

[0023] In any of the embodiments, a woman wearing Gucci clothes might receive a Gucci ad or other ad for designer clothing. A woman wearing pastel clothes might receive advertisements for more casual type clothing, or other clothes in pastel colors. A blonde haired woman might get advertisements for clothing that is styled or colored to look good with blonde hair.

[0024] In another embodiment, the system can be used in a cell phone, either to provide ads, to provide appropriate news, or for decoration. According to a decoration embodiment, the camera senses once per two hours, a color scheme of its surroundings, for example, what color the owner is wearing. Based on the color of the owner is wearing, the camera changes the color screen scheme on the display.

[0025] The above embodiment may use the demographics to display informative information instead of advertisements. For example, teenagers might be interested in different kinds of information then 50 plus or 70 plus people.

[0026] This can also be used for example in a waiting room, to decide what kinds of videos to play.

[0027] A television can also use the system, and in one embodiment, an autoplay television is described. The autoplay television automatically selects programs to play based on the audience, who is near the television. This can be used, for example, with a digital video recorder, such as a TIVO™. These devices often record “tivo picks”, that is, programs that the system has automatically postulated as being appropriate for the kind of watching done by that tivo. In this embodiment, one of those programs may be selected for play, based on characteristics of the person near the television. In this embodiment, the video recorder is the media storage.

[0028] This can also be used in a computer, and can be used on a video wall. Any device that is playing images or videos can use this system.

[0029] The embodiment can use rules which relate image content to display content. For example, the rules may specify that if the image has a teenage girl therein, then video clip 1 should be selected, else_if the image has a teenage boy therein, video clip 2 is selected.

[0030] The above has described how the use of a camera can automatically select the content that is displayed on a display screen. This produces the advantage that the display screen becomes interactive and better matches the display to the likely interests of the watcher. This makes the displayed information more likely to be relevant to the watcher, something that could not be expected based on the prior art.

[0031] Another aspect operates to detect faces within the acquired image, and superimposes those faces on corresponding faces within the media served by the media storage unit 110.

[0032] The general structure and techniques, and more specific embodiments which can be used to effect different ways of carrying out the more general goals are described herein.

[0033] Although only a few embodiments have been disclosed in detail above, other embodiments are possible and the inventor intends these to be encompassed within this specification. The specification describes specific examples to accomplish a more general goal that may be accomplished in another way. This disclosure is intended to be exemplary, and the claims are intended to cover any modification or alternative which might be predictable to a person having ordinary skill in the art. For example, other display systems can be used herein, and other criteria can be used to select items from the media storage to be displayed.

[0034] Also, the inventor(s) intend that only those claims which use the words “means for” are intended to be interpreted under 35 USC 112, sixth paragraph. Moreover, no limitations from the specification are intended to be read into any claims, unless those limitations are expressly included in the claims. The computers described herein may be any kind of computer, either general purpose, or some specific purpose computer such as a workstation. The computer may be an Intel (e.g., Pentium or Core 2 duo) or AMD based computer, running Windows XP or Linux, or may be a Macintosh computer. The computer may also be a handheld computer, such as a PDA, cellphone, or laptop.

[0035] The programs may be written in C or Python, or Java, Brew or any other programming language. The programs may be resident on a storage medium, e.g., magnetic or optical, e.g. the computer hard drive, a removable disk or media such as a memory stick or SD media, wired or wireless network based or Bluetooth based Network Attached Storage (NAS), or other removable medium or other removable medium. The programs may also be run over a network, for example, with a server or other machine sending signals to the local machine, which allows the local machine to carry out the operations described herein.

[0036] The term “image” is used herein to represent a still image, e.g., a single image, or a single frame of a multiple frame video.

[0037] Where a specific numerical value is mentioned herein, it should be considered that the value may be increased or decreased by 20%, while still staying within the teachings of the present application, unless some different range is specifically mentioned. Where a specified logical sense is used, the opposite logical sense is also intended to be encompassed.