Bandwidth effective method for communicating interactive links

[0002] This application is related to our U.S. patent application entitled “Method and Apparatus for Interactive Marking of Dynamically Moving Objects in Video Streams and Animated Sequences”, filed on the same day as the present application and incorporated herein by reference, Ser. No. 09/390,044.

[0003] This application is also related to our U.S. patent application entitled “Method and Apparatus for Recognition of Optimal Intermediate Key Frame for Object Tracking in Video Streams and Animated Sequences”, filed on the same day as the present application and incorporated herein by reference, Ser. No. 09/389,916.

[0004] This invention pertains to video programs containing interactive links.

[0005] It is known in the art to provide video. information containing interactive links; The video information can be in the form of a video movie, television program, or other type of video clip. Embedded within the video is one or more. interactive links. This means that during the video, a user can move a cursor over an object of interest and click on that object, e.g. using a mouse, touch pad, track ball, or other type of device. This activates a link, e.g. to a web page, a piece of software, or other source of information. The way this might work is as follows. Suppose a viewer is watching a video clip of an auto race on a television screen 1, and there are several automobiles 2, 3 visible on screen 1. Suppose the viewer desires further information concerning the driver of one of the automobiles (e.g. automobile 2). He might click on automobile 2 and activate a web page. link that will cause his television to display a web page providing information concerning the car and driver.

[0006] In order for such a system to function, the viewer's television system must receive a video signal corresponding to the program he is watching. The viewer's television system must also receive data identifying the source of information if the viewer clicks on a particular object displayed on the screen. For example, if the link is to a web site page, the viewer's television system must receive the URL (or other addressing means) for identifying which web page should be displayed if the viewer clicks on that object.

[0007] The viewer's television system must also receive data corresponding to the location of the “active area” on the television screen. The active area is that portion of the area of the television screen that the viewer will click on in order to access the desired link. Thus, in the case of the automobile race, the active area corresponding to a particular automobile in the race should be a portion of the screen area where the automobile is shown, e.g. as identified by dotted lines 6. (The position the active area may or may not be visually indicated on television screen 1.)

[0008] There are several ways for the television system to determine where active area 6 should be. One way is to design the television system such that it has pattern recognition capabilities, and can recognize where automobile 2 is located. Unfortunately, this would make television 1 quite expensive. Alternatively, for each frame of the video, one could communicate to the television system where active area 6 is to be located. Unfortunately, this would require a considerable amount of bandwidth, since the size and location for each active area on the screen would have to be communicated to the television system. This would have to be done for each frame of the video. (During the automobile race, automobiles 2 and 3 move about screen 1. Thus, the location and size of the active areas associated with automobiles 2 and 3 must be constantly updated.) Also, there could be many active areas on the screen, each corresponding to a link to a web page. Accordingly, it would be desirable to communicate information concerning the location and/or size of the active area for each frame without having to use a large signal bandwidth.

[0009] A video signal is provided to a video system. In accordance with our invention, the location of one or more active areas associated with the video signal is communicated to the video system for some of the frames of the video signal, but not other frames of the video signal. The video system determines the location of the one or more active areas for the other frames by interpolation. In this way, instead of having to provide data to the video system concerning the active area locations for each frame, such data is provided only for selected frames, and the video system calculates where the active areas for the other frames should be by interpolating, e.g. using either a linear interpolation technique or a spline interpolation technique. Because the source of the video signal need not provide active area information for each frame of the signal, the bandwidth requirements for communicating this information are reduced.

[0010] In one embodiment, not only is the location of the active area communicated in this way, but the size of the active area is communicated in this way as well.

[0011] In accordance with another aspect of the invention, a video signal comprising a set of frames is communicated along with the location of an active area for some of the frames but not others of the frames. The active area corresponds to an object depicted in those frames. The object can be a product, a person, a cartoon character or anything else that can be depicted in a video. As mentioned above, the active area is not communicated for the others of the frames, although the object is depicted in the others of the frames. A receiving system calculates the location of the active area by interpolation based on the location of the location of the active area for said some of the frames. In this way, bandwidth requirements for communicating the location of the active areas is reduced.

[0012] FIG. 1 illustrates a television showing an automobile race, with an active area of the screen highlighted.

[0013] FIG. 2 is a block diagram of a signal transmission system and a reception and display system in accordance with the invention.

[0014] FIG. 3 illustrates the position of the active area along the y axis vs the frame number of a video clip.

[0015] Referring to FIG. 2, a video transmission system 400 comprises a data mass storage device 406 including a portion 420 that stores a video program to be transmitted.

[0016] The video program can be any type of program—sporting event, soap opera, educational program, etc. Also included within mass storage device 406 is a portion 421 that contains annotation information. This annotation information includes a) the location of several active areas associated with various frames of the video program; b) the size of those active areas; and c) the link that those active areas are associated with. (The link can be in the form of a URL.) In other words, the annotation information might indicate that in a particular frame of the video program, there is a first active area located in the upper left portion of the video frame, and that active area has a certain size. If the viewer clicks on that active area, that should activate a link to a particular web page.

[0017] Mass storage device 406 can be any type of mass storage device, e.g. a magnetic disk, magnetic tape, optical or magnetic-optic media, or other data storage structure.

[0018] Information from mass storage device 406 is provided to a data separation circuit 405 that separates out the video signal and provides that video signal to a conventional transmitter 404. (Data separation circuit 405 is typically a server.) A television system 401 includes a receiver 411 for receiving the video signal. While FIG. 2 includes symbols indicating that the video signal is transmitted and received as conventional television radio signals, the video signal can be communicated by other means, e.g. electrical cable, optical cable, microwave, satellite communication, telephone lines, the internet, or other signal communication technique.

[0019] The annotation information (e.g. the location and size of the active area, along with the URL information associated with the active area) is communicated to television system 401 via a communication medium 403, which can be the same as, or different from, the medium by which the video signal is communicated. Thus, communication medium 403 can be conventional radio signals, an electrical cable, optical cable, microwave, satellite communication, telephone lines, the internet, or other signal communication technique:

[0020] Of importance, active area location information is not communicated for every single frame of the video signal. Instead, active area location information is only provided for certain frames of the video signal.

[0021] The annotation information is received by a preprocessing unit 412. Preprocessing unit 412 calculates, via an interpolation technique, the location of each active area of each frame for those frames in which that data was not provided by transmission system 400. In one embodiment, the interpolation technique is a linear interpolation technique. For example, suppose transmission system 400 provides the location for an active area every tenth frame. Also suppose, the active area “moves” about the video screen from frame to frame FIG. 3 illustrates the y coordinate values of the active area for frames 1, 10, 20 and 30, indicated by points 500, 501, 502 and 503, respectively. If a linear interpolation technique is used, it will be assumed that the y coordinate for the active area will be as established by lines 504, 505 and 506. Other interpolation techniques can be used, e.g. techniques that generate a curve instead of a straight line between specific points. For example, a spline interpolation technique can be used. If a spline interpolation technique is used, depending upon the spline calculation parameters, it will be assumed that the y coordinate for the active area will be along curve 507.

[0022] FIG. 3 illustrates the y coordinate of the active area vs. frame number. Similar calculations are performed for the x coordinate of the active area, and the size of the active area. (The size of the active area can increase or decrease, e.g. depending upon whether the object associated with the active area moves toward or away from the camera.)

[0023] Referring again to FIG. 2, it is seen that preprocessing unit 412 provides the video signal to a video display unit 414. Preprocessing unit 412 is coupled to a user interface. User interface 413 permits a user to control certain features of the image displayed on display unit 414, e.g. whether a marker will be shown on the screen indicating the location of the active area. User interface 413also permits a user to click on the various active areas to activate the links associated therewith.

[0024] User interface 413 typically has an appropriate control mechanism such as a trackball, mouse, touchpad or joystick. This interface is used to position a cursor on video display unit 414 so that the viewer knows what object he is clicking on. Preprocessing unit 412 determines whether the cursor has been positioned over an active area. In one embodiment, preprocessing unit 412 comprises a microprocessor system.

[0025] In our above-mentioned patent applications filed concurrently herewith, we describe a method wherein the active areas for selected frames are established, e.g. using a manual annotation system. The active areas are associated with objects depicted in the video frames. During this method, the location, size and shape of each active area is established for some (but not all) of the frames of the video clip. For the other frames of the video clip, the location and size of the active area is established by interpolating between the frames. In one embodiment, the location and size of the active areas for those frames that are manually annotated are communicated to video system 401. In other embodiments, the frames for which active area locations and sizes are communicated are selected on other bases. For example, the location and size of the active area can be communicated to receiving video system 401 for every tenth frame. Alternatively, the location and size of the active areas can be communicated only for the so-called “key frames” of the video clip. (Certain video compression algorithms communicate the entire pixel array only for certain key frames. For other frames, only information corresponding to the change between one frame and the next is communicated. The key frames are those frames for which the entire pixel array is communicated.)

[0026] Referring again to FIG. 2, in one embodiment a workstation 450 can be coupled to video transmission system 400. Workstation 450 comprises a personal computer (“PC”) 451 and a local memory storage device 452 (e.g. a disk) or other non-local storage (e.g. a server) that contains editing software 453. Workstation 450 is used to annotate the video clip. In this example, workstation 450 is coupled Via a communication line 454 to server 405, but many other ways to network workstation 450 to server 405 can be used. In some embodiments, editor software 453 can run on server 405. In other embodiments, the files (both the video and annotation files) may reside on the local memory 452.

[0027] While the invention has been described with respect to specific embodiments, those'skilled in the art will recognize that changes can be made in form and detail without departing from the spirit and scope of the invention. For example, both the annotation information and video signal are shown as being sent as separate signals to receiving system 401, in some embodiments, the annotation information and video signal are communicated as a combined signal. This combined signal can be provided by transmitter 404 via broadcast medium 402. Alternatively, the combined signal can be provided via any of the above-mentioned communications media, e.g. optical or, electrical cable, microwave transmission, satellite transmission, etc. The video display unit 414 can employ any appropriate display technology, e.g. a projection video system, a CRT, LCD or other technology. Accordingly, all such changes come within the invention.