Synchronizing video content with extrinsic data

[0001] People often want more information about the movies and other video content they are watching. To this end, people may search the Internet to find out more information about the video content. This information may include, for example, biographies of actors, production information, trivia, goofs, and so on.

[0010] The present disclosure relates to associating and synchronizing extrinsic data with video content. Consumers of video content such as movies, television shows, etc. often would like additional information about what they are seeing. The additional information may relate to trivia, mistakes, or further information about what is currently being shown such as actors appearing on screen and/or other information. The additional information may involve biographies, filmographies, pictures, audio clips, video clips, and/or other forms of rich content. The additional information, referred to herein as extrinsic data, may be available in a database that may be accessed through a web site. However, the extrinsic data may not be associated with any specific time in the movie, making it difficult to locate extrinsic data that is relevant to what is currently being shown.

[0011] Various embodiments of the present disclosure facilitate associating extrinsic data items with video content features at specific times. In some embodiments, the associations may be crowd-sourced and then validated through various approaches. Once obtained, extrinsic data items may be displayed in conjunction with the video content feature so that their appearance is synchronized according to the stored, validated extrinsic data-time associations. User interfaces for obtaining the associations and presenting the extrinsic data items may be displayed along with the video content within a single client device. Alternatively, the user interfaces may be rendered in a different client device from a device that is presenting the video content. In the following discussion, a general description of the system and its components is provided, followed by a discussion of the operation of the same.

[0012] With reference to FIG. 1, shown is a networked environment 100 according to various embodiments. The networked environment 100 includes one or more computing devices 103 in data communication with a plurality of clients 106a . . . 106N by way of a network 109. The network 109 includes, for example, the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, etc., or any combination of two or more such networks. The networked environment 100 may also include one or more content display devices 112 such as televisions, video projectors, and so on, which may or may not be coupled to the network 109. In one embodiment, one or more of the clients 106 may be in data communication with the content display device 112 by way of another local network (e.g., Bluetooth, infrared, Wi-Fi, etc.) that may be separate from the network 109.

[0013] The computing device 103 may comprise, for example, a server computer or any other system providing computing capability. Alternatively, a plurality of computing devices 103 may be employed that are arranged, for example, in one or more server banks or computer banks or other arrangements. For example, a plurality of computing devices 103 together may comprise a cloud computing resource, a grid computing resource, and/or any other distributed computing arrangement. Such computing devices 103 may be located in a single installation or may be distributed among many different geographical locations. For purposes of convenience, the computing device 103 is referred to herein in the singular. Even though the computing device 103 is referred to in the singular, it is understood that a plurality of computing devices 103 may be employed in the various arrangements as described above.

[0014] Various applications and/or other functionality may be executed in the computing device 103 according to various embodiments. Also, various data is stored in a data store 115 that is accessible to the computing device 103. The data store 115 may be representative of a plurality of data stores 115 as can be appreciated. The data stored in the data store 115, for example, is associated with the operation of the various applications and/or functional entities described below.

[0015] The components executed on the computing device 103, for example, include a content delivery service 118, an extrinsic data association service 121, a content enhancement service 124, and other applications, services, processes, systems, engines, or functionality not discussed in detail herein. The content delivery service 118 is executed to serve up or stream video content to clients 106 and/or content display devices 112. The content delivery service 118 may support a resume functionality such that playback of video content may be stopped at a point in the video content on one device and later resumed at that point on the same device or a different device.

[0016] The extrinsic data association service 121 is executed to facilitate the association of extrinsic data with particular times within video content. To this end, the extrinsic data association service 121 may generate data for a user interface that presents various options for extrinsic data to a user. From the options, the user may select one or more extrinsic data items to associate with a particular time in the playback of the video content. The extrinsic data association service 121 may be configured to validate the user-supplied association through various approaches as will be described.

[0017] The content enhancement service 124 is executed to present extrinsic data items in synchronization with video content according to stored associations of the extrinsic data with the video content. To this end, the content enhancement service 124 may determine the time in the video content as it is being presented to the user and obtain extrinsic data items associated with the video content relative to that time. The extrinsic data items may be presented on a distinct display, alongside the video content, overlaid on top of the video content, or in some other manner in conjunction with the video content.

[0018] The data stored in the data store 115 includes, for example, a content library 127, user data 130, fingerprint data 133, an extrinsic data library 136, and potentially other data. The content library 127 may include multiple video content features 139 such as movies, television shows, video clips, and/or other forms of video content. It is noted that the content library 127 may be absent in some embodiments as the computing device 103 may maintain the extrinsic data and video content associations and not actually present the content.

[0019] The user data 130 includes various data associated with users of the system. The user data 130 may include one or more watchlists 142 that indicate video content features 139 that the users are interested in watching. The user data 130 may also include one or more resumption times 145, which correspond to times in video content features 139 at which the video content feature 139 was paused or stopped. In one embodiment, the user data 130 may indicate a reputation or feedback score for the user that may be employed in assessing the validity of extrinsic data associations created by the users.

[0020] The fingerprint data 133 may include, for example, audio fingerprinting data and/or video fingerprinting data that may be employed in some embodiments to identify video content features 139 that are being presented to users and/or to identify times, dialogue, or scenes within the video content features 139. The fingerprint data 133 may include script information that may be used to identify times in video content features 139 from subtitles or recognized audio. Although the fingerprint data 133 is depicted as being within the data store 115, in some embodiments portions of the fingerprint data 133 may be stored within the clients 106 to facilitate identification of video content features 139 or times in the video content feature 139.

[0021] The extrinsic data library 136 includes extrinsic data items 148 that are associated with the video content features 139. The extrinsic data items 148 may correspond to metadata about the video content features 139 that is not obtained directly from the video content features 139. In other words, the extrinsic data items 148 may be obtained from an external source, such as a source of curated data. This is in contrast to intrinsic data, which may be obtained from the video or audio of video content features 139. Non-limiting examples of the extrinsic data items 148 may include names or descriptions of performers in the video content features 139, biographies or filmographies of the performers, commentary, trivia, mistakes, user comments, images, and/or other data. The extrinsic data items 148 may include curated data that is professionally managed, verified, or is otherwise trustworthy.

[0022] Stored along with the extrinsic data items 148 may be extrinsic data-time associations 151. The extrinsic data-time associations 151 are user-generated associations between extrinsic data items 148 and specific times in the running of a video content feature 139. In some embodiments, the time may be represented as a frame number, or a range of frame numbers, in the video content feature 139. The extrinsic data-time associations 151 may also indicate an association of extrinsic data items 148 with a specific graphical position on one or more video frames of the video content feature 139.

[0023] As an example, the extrinsic data-time associations 151 may indicate which performers appear on screen at a point fifteen minutes into the running of a video content feature 139. As another example, the extrinsic data-time associations 151 may indicate that a certain goof or mistake is associated with a point 45 minutes and twenty seconds into the running of a video content feature 139. In various embodiments, the user-generated extrinsic data-time associations 151 may undergo a validating procedure before they are considered valid.

[0024] The clients 106 are representative of a plurality of client devices that may be coupled to the network 109. Each client 106 may comprise, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, personal digital assistants, cellular telephones, smartphones, set-top boxes, network-enabled televisions, music players, web pads, tablet computer systems, game consoles, electronic book readers, or other devices with like capability. Each client 106 may include a respective display 154a . . . 154N. The display 154 may comprise, for example, one or more devices such as cathode ray tubes (CRTs), liquid crystal display (LCD) screens, gas plasma-based flat panel displays, LCD projectors, or other types of display devices, etc.

[0025] Each client 106 may be configured to execute various applications such as a browser 157, a respective content access application 160a . . . 160N, a respective content enhancement application 163a . . . 163N, and/or other applications. The browser 157 may be executed in a client 106, for example, to access and render network pages, such as web pages, or other network content served up by the computing device 103 and/or other servers. The content access application 160 is executed to obtain and render for display video content features 139 from the content delivery service 118, other servers, and/or local storage media. The content enhancement application 163 is executed to facilitate user-generation of extrinsic data-time associations 151 as well as to present extrinsic data items 148 in synchronization with the video content features 139.

[0026] In some embodiments, the content access application 160 and/or the content enhancement application 163 may correspond to code that is executed in the browser 157 or plug-ins to the browser 157. In other embodiments, the content access application 160 and/or the content enhancement application 163 may correspond to standalone applications such as mobile applications. In some scenarios, the content access application 160 and the content enhancement application 163 may be executed on different clients 106 for one user. The client 106 may be configured to execute applications beyond those mentioned above such as, for example, mobile applications, email applications, instant message applications, and/or other applications.

[0027] Next, a general description of the operation of the various components of the networked environment 100 is provided. To begin, users at clients 106 access video content features 139 through content display devices 112 or through content access applications 160 executed in clients 106. The video content features 139 accessed through the content display devices 112 may be obtained through multi-channel television, video-on-demand, optical disc, or other media. Where a content access application 160 is employed to access video content features 139, the video content features 139 may be served up by the content delivery service 118, potentially from a watchlist 142 of the user.

[0028] The content enhancement application 163 executed in the client 106 is able to track the progress of the video content feature 139 through communication with the content access application 160, content delivery service 118, or the content display device 112. In some embodiments, the content enhancement application 163 may be equipped to recognize video content features 139 presented in the environment of the user via audio and/or video fingerprinting. The video content features 139 may also be identified through subtitle information, program identifier information, and/or other information available from a content display device 112 or a content access application 160. Ultimately, a time in the video content feature 139 is determined. It is noted that a movie or other video content feature 139 may have multiple versions with different running times, so the specific version of the video content feature 139 may be identified.

[0029] The extrinsic data association service 121 is configured to send user interface data to the content enhancement application 163 in order to present a user interface for selecting from various extrinsic data items 148. When an extrinsic data item 148 is selected, an association between a time in the video content feature 139 and the extrinsic data item 148 may be reported by the content enhancement application 163 to the extrinsic data association service 121. The extrinsic data association service 121 may record the association as an extrinsic data-time association 151 in the data store 115.

[0030] The extrinsic data association service 121 may be configured to validate the user-generated association before it can be presented to other users. In some cases, the validation may depend at least in part on a reputation score associated with the user. In one embodiment, the user-generated association is considered valid if it is not a statistical outlier compared to similar user-generated associations. Under a crowd-sourcing model, many different users may supply similar user-generated associations, which may be used to indicate a valid association if the user-generated associations are generally in agreement. In another embodiment, the user-generated association may be presented to other users to confirm its correctness manually. Whether the user supplies valid associations may affect the reputation score of the user.

[0031] It is noted that the extrinsic data-time associations 151 correspond to relatively high quality data despite being user generated. This is in part because the extrinsic data items 148 may already be correlated or associated with the video content features 139. As a non-limiting example, a user would be unable to specify that “Martin Sheen” is an actor in a scene of the movie “Ferris Bueller's Day Off,” because there is no existing association between “Martin Sheen” and the movie. However, “Charlie Sheen” is an actor in the movie, and the user would be able to specify that “Charlie Sheen” is in a scene in the movie. Accordingly, the extrinsic data association service 121 preempts potential confusion between “Martin Sheen” and “Charlie Sheen” with respect to “Ferris Bueller's Day Off.” Similarly, in another non-limiting example, a user may be unable to specify that “Charlie Chaplin” appears in a movie in 1999 because the extrinsic data library 136 indicates that “Charlie Chaplin” does not appear in any movies after 1967.

[0032] When a user views the video content feature 139, the content enhancement application 163 is configured to render the extrinsic data items 148 that are associated with specific times in the video content feature 139. For example, a balloon, box, bar, etc. may be rendered on top of or in conjunction with the video content feature 139 to show extrinsic data items 148 that are relevant to the current time period in the video content feature 139. In one embodiment, the user may be viewing the video content feature 139 on one device and viewing the extrinsic data items 148 on another device. The content enhancement service 124 is configured to send data that implements rendering of the related extrinsic data items 148 to the content enhancement application 163.

[0033] Although a specific structure may be described herein that defines the server-side roles (e.g., of the content delivery service 118, the extrinsic data association service 121, and the content enhancement service 124) and client-side roles (e.g., of the content access application 160 and the content enhancement application 163), it is understood that various functions may be performed at the server side or the client side. As a non-limiting example, the identity of the video content feature 139 may be determined in the computing device 103 or in the client 106. Additionally, although various embodiments are described with respect to video content which may include audio content, it is understood that the principles of the present disclosure may also be applied to audio content without a corresponding video component.

[0034] Turning now to FIG. 2A, shown is one example of a client 106 rendering a user interface 200 in connection with a content display 203 of a content display device 112 (FIG. 1) in the networked environment 100 (FIG. 1). In this non-limiting example, the client 106 corresponds to a mobile device such as a smartphone or tablet computer, and the content display device 112 corresponds to a theater projector or a television. A user of the client 106 is currently watching a video content feature 139 (FIG. 1) on the content display device 112. Concurrently, the client 106 executes the content enhancement application 163 (FIG. 1) to render additional information about the video content feature 139 embodied in time-relevant extrinsic data items 148 (FIG. 1).

[0035] A title 206 in user interface 200 indicates that the user is currently being presented with a video content feature 139 entitled “World of the Gorillas.” The identity of the video content feature 139 may be ascertained through communication with an application programming interface (API) of the content display device 112 (e.g., through a Bluetooth, infrared, or other network). Alternatively, the identity may be ascertained through video or audio fingerprinting analysis performed by the content enhancement application 163 using the fingerprint data 133 (FIG. 1). To this end, the content enhancement application 163 may capture the audio and/or video of the video content feature 139.

[0036] The user interface 200 may display the current time 209 in the video content feature 139, which in this example is 43 minutes and 10 seconds. A prompt 212 introduces a plurality of selection options 215a, 215b, 215c, and 215d. The prompt 212 in this example asks the question, “who is currently shown in the film?” The selection options 215a, 215b, 215c, and 215d each correspond to one or more extrinsic data items 148 (FIG. 1) that may be associated with the current time 209 in the video content feature 139. In this example, each selection option 215 indicates a respective performer with corresponding character in the video content feature 139. Although the selection options 215 are shown as selectable buttons, the selection options 215 may be selected through various user interface components. In one embodiment, images associated with the respective selection option 215 (e.g., headshots for the performers, etc.) may also be presented.

[0037] Moving on to FIG. 2B, shown is one example of a screen 218a rendered on a display 154 (FIG. 1) of a client 106 (FIG. 1) in the networked environment 100 (FIG. 1). The screen 218a includes the content display 203 from FIG. 2A, with the user interface 200 of FIG. 2A being rendered alongside, in visual association, with the content display. In this example, the content display 203 is rendered by a content access application 160 (FIG. 1), which is executed in the same client 106 as the content enhancement application 163 (FIG. 1) that renders the user interface 200. Accordingly, the content enhancement application 163 is able to render the user interface 200 on top of, or alongside, the video content feature 139 (FIG. 1). In one embodiment, the user selection may be made by selecting a location on the screen 218a. The location on the video frame of the video content feature 139 may be recorded as part of the selection. This may facilitate showing the associated extrinsic data items 148 (FIG. 1) relative to a specific location on the screen 218a.

[0038] With reference to FIG. 2C, shown is another example of a screen 218b rendered on a display 154 (FIG. 1) of a client 106 (FIG. 1) in the networked environment 100 (FIG. 1). Unlike FIGS. 2A and 2C, the screen 218b presents extrinsic data-time associations instead of obtaining them. To this end, rendered extrinsic data items 221a and 221b are shown superimposed on the screen 218b in visual association with the content display 203.

[0039] Rendered extrinsic data item 221a is a trivia item that indicates that the scene that is currently shown was “filmed on location in the Mojave Desert.” Rendered extrinsic data item 221b indicates that the character of “Augustus,” which is played by “Ted Smalls” is currently on the screen. In various embodiments, the rendered extrinsic data items 221 may include selectable components to obtain and/or display further information (e.g., images, biographies, filmographies, etc. for performers). In some embodiments, the user interface 200 (FIG. 2B) may be shown along with rendered extrinsic data items 221.

[0040] Referring next to FIG. 3A, shown is a flowchart that provides one example of the operation of a portion of the content delivery service 118 according to various embodiments. It is understood that the flowchart of FIG. 3A provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of the content delivery service 118 as described herein. As an alternative, the flowchart of FIG. 3A may be viewed as depicting an example method implemented in the computing device 103 (FIG. 1) according to one or more embodiments.

[0041] Beginning with box 303, the content delivery service 118 obtains a request from a client 106 (FIG. 1) to stream or otherwise download a video content feature 139 (FIG. 1). In box 306, the content delivery service 118 determines whether the video content feature 139 was previously stopped or paused by the user or otherwise interrupted. If the video content feature 139 was stopped, paused, or interrupted, the content delivery service 118 moves to box 309 and identifies a time in the video content feature 139 at which streaming was previously stopped. In box 312, the content delivery service 118 establishes a starting time in the video content feature 139 for the user as being the time in the video content feature 139 at which streaming was previously stopped. This time may be obtained from the respective resumption time 145 (FIG. 1) in the user data 130 (FIG. 1). The content delivery service 118 then continues to box 315.

[0042] If the content delivery service 118 instead determines in box 306 that the video content feature 139 was not previously stopped by the user, the content delivery service 118 moves from box 306 to box 318. In box 318, the content delivery service 118 establishes a starting time in the video content feature 139 for the user as being the beginning of the video content feature 139. The content delivery service 118 then proceeds to box 315.

[0043] In box 315, the content delivery service 118 commences streaming of the video content feature 139 to the client 106 at the established starting time in the video content feature 139. In box 321, the content delivery service 118 determines whether the streaming of the video content feature 139 has been stopped or otherwise interrupted before completion. If so, the content delivery service 118 continues to box 324 and records the time at which the streaming was stopped as a resumption time 145 in the user data 130 for the user. Thereafter, the portion of the content delivery service 118 ends. Although the flowchart of FIG. 3A shows the time at which streaming is stopped as being recorded in box 324, the time in the video content feature 139 may be continuously determined and recorded in other embodiments. If streaming has instead completed, the portion of the content delivery service 118 ends.

[0044] Turning now to FIG. 3B, shown is a flowchart that provides one example of the operation of a portion of the extrinsic data association service 121 according to various embodiments. It is understood that the flowchart of FIG. 3B provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of the extrinsic data association service 121 as described herein. As an alternative, the flowchart of FIG. 3B may be viewed as depicting an example method implemented in the computing device 103 (FIG. 1) according to one or more embodiments.

[0045] Beginning with box 327, the extrinsic data association service 121 identifies a video content feature 139 (FIG. 1) that is being presented to a user. Such a video content feature 139 may be presented to the user via a content display device 112 (FIG. 1) or a client 106 (FIG. 1). The extrinsic data association service 121 may identify the video content feature 139 via communication with the content delivery service 118 (FIG. 1), if applicable, or the content enhancement application 163 (FIG. 1) executed in the client 106.

[0046] In some embodiments, the video content feature 139 may be directly identifiable by a unique identifier, title, dialogue, etc. In other embodiments, information such as user location, video content provider, etc. may be employed to narrow down the possibilities to ease identification of video content features 139. As a non-limiting example, knowledge of what movies are showing in a particular movie theater where the user is located (e.g., with location determined by global positioning system (GPS), etc.) may be used in determining the video content feature 139. As another non-limiting example, knowledge of what titles are currently being shown on channels of a cable television provider may be used in determining the video content feature 139.

[0047] In box 330, the extrinsic data association service 121 determines the extrinsic data items 148 (FIG. 1) in the extrinsic data library 136 (FIG. 1) regarding the video content feature 139. The extrinsic data items 148 that are determined may correspond to a subset of the extrinsic data items 148 that are associated with the video content feature 139, for example, a subset of the extrinsic data items 148 that are thought to pertain to a particular time or slice of time in the video content feature 139.

[0048] In box 333, the extrinsic data association service 121 sends data to the client 106 associated with the user that facilitates a user selection from the extrinsic data items 148 that were determined in box 330. In box 336, the extrinsic data association service 121 obtains a selection by the user of one or more of the extrinsic data items 148. In box 339, the extrinsic data association service 121 determines the time and/or video frame number in the video content feature 139 relative to which the user selection was made. To this end, a timecode or frame number may be sent to the extrinsic data association service 121 from the client 106 along with the selection. Alternatively, the extrinsic data association service 121 may communicate with the content delivery service 118 to determine the time associated with a streaming instance of the video content feature 139.

[0049] In box 342, the extrinsic data association service 121 validates the association of the selected extrinsic data items 148 and the time in the video content feature 139. To this end, the extrinsic data association service 121 may compare previous user associations to determine whether the current association is a statistical outlier. Also, the extrinsic data association service 121 may obtain feedback from other users to determine the validity of the current association. Further, the extrinsic data association service 121 may refer to a reputation score associated with the user to evaluate, at least in part, whether the current association is trustworthy.

[0050] Upon determining that the current association is valid, the extrinsic data association service 121 stores the association of the selected extrinsic data items 148 and the time in the video content feature 139 in box 345. The association may be stored as an extrinsic data-time association 151 (FIG. 1). In other embodiments, the extrinsic data-time association 151 may undergo asynchronous post-processing to determine validity. In box 348, the extrinsic data association service 121 determines whether the user makes another selection. If the user makes another selection, the extrinsic data association service 121 returns to box 336 and obtains the next selection. If the user does not make another selection, the portion of the extrinsic data association service 121 ends.

[0051] Moving on to FIG. 3C, shown is a flowchart that provides one example of the operation of a portion of the content enhancement service 124 according to various embodiments. It is understood that the flowchart of FIG. 3C provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of the content enhancement service 124 as described herein. As an alternative, the flowchart of FIG. 3C may be viewed as depicting an example method implemented in the computing device 103 (FIG. 1) according to one or more embodiments.

[0052] Beginning with box 351, the content enhancement service 124 identifies a video content feature 139 (FIG. 1) that is being presented to the user. For example, the content enhancement service 124 may communicate with the content delivery service 118 (FIG. 1) or the content enhancement application 163 (FIG. 1) executed in a client 106 (FIG. 1) to determine the identity of the video content feature 139. In box 354, the content enhancement service 124 determines a current time in the video content feature 139.

[0053] In box 357, the content enhancement service 124 identifies extrinsic data items 148 (FIG. 1) in the extrinsic data library 136 (FIG. 1) that are associated with the video content feature 139 at the current time in the video content feature. The content enhancement service 124 may refer to the extrinsic data-time associations 151 (FIG. 1) to make this determination. Alternatively, the content enhancement service 124 may review the extrinsic data-time associations 151 and identify in advance at which times the extrinsic data items 148 are to be presented relative to the video content feature 139.

[0054] In box 358, the content enhancement service 124 may filter the extrinsic data items 148 that were identified in box 357. For example, it may be distracting or otherwise problematic to display too many of the extrinsic data items 148 at a certain time or during a scene of the video content feature 139. Accordingly, the content enhancement service 124 may be configured to filter the extrinsic data items 148 by relevance, popularity, and/or other criteria. Scores for the extrinsic data items 148 may be crowd-sourced in a similar manner to the extrinsic data-time associations 151 and validations thereof. Popularity, relevance, etc. may be determined explicitly by voting, implicitly by number of confirmations that an association is correct, and so on. In one embodiment, the extrinsic data items 148 are filtered to meet a predetermined threshold for relevance, which excludes the extrinsic data items 148 that do not meet the threshold. In some embodiments, the less relevant extrinsic data items 148 may be presented in an abbreviated or otherwise minimized format.

[0055] In box 360, the content enhancement service 124 sends the extrinsic data items 148 or a description thereof to a client 106 associated with the user. The extrinsic data items 148 are to be rendered by a content enhancement application 163 relative to the current time in the video content feature 139. For example, the content enhancement service 124 may show an extrinsic data item 148 that is associated with a point in time 15 minutes into a video content feature 139 exactly at 15 minutes, at 15 minutes, 30 seconds, or at some other time relative to the point of association so as to maintain a logical synchronization of the extrinsic data with the content.

[0056] In box 363, the content enhancement service 124 determines whether extrinsic data items 148 for another time in the video content feature 139 are to be processed. If so, the content enhancement service 124 returns to box 354 and determines the next time. If another time is not to be processed, the portion of the content enhancement service 124 ends. Although the content enhancement service 124 is described as processing times discretely, it is understood that the process may be continuous in some embodiments or prompted by pre-identified times that are associated with extrinsic data items 148.

[0057] Referring next to FIG. 4A, shown is a flowchart that provides one example of the operation of a portion of the content enhancement application 163 according to various embodiments. In particular, the portion of the content enhancement application 163 relates to user generation of extrinsic data-time associations 151 (FIG. 1). It is understood that the flowchart of FIG. 4A provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the portion of the content enhancement application 163 as described herein. As an alternative, the flowchart of FIG. 4A may be viewed as depicting an example method implemented in the client 106 (FIG. 1) according to one or more embodiments.

[0058] Beginning with box 403, the content enhancement application 163 identifies the video content feature 139 (FIG. 1) that is currently being presented to the user. The video content feature 139 may be presented to the user through a content access application 160 (FIG. 1) executed in the same client 106, a different client 106, or some other content display device 112 (FIG. 1). To this end, the content enhancement application 163 may identify the video content feature 139 via an application programming interface (API) of the content access application 160 or the content display device 112. In some embodiments, the content enhancement application 163 may identify the video content feature 139 in the physical environment surrounding the client 106 using audio fingerprinting or other techniques.

[0059] In box 406, the content enhancement application 163 obtains user interface data from the extrinsic data association service 121 (FIG. 1) over the network 109 (FIG. 1). In box 409, the content enhancement application 163 renders a user interface according to the user interface data. The user interface facilitates a user selection from a group of extrinsic data items 148 (FIG. 1) that are associated with the video content feature 139. For example, the group of extrinsic data items 148 may include all of the performers who appear in the video content feature 139, a selection of trivia items, a selection of goofs, and so on.

[0060] In box 412, the content enhancement application 163 obtains a user selection of one or more of the extrinsic data items 148. For example, the user may select a particular performer, trivia item, goof, etc. that is associated with the current scene or time in the video content feature 139. In box 415, the content enhancement application 163 determines the current time in the video content feature 139. The time may be determined through an API of the content access application 160, content display device 112, or through fingerprinting or another technique.

[0061] In box 418, the content enhancement application 163 reports the user selection and the time in the video content feature 139 that is associated with the user selection to the extrinsic data association service 121 over the network 109. The content enhancement application 163 may also report a graphical position to the extrinsic data association service 121. The graphical position, which may correspond to the user selection, is relative to a frame in the video content feature 139. In box 421, the content enhancement application 163 determines whether another selection is to be made. If another selection is to be made, the content enhancement application 163 returns to box 412 and obtains another selection of an extrinsic data item 148. In some cases, the content enhancement application 163 may obtain additional user interface data to facilitate a selection from a different set of extrinsic data items 148. If no other selection is to be made, the portion of the content enhancement application 163 ends.

[0062] Continuing now to FIG. 4B, shown is a flowchart that provides one example of the operation of another portion of the content enhancement application 163 according to various embodiments. In particular, the other portion of the content enhancement application 163 relates to presentation of extrinsic data items 148 (FIG. 1) at appropriate times in conjunction with a video content feature 139 (FIG. 1). It is understood that the flowchart of FIG. 4B provides merely an example of the many different types of functional arrangements that may be employed to implement the operation of the other portion of the content enhancement application 163 as described herein. As an alternative, the flowchart of FIG. 4B may be viewed as depicting an example method implemented in the client 106 (FIG. 1) according to one or more embodiments.

[0063] Beginning with box 424, the content enhancement application 163 identifies a video content feature 139 that is being presented to the user through a content access application 160 (FIG. 1) or through a content display device 112 (FIG. 1). If the video content feature 139 is being presented through a content access application 160, the content access application 160 may be executed on the same client 106 or a different client 106 from the content enhancement application 163. In box 427, the content enhancement application 163 determines the current time in the video content feature 139.

[0064] In box 430, the content enhancement application 163 obtains extrinsic data items 148 that are associated with the video content feature 139 at the current time from the content enhancement service 124 (FIG. 1) over the network 109 (FIG. 1). In some cases, the content enhancement application 163 may obtain extrinsic data items 148 that are associated with the video content feature 139 at future times as well. In box 433, the content enhancement application 163 presents the extrinsic data items 148 that are associated with the current time (or time period) in the video content feature 139. Where the video content feature 139 is being rendered on the display 154 (FIG. 1) of the same client 106 as the content enhancement application 163, the extrinsic data item 148 may be rendered on top of, as an overlay, alongside, or otherwise in visual association with the rendered video content feature 139.

[0065] In box 436, the content enhancement application 163 determines whether the video content feature 139 has finished. If the video content feature 139 has not finished, the content enhancement application 163 returns to box 427 and determines another current time in the video content feature 139. Otherwise, if the video content feature 139 has finished, the portion of the content enhancement application 163 ends. Although the time comparison is described as being performed discretely for a time or range of time, it is understood that it may be performed continuously, potentially in anticipation of future times in the video content feature 139. Also, it may be that the content enhancement service 124 may push associated extrinsic data items 148 to the content enhancement application 163 in advance for an entire video content feature 139 or a portion of time in the video content feature 139.

[0066] With reference to FIG. 5, shown is a schematic block diagram of the computing device 103 according to an embodiment of the present disclosure. The computing device 103 includes at least one processor circuit, for example, having a processor 503 and a memory 506, both of which are coupled to a local interface 509. To this end, the computing device 103 may comprise, for example, at least one server computer or like device. The local interface 509 may comprise, for example, a data bus with an accompanying address/control bus or other bus structure as can be appreciated.

[0067] Stored in the memory 506 are both data and several components that are executable by the processor 503. In particular, stored in the memory 506 and executable by the processor 503 are the content delivery service 118, the extrinsic data association service 121, the content enhancement service 124, and potentially other applications. Also stored in the memory 506 may be a data store 115 and other data. In addition, an operating system may be stored in the memory 506 and executable by the processor 503.

[0068] It is understood that there may be other applications that are stored in the memory 506 and are executable by the processor 503 as can be appreciated. Where any component discussed herein is implemented in the form of software, any one of a number of programming languages may be employed such as, for example, C, C++, C#, Objective C, Java®, JavaScript®, Perl, PHP, Visual Basic®, Python®, Ruby, Delphi®, Flash®, or other programming languages.

[0069] A number of software components are stored in the memory 506 and are executable by the processor 503. In this respect, the term “executable” means a program file that is in a form that can ultimately be run by the processor 503. Examples of executable programs may be, for example, a compiled program that can be translated into machine code in a format that can be loaded into a random access portion of the memory 506 and run by the processor 503, source code that may be expressed in proper format such as object code that is capable of being loaded into a random access portion of the memory 506 and executed by the processor 503, or source code that may be interpreted by another executable program to generate instructions in a random access portion of the memory 506 to be executed by the processor 503, etc. An executable program may be stored in any portion or component of the memory 506 including, for example, random access memory (RAM), read-only memory (ROM), hard drive, solid-state drive, USB flash drive, memory card, optical disc such as compact disc (CD) or digital versatile disc (DVD), floppy disk, magnetic tape, or other memory components.

[0070] The memory 506 is defined herein as including both volatile and nonvolatile memory and data storage components. Volatile components are those that do not retain data values upon loss of power. Nonvolatile components are those that retain data upon a loss of power. Thus, the memory 506 may comprise, for example, random access memory (RAM), read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via a memory card reader, floppy disks accessed via an associated floppy disk drive, optical discs accessed via an optical disc drive, magnetic tapes accessed via an appropriate tape drive, and/or other memory components, or a combination of any two or more of these memory components. In addition, the RAM may comprise, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM) and other such devices. The ROM may comprise, for example, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other like memory device.

[0071] Also, the processor 503 may represent multiple processors 503 and the memory 506 may represent multiple memories 506 that operate in parallel processing circuits, respectively. In such a case, the local interface 509 may be an appropriate network that facilitates communication between any two of the multiple processors 503, between any processor 503 and any of the memories 506, or between any two of the memories 506, etc. The local interface 509 may comprise additional systems designed to coordinate this communication, including, for example, performing load balancing. The processor 503 may be of electrical or of some other available construction.

[0072] Although the content delivery service 118, the extrinsic data association service 121, the content enhancement service 124, the content access application 160 (FIG. 1), the content enhancement application 163 (FIG. 1), and other various systems described herein may be embodied in software or code executed by general purpose hardware as discussed above, as an alternative the same may also be embodied in dedicated hardware or a combination of software/general purpose hardware and dedicated hardware. If embodied in dedicated hardware, each can be implemented as a circuit or state machine that employs any one of or a combination of a number of technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions upon an application of one or more data signals, application specific integrated circuits having appropriate logic gates, or other components, etc. Such technologies are generally well known by those skilled in the art and, consequently, are not described in detail herein.

[0073] The flowcharts of FIGS. 3A-4B show the functionality and operation of an implementation of portions of the content delivery service 118, the extrinsic data association service 121, the content enhancement service 124, and the content enhancement application 163. If embodied in software, each block may represent a module, segment, or portion of code that comprises program instructions to implement the specified logical function(s). The program instructions may be embodied in the form of source code that comprises human-readable statements written in a programming language or machine code that comprises numerical instructions recognizable by a suitable execution system such as a processor 503 in a computer system or other system. The machine code may be converted from the source code, etc. If embodied in hardware, each block may represent a circuit or a number of interconnected circuits to implement the specified logical function(s).

[0074] Although the flowcharts of FIGS. 3A-4B show a specific order of execution, it is understood that the order of execution may differ from that which is depicted. For example, the order of execution of two or more blocks may be scrambled relative to the order shown. Also, two or more blocks shown in succession in FIGS. 3A-4B may be executed concurrently or with partial concurrence. Further, in some embodiments, one or more of the blocks shown in FIGS. 3A-4B may be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages might be added to the logical flow described herein, for purposes of enhanced utility, accounting, performance measurement, or providing troubleshooting aids, etc. It is understood that all such variations are within the scope of the present disclosure.

[0075] Also, any logic or application described herein, including the content delivery service 118, the extrinsic data association service 121, the content enhancement service 124, the content access application 160, and the content enhancement application 163, that comprises software or code can be embodied in any non-transitory computer-readable medium for use by or in connection with an instruction execution system such as, for example, a processor 503 in a computer system or other system. In this sense, the logic may comprise, for example, statements including instructions and declarations that can be fetched from the computer-readable medium and executed by the instruction execution system. In the context of the present disclosure, a “computer-readable medium” can be any medium that can contain, store, or maintain the logic or application described herein for use by or in connection with the instruction execution system. The computer-readable medium can comprise any one of many physical media such as, for example, magnetic, optical, or semiconductor media. More specific examples of a suitable computer-readable medium would include, but are not limited to, magnetic tapes, magnetic floppy diskettes, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Also, the computer-readable medium may be a random access memory (RAM) including, for example, static random access memory (SRAM) and dynamic random access memory (DRAM), or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other type of memory device.

[0076] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications may be made to the above-described embodiment(s) without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.