Scene identification

[0001] Increases in network speeds combined with the benefits associated with viewing content from one's own home have resulted in the growing popularity of watching video content over a network. For example, a user may watch a movie at home without having to drive to a video rental kiosk. In addition, the user may avoid being inconvenienced by having to watch the movie at a pre-scheduled time (e.g., during a television broadcast or according to play times at a movie theater or using pay-per-view on cable). The ability to watch video content over a network at a desired time may provide flexibility to the user's schedule.

[0002] A user may also select from a wide variety of video content based on individual preference and/or mood when viewing video over a network. For example, the user may be in the mood to watch a horror movie, and therefore may select from a list of horror movies without having to watch an already scheduled program. The user may also view the latest episode of a favorite television series. In addition, the ability to watch video content on a wide variety of devices (e.g., desktop computers, laptop computers, televisions, cell phones, gaming systems, tablet computers) may provide the additional convenience of watching the video content in a variety of places (e.g., a coffee place, bookstore, or mall). When watching video content over the network, the user may desire to skip certain parts of the video (e.g., a boring scene in a movie) and jump to subsequent scenes in the video content. In addition, the user may desire to replay a previously watched scene in the video content.

[0014] This technology relates to identifying and providing scenes for video content obtained over a network. A user may watch video content (e.g., movies, television programs, live broadcasts) on a device connected to the network. The video content may be live or pre-recorded. The video content may be streamed over a network, transmitted in burst mode, partially downloaded, or obtained via any other suitable mechanism for transferring video content over a network. When watching video content over the network, users may identify a scene in the video content that the user has viewed in the video content to enable replaying of the identified scene from the video content.

[0015] The video content may be provided to one or more devices over the network. For example, the video content may be provided simultaneously to a first screen device (e.g., a television) and a second screen device (e.g., a tablet computer). Video content provided to the first screen device may be different than the video content provided to the second screen device. The user may replay scenes on the second screen device that were previously played on the first screen device. For example, the user may be streaming a live tennis match on the first screen device. The tennis match may include an interest event (i.e., an event that the user wishes to watch again). In this example, the interest event may be a spectacular shot made by a tennis player. An option to replay the scene with the interest event may be displayed on the second screen device. The user may replay the scene on the second screen device, while the live tennis match is being played on the first screen device. The first screen device and/or the second screen device may store a predetermined amount of video (e.g., the first 5 seconds of each scene) in a cache so that the video may be played with minimal delay. Further, the cache may store the first five seconds of each scene of the video that was previously watched by the user.

[0016] In one configuration, the scenes displayed to the user to be replayed may be identified automatically based on user actions. For example, while watching video content, the user may perform a variety of actions (e.g., reversing the video content, forwarding the video content, replaying a portion of the video content, pausing the video content, etc.) Based on the user's actions, interest events from the video content may be inferred, and a scene including the inferred interest event may be provided to the user. As an example, while watching the video content, the user may reverse a portion of the video content (e.g., the previous 40 seconds) and replay that portion of the video content. An interest event may be inferred to have occurred during those 40 seconds of video content because the user desired to watch that segment again. Therefore, based on the user's action of reversing the video content, a scene of at least 40 seconds in length and containing the inferred interest event may be automatically created and provided to the user for replay. In some configurations, an interest event may be identified in response to a group of users performing a similar action on the same content.

[0017] When providing a replay of a scene from the live video content to the device, various parameters of the scene (e.g., a starting time code) may be determined. For example, the time parameters of the scene may be determined from interest event indicators that occur during a live video broadcast. Interest event indicators may be created by a user who is viewing the broadcast, or the interest event indicators may be created automatically using machine identification. The interest event indicators may mark at what time point an interest event has occurred. The occurrence of an interest event indicator may automatically result in an interest event being identified and tagged. For example, the interest event indicators may include a score change (e.g., from a touchdown, a goal, etc.) in a game of football, soccer, basketball, tennis, etc. In addition, the interest event indicators may include increased cheering from the crowd (e.g., in a boxing match after a knockout punch). Once an interest event is identified and tagged, a scene with the interest event may be defined.

[0018] In one configuration, the scene may begin a certain time period before and end a certain period after the interest event occurs to form a time window. For example, the scene may begin 20 seconds before a touchdown is scored and end 30 seconds after the touchdown is scored. The time period may vary depending on the type of video content being provided (e.g., the time period may be 15 seconds for a basketball game, but 30 seconds for a football game). Once the scene is defined, an option to replay the scene may be displayed on the first screen device and/or the second screen device. The scene may be replayed in slow motion or at a normal speed.

[0019] The parameters of each scene (e.g., the length of each scene) may be identified using heuristics. For example, the length of each scene may be inferred based on the type of video content (e.g., television show, sporting event, political event, etc.). The length of each scene may be heuristically determined based on the overall length and structure for that particular show. In addition, the occurrence of a “fade to black” (e.g., when the screen briefly becomes black to indicate that a new scene is starting) in video content may be used to help determine the parameters of the scene.

[0020] When watching video content over a network, users may identify or mark selected portions of video content. For example, the user may identify an interesting portion of video content (e.g., an exciting play, a thought-provoking statement). By identifying the portion of video content, the user may create a scene with the selected portion of video content. The user may then replay the scene on a first screen device or a second screen device. The scene may include the interest event, as well as a time interval before and after the interest event (e.g., 20 seconds before the interest event begins and 20 seconds after the interest event finishes). The time interval may be referred to as a time window surrounding the interest event. The user may be able to customize the time window based on personal preference by accessing a menu on the second screen device. The user may then select the scene to be replayed using the second screen device. When the user identifies or marks an interest event in the video content, a new starting time code associated with the interest event may be created. Furthermore, the new time code may be embedded into video content metadata that describes the video content. Thus, when the user wishes to replay the scene, the video content may be provided to the device starting from the new starting time code embedded into the video content metadata.

[0021] In addition, the user may share the scene created by the user with a group of users, thus enabling the group of users to view the scene on the group's first or second display devices. A list of scenes based on feedback from a group of users (i.e., group sourcing) may also be displayed on the first screen device or the second screen device. The scenes may be ranked or categorized based on viewings, voting, and/or ratings from the group of users. Furthermore, the user may be able to watch a highlight video containing a plurality of scenes that were recommended by the group of users (e.g., a highlight reel from a basketball game). The highlight video may contain a plurality of related scenes or various different scenes from a group of related movies, television shows, etc. The user may select a scene from the highlight video in order to play a full video containing that scene from the beginning. The group of users may share scenes of live video content, as well as scenes in pre-recorded video content, such as movies and television programs. When a user shares a scene, a link with a starting time code of the scene may be sent to the group of users. When the user wishes to watch the scene, the scene may begin playing at the starting time code as found in video content metadata created by the sender. The scenes that are shared by the group of users may be displayed on the second screen device, while video content is being played on the first screen device. For example, a user may be watching a basketball game on the first screen device. While on the second screen device, a list of shared scenes from the basketball game may be displayed on the second screen device for replay.

[0022] FIG. 1 is an illustration that depicts a system 100 for scene identification in video content according to various examples of the present technology. A first screen device 110 may be coupled to a second screen device 130 over a network 120. Alternatively, the first screen device 110 may be paired directly to a video playback console which is capable of displaying video on the first screen device.

[0023] The first screen device 110 may be playing video content (e.g., a basketball game). A time code associated with the video content (e.g., 6:30:15 PM) may also be displayed, and the time code may include the current hour, minute, and second of the video content. The second screen device 130 may include a user interface with options to navigate the video content (e.g., reverse, pause, stop, play, forward). The user may also navigate through the video content by sliding to a previous or subsequent time point on a time bar. In addition, the user interface may include options to adjust the volume, zoom level, panning, and closed captioning. A scene with an interest event may be identified based on user actions. In particular, a user may perform actions using the user interface of the second screen device 130. For example, while watching a basketball game, the user may reverse the video content from a time code of 1:30:57 to 1:30:15. As a result, a scene may be automatically identified between the time codes of 1:30:15 and 1:30:57. In other words, the user action of reversing the video content may infer that an interest event occurred during that time period. As a result, a scene with that interest event may be automatically created and provided to the user for replay. The user interface may include an option for the user to view scenes 135, which may include scenes that were automatically created based on the user actions. In addition, the user interface may include scenes that have been inferred based on group sourcing.

[0024] In the following discussion, a general description of an example system and the system's components are provided, followed by a discussion of the operation of the system for providing the technology. FIG. 2 shows a networked environment 200 according to various examples of the present technology. The networked environment 200 may include one or more computing devices 210 in data communication with a first screen client 280 and a second screen client 290 by way of a network 275. The network 275 may include the Internet, intranets, extranets, wide area networks (WANs), local area networks (LANs), wired networks, wireless networks, or other suitable networks, etc., or any combination of two or more such networks.

[0025] The computing device 210 may comprise, for example, a server computer or any other system providing computing capability. Alternatively, a plurality of computing devices 210 may be employed that are arranged, for example, in one or more server banks, computer banks or other computing arrangements. For example, a plurality of computing devices 210 together may comprise a cloud computing resource, virtualization server, a grid computing resource, and/or any other distributed computing arrangement. Such computing devices 210 may be located in a single installation or may be distributed among many different geographical locations. For purposes of convenience, the computing device 210 is referred to herein in the singular. Even though the computing device 210 is referred to in the singular, it is understood that a plurality of computing devices 210 may be employed in the various arrangements as described above.

[0026] The first screen client 280 and the second screen client 290 are representative of a plurality of client devices that may be coupled to the network 275. The first and second screen clients 280 and 290 may comprise, for example, a processor-based system such as a computer system. Such a computer system may be embodied in the form of a desktop computer, a laptop computer, personal digital assistants, cellular telephones, smartphones, set-top boxes, network-enabled televisions, music players, tablet computer systems, game consoles, electronic book readers, or other devices with like capability. The first screen client 280 (e.g., a television) may have lessor mobility and be larger in size as compared with the second screen client 290 (e.g., a mobile phone, a tablet computer).

[0027] The first screen client 280 may be configured to execute various applications such as a browser 282, and/or other applications 284. The applications 284 may correspond to code that is executed in the browser 282. The applications 284 may also correspond to standalone applications, such as mobile applications. In addition, the first screen client 280 may be configured to execute applications 284 that include, but are not limited to, video playback applications, standalone applications, email applications, instant message applications, and/or other applications. The applications 284 may play video content on the first screen client 280.

[0028] The first screen client 280 may include or be coupled to a respective display 286, as well as the browser 282. The browser 282 may be executed on the first screen client 280, for example, to access and render network pages, such as network pages (e.g. web pages), or other network content served up by the computing device 210 and/or other servers. The display 286 may comprise, for example, one or more devices such as cathode ray tubes (CRTs), liquid crystal display (LCD) screens, gas plasma-based flat panel displays, LCD projectors, or other types of display devices, etc.

[0029] The second screen client 290 may include a play module 292, a display module 294, and an event identification module 296. The play module 292 may be programmed to play video content 222 received from the computing device 210. The play module 292 may play live video content (e.g., live sporting events) or pre-recorded video content (e.g., movies, television shows). The play module 292 may replay a previously played scene (e.g., an exciting soccer goal). In other words, the play module 292 may replay a scene containing an interest event. The play module 292 may replay the video scene or identified video segment in response to receiving instructions from a user. The play module 292 may play video scenes from the overall video content 222 that is simultaneously being played on the first screen client 280. Alternatively, the play module 292 may play video content 222 that is different than the video content being played on the first screen client 280. The play module 292 may also play a highlight video containing a plurality of scenes with interest events.

[0030] The display module 294 may be programmed to display a list of scenes for replay. The display module 294 may display the list of scenes from live video content while the live video content is being played on the first screen client 280. The second screen client 290 may possess a smaller screen and greater mobility as compared with the first screen device 280. The display module 294 may display a video preview and/or an image preview for each scene in the list of scenes. As an additional preview mechanism, the display module 294 may display a sequence of images from the scene in response to moving a cursor over a scene on the display or by touching the scene of a touch screen device. The display module 294 may display scene information associated with the scene (e.g., date, length, number of views, ratings, etc.). The display module 294 may include options to play the scene, preview the scene, replay the scene, share the scene, and/or view user comments associated with the scene. Scene lists may be displayed on the display module 294 based on popularity, number of views, ratings, and/or by other metrics.

[0031] Various applications and/or other functionality may be executed in the computing device 210 according to various embodiments. Also, various data may be stored in a data store 220 that is accessible to the computing device 210. The term “data store” may refer to any device or combination of devices capable of storing, accessing, organizing, and/or retrieving data, which may include any combination and number of data servers, relational databases, object oriented databases, simple web storage systems, cloud storage systems, data storage devices, data warehouses, flat files, and data storage configuration in any centralized, distributed, or clustered environment. The storage system components of the data store may include storage systems such as a SAN (Storage Area Network), cloud storage network, volatile or non-volatile RAM, optical media, or hard-drive type media. The data stored in the data store 220, for example, may be associated with the operation of the various applications and/or functional entities described below.

[0032] The data stored in the data store 220 may include video content 222, video content metadata 224, scenes 226 and/or heuristics 228. As used herein, the term “video content” generally refers to live video content and/or pre-recorded video content. The live video content may include live broadcasts, such as sporting events, political events, local and world events, live dramatic shows, news casts, live reality television, etc. The pre-recorded video content may include pre-recorded television shows, movies, home videos, etc. The video content 222 may be uploaded or transmitted to the data store 220 by a plurality of users or the video content may be professionally created. More specifically, the video content 222 may be uploaded to the data store 220 after being received by video content providers (e.g., movie studios, television networks). The video content 222 may be streamed over the network 275 to the first and second screen clients 280 and 290. In the case of live video, the video content may be send out to users within seconds are being stored in the video content 222 data store.

[0033] The data stored in the data store 220 may include video content metadata 224. The video content metadata 224 may be linked to individual video content included in the video content 222 data store. The video content metadata 224 may include one or more time codes indicating a beginning and/or an ending of a scene. The video content metadata 224 may be provided by video content providers and/or created by the user. The video content metadata 224 may be modified when a user identifies and/or creates a scene 224 from the video content 222. The time codes defining the scene created by the user may be included in the video content metadata 224. Thus, the video content metadata 224 created by a user may be stored in the data store 220. When the user shares the scene with a plurality of users, the video content metadata 224 may be accessed by the plurality of users. The video content metadata 224 may also include information about an encoding/decoding scheme for the video content, frames per second, bit rate, video resolution, key words that describe the nature of the video content, and other types of data for describing the video content 222.

[0034] The data stored in the data store 220 may include scenes 224 identified in the video content 222. For example, the video content 222 may be a 1-hour presidential speech, and the scene 226 may be a 1-minute clip of the speech. The scene 226 may include an occurrence of an interest event (e.g., an exciting play, a memorable line from a speech, etc.). The scenes 226 may include a time window (e.g., 30 seconds) before and after the occurrence of the interest event. The time window may automatically vary depending on the type of event (e.g., 20 seconds before a basketball play occurs or one minute after a football touchdown). In addition, the time window may be determined by the user. For example, an exciting soccer goal may occur at a time code of 66:33 and based on user instructions, the scene 226 may begin at a time code of 66:03 (i.e., 30 seconds prior to an occurrence of the interest event) and end at a time code of 66:53 (i.e., 20 seconds after the interest event occurs). In addition, the scenes 226 may be identified by video content providers and/or a plurality of users.

[0035] The data stored in the data store 220 may include heuristics 224. The heuristics 224 may be used to identify scenes 226 in the video content 222. The heuristics 224 may include historical scene data to assist in creating rules that enable the identification of scenes in various types of video content (e.g., live content, movies). The historical scene data may include typical scene lengths for a number of different video content types (e.g., typical scene lengths are nine minutes long for movies and seven minutes long for live basketball games). In addition, the heuristics 228 may include using interest event indicators to identify scenes 226. For example, the interest event indicators may detect a fade-to-black event (i.e., when the screen temporarily becomes black to indicate a change of scene). The interest event indicators may also detect various events that occur when the live video content is played. For example, an increase in crowd cheering from a boxing match may be used to identify the scene 226. As another example, a change in score (e.g., a football team's score changes from three to ten) may be used to identify the scene 226. Additional heuristics 228 may be used for identifying scenes 226 in the video content 222. When a heuristic rule is used and detects a possible scene, the scene may automatically be created for live video or the system may query the user as to whether the user may want to create a scene based on the interest event indicator that was detected.

[0036] The components executed on the computing device 210 may include a video content module 240, an event identification module 245, a scene definition module 250 and other applications, services, processes, systems, engines, or functionality not discussed in detail herein. The video content module 240 may be programmed to provide live video content (e.g., sporting events, political events, local or world news, game shows, etc.) to the first and second client devices 280 and 290 to be played. The video content module 240 may provide the live video content to a first screen device (e.g., a television) and/or a second screen device (e.g., a tablet computer). In addition, the video content module 240 may provide a user selected or automatically identified scene 226 taken from the live video content. The scene 226 may include an interest event (e.g., an exciting soccer goal, a humorous comment).

[0037] In another configuration, the video content module 240 may provide the live video content and/or the scene to the first screen device while simultaneously providing a replay of the scene 226 to the second screen device. The video content module 240 may provide the scene 226 to the device with a time window before and after the interest event occurs. The video content module 240 may determine a starting time code of a scene 226 from the video content 222, and then provide the scene 226 based on the starting time code. Further, the video content module 240 may provide a highlight video or list of highlights containing a plurality of scenes 226 based on a popularity ranking.

[0038] The event identification module 245 may be programmed to identify an interest event that occurs in the video content 222. The event identification module 245 may identify the interest event based on playback instructions received while the video content is being provided to the device. In particular, the playback instructions may include user actions, such as reversing video content over a previous section of the video content, forwarding video content over a subsequent section of the video content, replaying a previous section of the video content, playing video content at a reduced or an increased speed, adjusting a volume of the video content, zooming the video content, panning the video content, and/or pausing the video content. For example, the event identification module 245 may identify a scene based on the user reversing the video content for 30 seconds. In addition, the event identification module 245 may identify the interest event based on heuristics 228 (e.g., fade to black, historical scene data, crowd cheering, change of score in sporting event, etc.). In another configuration, the event identification module 245 may identify the interest event based on instructions received by a user. Thus, the event identification module 245 may be included in the second screen client 290. For example, the user may identify a scene of a live tennis match as an interest event, and the video content module 240 may provide a replay of the scene. A starting time code for the scene can also be stored in the scenes 226 data store. In addition, the event identification module 245 may identify interest events in live video content based on feedback from a group of users (i.e., group sourcing). For example, a user in the group of users may identify a scene in the tennis match as having an interesting event. Then the user may share the scene with other users in the group of users.

[0039] The scene definition module 250 may be programmed to automatically define a scene 226 with an interest event that was identified automatically based on user actions or heuristics The scene definition module 250 may define the scene 226 based on a starting time code of the interest event as provided by the user. For example, the interest event may be a three-point basket in a basketball game. The interest event (i.e., the scored basket) may be two seconds long. However, a replay of the interest event may provide the user with some context surrounding the play. Thus, 10 seconds may be provided in the scene to surround the interest event and the interest event may be at the center of the scene. Of course, the interest event may also be at the beginning or end of the scene. Similarly, the scene definition module 250 may create the scene 226 with a time window before and after the interest event occurs. For example, the 20 seconds of the basketball game prior to the three-point basket may be included in the scene 226. The scene identification module 250 may determine the time window automatically based on user settings. In one configuration, the time window may include a period of time since the previous change of score in the sporting event. In addition, the scene definition module 250 may determine the time window based on instructions entered by the user for a particular interest event. For example, after witnessing an exciting soccer goal, the user may instruct the scene definition module 250 to create a scene beginning two minutes before the soccer goal was scored and ending one minute after the soccer goal. When the user creates a new scene, the scene definition module 250 may modify the video content metadata 224 to include the time codes associated with the new scene.

[0040] Certain processing modules may be discussed in connection with this technology and FIG. 2. In one example configuration, a module of FIG. 2 may be considered a service with one or more processes executing on a server or other computer hardware. Such services may be centrally hosted functionality or a service application that may receive requests and provide output to other services or consumer devices. For example, modules providing services may be considered on-demand computing that is hosted in a server, cloud, grid, or cluster computing system. An application program interface (API) may be provided for each module to enable a second module to send requests to and receive output from the first module. Such APIs may also allow third parties to interface with the module and make requests and receive output from the modules. Third parties may either access the modules using authentication credentials that provide on-going access to the module or the third party access may be based on a per transaction access where the third party pays for specific transactions that are provided and consumed.

[0041] FIG. 3 is a drawing that illustrates an exemplary user interface 305 rendered by a device 300 for viewing video content based on user feedback. The user interface may include a plurality of scenes, such as Scene A 310, Scene B 312, Scene C 314, Scene D 316, and Scene E 318. The scenes may be identified based on feedback from a plurality of users. For example, a user may be watching an awards ceremony as live video. Other users may identify and create scenes from the awards ceremony. For example, a user may create a scene with a thank you speech by an award winner, and an additional user may create a scene with a musical performance included in the awards ceremony. These scenes may be included as Scene A 310, Scene B 312, etc. While watching the awards ceremony on a first screen device, the user interface 305 displaying the plurality of scenes may be part of a second screen device. Alternatively, the user may switch from the awards ceremony to the user interface 305 on the first screen device.

[0042] The user interface 305 may allow the user to view information associated with a specific scene (e.g., Scene B 312). For example, such information may be displayed when the user moves a cursor over the scene. In addition, moving the cursor over the scene may highlight the scene. The user interface 305 may include a play option 340 to play the highlighted scene, and the user interface 305 may include a preview option 350 to preview the highlighted scene via a video play area 320. The video play area 320 may provide a video preview and/or an image preview of the highlighted scene. The user interface 305 may include comments 360 associated with the highlighted scene. In addition, the user interface 305 may include scene information 330 about the highlighted scene. The scene information 330 may include the author, the date of creation, the length, the number of views, and the ratings associated with the highlighted scene.

[0043] In one configuration, the plurality of scenes displayed on the user interface 305 may be automatically generated based on interest event indicators. For example, the interest event indicators may include an increase in crowd cheering, a change of score in a sporting event, a fade-to-black scenario, etc. Upon an occurrence of an interest event indicator, a scene with the interest event may be automatically generated. The user interface 305 may allow the user to replay the scene that was automatically generated based on the interest event indicator. In other words, while the user is watching video content on a first screen device, a plurality of scenes relating to the video content may be displayed for replay on a second screen device. When the scene is replayed, the video content may be provided according to the beginning time code associated with the interest event.

[0044] FIG. 4 is a drawing that illustrates an exemplary user interface 405 rendered by an example device 400 for creating and sharing scenes of video content. For example, a user may be watching video content (e.g., a live soccer match) on a first screen device (e.g., a television). The video content may include live video content and/or pre-recorded video content. The user may use a device, via the user interface 405, to create and share scenes taken from video content currently being played. In particular, while watching the live soccer match on the first screen device, the user may use the second screen device to create a scene 410 from the video content, view bookmarked scenes 420, share the scene 430 with other users, and/or change scene options 430. The user may create the scene 410 by indicating that an interest event has occurred. For example, the user may select an option to create a scene after a goal is scored in the soccer match. The video content metadata may subsequently be modified to include a starting time code of the newly created scene.

[0045] While watching video content, the user may bookmark a plurality of scenes that have been created in order to replay the scenes in the future. For example, while watching a live basketball game, the user may bookmark a number of plays from the game in order to replay the scenes at a later time. Later, the user may access the bookmarked scenes via the user interface 405. When the user instructs the device to replay a bookmarked scene, the bookmarked scene may be played based on a starting time code of the bookmarked scene as found in the video content metadata created by the user.

[0046] The user may share the bookmarked scenes with a plurality of users. For example, the user may share the bookmarked scenes with an existing social group (e.g., the user's friends) or an interest oriented group (e.g., a fantasy football league group). In this example, the user may choose to select specific social associates for scene sharing. Alternatively, the user may share the bookmarked scenes via a web application, which in turn provides the bookmarked scenes to a plurality of users. For example, a user watching a live global news program may identify an interesting scene, and then upload a link to that scene to the web application. The web application may provide a link to that interest scene to additional users who are watching the same news program. The additional users may watch the scene based on the starting time code of the scene included in the video content metadata.

[0047] The scene options 430 may determine certain parameters (e.g., a default length) for the scene created by the user. In one configuration, the user may set the default rule that the time window surrounding the interest event may be automatically determined based on the type of interest event. Alternatively, the user may set the time window manually for each scene that is created. By having the scene include a time window surrounding the interest event, additional context in the scene may be provided to the user. In addition, the scene options 440 may include a plurality of customizable settings relating to creating scenes with interest events, bookmarking the scenes, and sharing the scenes with a plurality of users.

[0048] FIG. 5 is a drawing that illustrates an exemplary user interface rendered by a device 500 for navigating through scenes in video content. The user interface 502 may include a list of scenes in the video content to be played, such as Scene 1 504a, Scene 2 504b, Scene 3 504c, and Scene 4 504d. The device may play the scenes upon receiving instructions from a user. The list of scenes may be displayed while the video content is being played on the device, or the list of scenes may be included as a separate menu. The list of scenes may include a video and/or image preview for each scene. In addition, the user interface 502 may indicate the scene currently being played on the device by highlighting the scene (e.g., Scene 3).

[0049] The user interface 502 may include options to access a main menu 506, a previous scene 508, a next scene 510, and scene options 512. The user may skip backwards to the previous scene and/or skip ahead to the next scene in the video content. In one configuration, the video content being provided to the device may already be divided into a plurality of scenes. For example, the user may be watching a movie with a new scene beginning approximately every seven minutes. Thus, the user may skip to the next scene based on existing metadata in the video content that indicates a starting time code of the next scene. The existing metadata may be provided by video content providers (e.g., television networks, movie studios). Additionally or alternatively, the existing metadata may be formed based on instructions received by the user. For example, the user may identify an additional scene in the video content by marking a starting time code of the additional scene. The additional scene identified by the user may be a subset of an existing scene in the video content. When the user marks the starting time code of the additional scene, the existing metadata of the video content may be modified to include the starting time code of the additional scene. Thus, the metadata of the video content may include time codes given by video content providers, as well as time codes that are added based on user instructions. Thus, when the user switches to a different scene, the starting point of the scene may be determined by accessing the time codes in the metadata. Furthermore, the scenes defined by the video content providers may be stored and accessed separately from the scenes that are defined by user instructions.

[0050] In addition, a portion of the previous scene and the next scene may be stored in a cache of the device. For example, the first ten seconds of the next scene may be stored in the cache in anticipation of the user skipping to the next scene. Furthermore, a portion of previously played scenes may be stored in the cache to provide a faster scene load time when the user skips backward to the previous scene.

[0051] FIGS. 6A and 6B illustrate exemplary metadata 610 and 620 that describe the scenes for the video content. The metadata 610 may be embedded with information identifying a specific time code as a beginning time code of a scene. When a user switches to a scene which is different than the scene currently being provided, the scene may be played on the device according to the starting time code of the scene as identified in the metadata 610. In other words, the video content may include time-coded metadata useful for identifying and providing scenes in the video content. As an example, the metadata 610 may be used to identify scenes in a television program having a length of 25 minutes. As shown in FIG. 6A, the metadata 610 may include video provider content metadata 630 and/or user metadata 640. The video provider content metadata 630 may include metadata supplied by video content providers (e.g., television networks, movie studios). For example, the video content providers may describe the length of each scene (i.e., the starting time codes of each scene) in the video content via the metadata 610.

[0052] Additionally, the metadata 610 may include user metadata 640 generated by a plurality of users. The users may generate the user metadata 640 by creating scenes for the video content. When the user identifies a new scene for the video content, the metadata 610 may be modified to include a starting time code associated with the new scene. The metadata 610 of the video content may be modified in real time when the user identifies new scenes while the video content is being provided. The metadata 610 may include both video provider content metadata 630 and the user metadata 640 in order to enable the identification and creation of scenes to the device.

[0053] FIG. 6B shows metadata 620 for video content (e.g., a 90-minute live video program) that does not include existing metadata from video content providers. In other words, the video content may not be divided into a plurality of scenes. Thus, the metadata 620 may be modified based on the user metadata 640 to create new scenes as the live video content is viewed. For example, a user may identify a plurality of scenes in the live video content. The starting time codes associated with the plurality of scenes may be added to the user metadata 640 in real-time (i.e., as the user identifies each scene). If the user switches to a preceding scene or replays a previously played scene, the device may play the scene based on the user metadata 640.

[0054] FIGS. 7A and 7B illustrate exemplary user actions at specific time codes in the video content. For example, in 710 of FIG. 7A, a user may be watching video content that is 25 minutes in length. The user may perform a variety of actions while watching the video content. Based on the user actions, interest events may be inferred and used to create scenes containing the inferred interest events. For example, the user may reverse the video content from 2:45 to 2:22. The user may wish to replay the segment because an interest event occurred. Additionally, the user may wish to replay the segment because a confusing event occurred and the meaning of the scene may be clarified by replaying the scene. Based on the user's actions, an interest event may be inferred by the technology to have occurred between 2:22 to 2:45. A scene may be identified starting at 2:22 and ending at 2:45. The scene may be provided to the user for replay on the first screen device and/or the second screen device. In addition, the user may perform additional actions (e.g. forwarding from 6:20 to 7:15, turning on the subtitles at 11:35, reversing the video content from 21:10 to 20:45 and increasing the volume) that may infer the occurrence of interest events in the video content.

[0055] In one configuration, an interest event may be inferred based on a plurality of users performing the same user action. In other words, interest events may be identified by using group sourcing. For example, if a plurality of users (e.g., 10 users) have rewatched the same portion of video content (e.g., the first five minutes of a movie), then an interest event may be inferred to occur during the first five minutes of the movie. In this group sourcing configuration, a single user's action may not infer an interest event when the single user's action is not performed by a plurality of users. For example, a single user may rewind a movie for two minutes after performing an errand, but this user action may not infer an interest event because a plurality of users may not perform the same user action (i.e. pause the movie at a similar location for a similar period of time). However, when tens or hundreds of users rewind to the same location in a movie this is more likely to indicate something of interest happened at that point in the movie. Furthermore, scenes containing certain types of interest events (e.g., sexual content, drug use, language) may not be provided for playback, even though these interest events may be inferred based on the user actions of the user and/or the plurality of users.

[0056] As another example, in 720 of FIG. 7B, the user may be watching video content that is 25 minutes in length. The user may pause the video content at 4:05 for approximately four minutes. In addition, the user may access video content information on the second screen device during the pausing action. The user may access the video content information for a variety of reasons. For example, the user may wish to read a plot summary of the video content, view a list of actors/actresses and characters in the video content, and/or read reviews/ratings of the video content. Therefore, an interest event may be inferred surrounding the time code of 4:05 because the user may have paused the video content in order to read about an event that just occurred. A scene with the inferred interest event may be created and provided to the user for replay. In addition, the user may perform additional actions (e.g. reversing from 7:10 to 7:02 and playing the video content in slow motion, increasing the volume from 12:30 to 12:50, and forwarding from 17:50 to 25:15) that may allow the technology to infer the occurrence of interest events in the video content.

[0057] FIG. 8 illustrates an example of a method for identifying a scene in video content. The method may include the operation of providing video content to a first screen device, as in block 810. The first screen device may include a television, projector, desktop computer, laptop computer, tablet device, mobile phone, etc. The video content may be divided into a plurality of scenes based on machine-driven scene identification. The video content may include live video content or prerecorded video content. The live video content may include sporting events, political events, local or world news, game shows, etc. The prerecorded video content may include television programs, movies, etc.

[0058] An interest event that occurs in the video content may be identified based on playback instructions received (e.g., from a user about the video playback) while the video content is being provided to the first screen device, and the interest event may have a starting time code, as in block 820. The interest event that occurs in the video content may be identified based on user actions performed while the video content is being provided. The interest event may be identified according to at least one of the following user actions: forwarding video content to a subsequent section of the video content, reversing the video content to a previous section of the video content, or replaying a previous section of the video content.

[0059] A scene for the interest event may be defined based on the starting time code of the interest event, as in block 830. The scene with the interest event may include a time window before and after the interest event occurs. For example, the scene may begin a certain period of time (e.g., 30 seconds) before the interest event (e.g., a touchdown) occurs. The time window may be determined automatically based on the type of event (e.g., tennis), or the time window may be determined by the user after the interest event occurs.

[0060] The scene information for the interest event may be provided to a second screen device while the video content is being provided to the first screen device, as in block 840. Thus, the first screen device may play video content while the second screen device replays the scene with the interest event. For example, a user may be watching a live soccer match on a television while watching a replay of a previous goal on a tablet computer. In addition, the first screen device and/or the second screen device may store a portion of the scene (e.g., the first five seconds of the scene) in the cache. By storing video in the cache, a loading time associated with watching the video may be reduced. In the case where a scene is created for live video, the entire live video may be cached on the first screen or second screen device when the interest event is recorded.

[0061] FIG. 9 illustrates an example of a method for identifying a scene in live video content. The method may include the operation of providing video content to be played on a device, as in block 910. The video content may be provided to a first screen device (e.g., a television) and/or a second screen device (e.g., a mobile phone). The video content may be divided into a plurality of scenes. The video content may include live video content or prerecorded video content, such as movies, television programs, sporting events, political events, local or world news, game shows, etc.

[0062] A segment of the video content may be identified that includes an occurrence of an interest event, the segment being identified based on instructions received while the video content is being provided to the device, as in block 920. The segment of the video content may be identified by inferring the interest event based on user actions performed while the video content is being provided. The user actions may include at least one of: reversing video content over a previous section of the video content, forwarding video content over a subsequent section of the video content, replaying a previous section of the video content, playing the video content at a reduced speed or an increased speed, adjusting a volume of the video content, zooming the video content, panning the video content, and/or pausing the video content. In addition, the interest event may be inferred based on the user pausing the video content on the first screen device and/or retrieving video content information on the second screen device.

[0063] The video content metadata describing the video content may be modified to include a starting time code of an inferred scene, wherein the scene may be identified based on user actions that infer interest events in the video content. An instruction to provide and display the scene with the inferred event from the video content may be received at the first screen and/or second screen device. In response, the scene may be provided to the first and/or second screen device according to the video content metadata that has been modified to include the starting time code of the scene with the inferred event.

[0064] A scene with the segment may be created based on a starting time code associated with the interest event, as in block 930. For example, the interest event may begin at a time code of 15:45. Thus, the scene with the interest event may begin at a time code no later than 15:45. In addition, the scene may include a time window surrounding the interest event (i.e., before and after the interest event occurs). The time window may depend on a type of interest event. For example, the time window may be 24 seconds before and after interest events that relate to basketball. The scene may be automatically created with a time window associated with the type of interest event, or the time window may be determined by the user.

[0065] The scene with the segment may be provided for playback, as in block 940. The scene containing the interest event may be replayed on the device in response to receiving instructions for replaying the scene. For example, a user may enter instructions to replay a previous scene in a political debate. The scene may be replayed on the second screen device while live video content is being played on the first screen device. Thus, the user may replay a thought-provoking line of the political debate while live coverage of the debate is being provided. A portion of the scene may (e.g., the first ten seconds of the scene) be stored in a cache of either the first screen device or the second screen device. By caching the beginning of the scene, the time involved in starting and/or loading the scene may be reduced.

[0066] A highlight video containing a plurality of scenes may be provided to the device. The plurality of scenes may be identified based on user actions performed while the video content is being provided. In addition, a list of scenes to be replayed may be displayed on the first screen device or the second screen device. The scenes in the list may be identified based on user actions performed while the video content is being provided. The list of scenes may be included in a menu displayed on the first screen or second screen device. Furthermore, the menu may include options to replay the scene and/or preview the scenes in the list of scenes.

[0067] FIG. 10 illustrates an example of a method for identifying a scene in video content. The method may include the operation of providing video content to be played on a device, as in block 1010. A segment of video content that includes an occurrence of an interest event may be identified, the interest event being identified using heuristics while the video content is being provided to the device, as in block 1020. For example, historical scene information, fade-to-black scenarios, crowd cheering, and other heuristics may be used to identify a segment of video content.

[0068] A scene with the interest event may be created based on a starting time code associated with the interest event, as in block 1030. The scene with the interest event may be provided to the device for playback, as in block 1040. The scene with the interest event may be provided to the device for playback by providing the scene beginning at a predetermined time period prior to the interest event occurring. The predetermined time period may be heuristically determined based on a type of event. For example, the length of the scene may vary depending on whether the event includes crowd cheering, a goal/basket being scored in a sporting event, etc.

[0069] FIG. 11 illustrates a computing device 1110 on which modules of this technology may execute. A computing device 1110 is illustrated on which a high level example of the technology may be executed. The computing device 1110 may include one or more processors 1112 that are in communication with memory devices 1120. The computing device may include a local communication interface 1118 for the components in the computing device. For example, the local communication interface may be a local data bus and/or any related address or control busses as may be desired.

[0070] The memory device 1120 may contain modules that are executable by the processor(s) 1112 and data for the modules. Located in the memory device 1120 are modules executable by the processor. For example, a video content module 1124, an event identification module 1126, and a scene definition module 1128, and other modules may be located in the memory device 1120. The modules may execute the functions described earlier. A data store 1122 may also be located in the memory device 1120 for storing data related to the modules and other applications along with an operating system that is executable by the processor(s) 1112.

[0071] Other applications may also be stored in the memory device 1120 and may be executable by the processor(s) 1112. Components or modules discussed in this description that may be implemented in the form of software using high programming level languages that are compiled, interpreted or executed using a hybrid of the methods.

[0072] The computing device may also have access to I/O (input/output) devices 1114 that are usable by the computing devices. An example of an I/O device is a display screen 1130 that is available to display output from the computing devices. Other known I/O device may be used with the computing device as desired. Networking devices 1116 and similar communication devices may be included in the computing device. The networking devices 1116 may be wired or wireless networking devices that connect to the internet, a LAN, WAN, or other computing network.

[0073] The components or modules that are shown as being stored in the memory device 1120 may be executed by the processor 1112. The term “executable” may mean a program file that is in a form that may be executed by a processor 1112. For example, a program in a higher level language may be compiled into machine code in a format that may be loaded into a random access portion of the memory device 1120 and executed by the processor 1112, or source code may be loaded by another executable program and interpreted to generate instructions in a random access portion of the memory to be executed by a processor. The executable program may be stored in any portion or component of the memory device 1120. For example, the memory device 1120 may be random access memory (RAM), read only memory (ROM), flash memory, a solid state drive, memory card, a hard drive, optical disk, floppy disk, magnetic tape, or any other memory components.

[0074] The processor 1112 may represent multiple processors and the memory 1120 may represent multiple memory units that operate in parallel to the processing circuits. This may provide parallel processing channels for the processes and data in the system. The local interface 1118 may be used as a network to facilitate communication between any of the multiple processors and multiple memories. The local interface 1118 may use additional systems designed for coordinating communication such as load balancing, bulk data transfer, and similar systems.

[0075] While the flowcharts presented for this technology may imply a specific order of execution, the order of execution may differ from what is illustrated. For example, the order of two more blocks may be rearranged relative to the order shown. Further, two or more blocks shown in succession may be executed in parallel or with partial parallelization. In some configurations, one or more blocks shown in the flow chart may be omitted or skipped. Any number of counters, state variables, warning semaphores, or messages might be added to the logical flow for purposes of enhanced utility, accounting, performance, measurement, troubleshooting or for similar reasons.

[0076] Some of the functional units described in this specification have been labeled as modules, in order to more particularly emphasize their implementation independence. For example, a module may be implemented as a hardware circuit comprising custom VLSI circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A module may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like.

[0077] Modules may also be implemented in software for execution by various types of processors. An identified module of executable code may, for instance, comprise one or more blocks of computer instructions, which may be organized as an object, procedure, or function. Nevertheless, the executables of an identified module need not be physically located together, but may comprise disparate instructions stored in different locations which comprise the module and achieve the stated purpose for the module when joined logically together.

[0078] Indeed, a module of executable code may be a single instruction, or many instructions, and may even be distributed over several different code segments, among different programs, and across several memory devices. Similarly, operational data may be identified and illustrated herein within modules, and may be embodied in any suitable form and organized within any suitable type of data structure. The operational data may be collected as a single data set, or may be distributed over different locations including over different storage devices. The modules may be passive or active, including agents operable to perform desired functions.

[0079] The technology described here can also be stored on a computer readable storage medium that includes volatile and non-volatile, removable and non-removable media implemented with any technology for the storage of information such as computer readable instructions, data structures, program modules, or other data. Computer readable storage media include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other computer storage medium which can be used to store the desired information and described technology.

[0080] The devices described herein may also contain communication connections or networking apparatus and networking connections that allow the devices to communicate with other devices. Communication connections are an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules and other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. A “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. The term computer readable media as used herein includes communication media.

[0081] Reference was made to the examples illustrated in the drawings, and specific language was used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the technology is thereby intended. Alterations and further modifications of the features illustrated herein, and additional applications of the examples as illustrated herein, which would occur to one skilled in the relevant art and having possession of this disclosure, are to be considered within the scope of the description.

[0082] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more examples. In the preceding description, numerous specific details were provided, such as examples of various configurations to provide a thorough understanding of examples of the described technology. One skilled in the relevant art will recognize, however, that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the technology.

[0083] Although the subject matter has been described in language specific to structural features and/or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the described technology.