METHOD, SYSTEM AND APPARATUS OF RECORDING AND PLAYING BACK AN EXPERIENCE IN A VIRTUAL WORLDS SYSTEM

[0001] The present disclosure relates to virtual computer-generated environments through which users are able to navigate.

[0002] Computer generated virtual environments are increasingly popular methods for people, both real and automated, to interact within a networked system. Various on-line environments are known in which a three-dimensional, 2.5-dimensional or two-dimensional physical world (actual or fantasy) is simulated. Environments of this type are sometimes referred to as “virtual reality” or “virtual reality universe” (VRU) environments. In known VRU environments, an actual or fantasy universe is simulated within a computer memory. Multiple players may participate in the environment through a computer network, such as a local area network or a wide area network. Each player is typically represented by an “avatar,” which may comprise a figure of a man, woman, or other being, to represent them in the VRU environment. Players send inputs to a VRU engine to move their avatars around the VRU environment, and are able to cause interaction between their avatars and objects in the VRU. For example, a player's avatar may interact with an automated entity or person, simulated static objects, or avatars operated by other players.

[0003] It is possible to record a state of a VRU environment so that users may log out of the VRU and, upon their return, resume an activity at the point where they left off. For example, when playing chess, users may take a break and return to the same chess board at a later time.

[0004] It may be desirable to replay a scene that occurred in a VRU environment, such as a concert, a wedding or a lecture, for example.

[0005] In an aspect of the present disclosure, there is provided, a method of playing back a recorded experience in a virtual worlds system, comprising: instantiating, using one or more processors of a server, a new instance of a scene, the new instance being defined by data stored in memory, at least one client device displaying and participating in the new instance; retrieving a recorded experience file from the memory, the recorded experience file having been generated by saving an initial scene state and saving subsequent changes and respective times during a time period of the recorded experience; and playing back the recorded experience file by rendering, for display by the at least one client device, objects of the initial scene state in the new instance and rendering updates to the initial scene state based on the subsequent changes over the time period.

[0006] In another aspect of the present disclosure, there is provided a virtual worlds system for playing back a recorded experience, comprising: one or more servers instantiating a new instance of a scene using one or more processors of the one or more servers and playing back a recorded experience in the new instance by rendering objects of a recorded initial scene state of the recorded experience in the new instance and rendering updates to the recorded initial scene state based on subsequent recorded changes over a time period, one or more recorded experience files comprising the recorded initial scene state and the subsequent recorded changes being stored in memory of the one or more servers; and one or more client devices in communication with the one or more servers, the one or more client devices participating in the new instance.

[0007] In another aspect of the present disclosure, there is provided an apparatus for playing back a recorded experience in a virtual worlds system, comprising: one or more processors of a server system for instantiating a new instance of a scene and for communicating with one or more client devices participating in the new instance; memory of the server for storing one or more recorded experience files, the one or more recorded experience files having been generated by saving an initial scene state and saving subsequent changes and respective times during a time period of the recorded experience; wherein the processor plays back the recorded experience file in the new instance by rendering objects of the initial scene state in the new instance and rendering updates to the initial scene state based on the subsequent changes over the time period.

[0008] Other aspects and features of the present disclosure will become apparent to those ordinarily skilled in the art upon review of the following description of specific embodiments in conjunction with the accompanying figures.

[0019] It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Also, the description is not to be considered as limiting the scope of the embodiments described herein.

[0020] Referring to FIG. 1, a computer network system 100 connects virtual world service providers 102 with users 104 via a network 106. The computer network system 100 is capable of rendering a VRU environment and managing changes to the VRU environment resulting from user participation in the VRU environment. Examples of virtual worlds software usable to render and participate in virtual worlds include Curio™ by The Virtual World Web Inc. and Second Life™.

[0021] Service providers 102 include computing devices and systems offering virtual worlds services. Service providers 102 may include computing hardware and/or software configured to perform one or more of the processes described herein. In an embodiment, multiple service providers 102 may be housed on a single computing device. Service providers 102 may also be distributed across multiple devices, as in cloud computing or distributed systems. In an embodiment, which is depicted in FIG. 1, each service provider 102 is hosted on a server 106 that includes a processor 108 and a memory 110. According to another embodiment, in order to use the computational ability of the computer network system 100 as a whole, some functions of the server(s) may be provided by the users. This distributed computing structure may allow for more powerful computational abilities, as the full computer power of all users may be utilized to provide some or all server functions to the network. In this embodiment, some or all of the users may function as both clients and servers.

[0022] Service providers 102 communicate via one or more networks 106 with users 104. The networks 106 may include, for example, Internet networks, cellular networks, local area networks, wide area networks, wireless networks, Ethernet networks, and so on, as well as combinations thereof. Users 104 may include user computing devices, such as desktop computers, laptop computers, mobile computing devices, mobile phones and tablets, for example. The term “user” throughout this specification may be used to refer to a user device such as the aforementioned and/or an entity, such as an individual, operating such a device, as appropriate to the context of the term. In an embodiment, users 104 operate.

[0023] Additional devices and systems may be included in the system of FIG. 1. For example, service providers 102 may communicate with common data stores, such as account databases, registration or authentication servers and management servers, for example. Such use of centralized systems may allow service providers 102 to easily synchronize and coordinate among each other, for example, to coordinate avatar location transitions, to synchronize user accounts, and so on.

[0024] FIG. 2 shows an example arrangement of virtual worlds data, as used in an embodiment. The blocks represent types of data objects, and interconnections may represent relationships among the data elements, such as pointers, references, subclass relationships, or the like. Data structures may be stored on computer-readable media of the server system 102, for example, such as a hard drive, SSD, tape backup, distributed storage, cloud storage, and so on, and may be structured as relational database tables, flat files, C structures, programming language objects, database objects, and the like. In various embodiments, additional elements may be included, some elements may be removed, and/or elements may be arranged differently from what is shown.

[0025] Service provider 102 may be represented as a data object with general data relating to the service provider. The service provider 102 may maintain accounts 200 for users of the service provider. The account objects 200 may include relevant data relating to users, such as usernames, authentication information, attribute information, personal information, and so on. Each account may further be associated with one or more avatars 202, which the user may select for appearing in a virtual world. The service provider 102 may allow users to maintain multiple avatars, may restrict users to a single avatar or may allow users to navigate through a VRU as observers with no avatar, in various embodiments. When allowing users 104 to maintain multiple avatars, a service provider 102 may allow users to take on different appearances and personas in different situations, as selected by the users 104 and/or the service provider 102.

[0026] The service provider 102 may further maintain worlds 204. Each world 204 may represent a virtual space in which avatars may interact. The service provider 102 may maintain multiple virtual worlds 204, and each virtual world 204 may be operated by different users 104 or administrators of service provider 102.

[0027] Virtual worlds 204 may in turn include scenes 206. Scenes 206 represent virtual locations where users may visit and enter. A scene 206 may be defined by various attributes such as appearance, size, furniture, included items, architecture and so on. Scripts may also be included as part of the scene 206. Scripts are generally software based instructions that, when executed, result in changes to the scene 206. Scripts may be executed in response to user actions, may be executed at specified times, or may be executed in response to trigger events. For example, curtains covering a stage may be opened in response to an avatar pulling a cord, may be opened when a concert start time is reached or may be opened when the capacity of the concert venue reaches a predetermined size.

[0028] In some situations, it may be desirable to have a single scene 200 available but have avatars enter different copies of that scene. Such copies are implemented by instances 208. An instance 208 represents a running version of a scene, and users 104 interact within instances 208 of scenes 206. In an embodiment, the computer network system 100 includes one or more servers in communication with the service providers 102 to operate as instance hosts that run the instances 208. The service provider 102 may use rules specified for a particular scene 206 in order to determine when to create a new instance. For example, one may create a scene arranged to appear like a virtual tennis court, and it may be desirable for pairs of tennis players to be provided distinct copies of the scene rather than making all players use the same virtual space. So, a new instance would be initiated each time a pair of tennis players enters the virtual tennis court and instances that are currently running have reached capacity.

[0029] New instances may also be created based on time intervals, avatar user requests, availability of objects within each instance, and so on. In one embodiment, an environment may automatically change, or the avatars may be automatically transported, when a threshold event takes place. For example, avatars waiting to play poker may be automatically transported to the poker room when eight avatars have signed up for the table. In another example, a prospective tennis player may wait in a waiting room until another tennis player arrives and the waiting room is transformed into a tennis court.

[0030] Instances may originally be configured based on the specification of the underlying scene. For example, a newly created instance may be arranged to have the same contained items, the same appearance, the same size, the same scripts and so on, with respect to the scene. In alternate embodiments, the scene may include instructions that randomize or otherwise alter the nature of each new instance. As users interact with an instance of a scene, those avatars may cause changes to the contained items, appearance, and so on. Such changes may or may not be reflected back to the original scene, to other copies of the scene or other instances of the scene, possibly depending on the configuration of the scene. Additionally, in an embodiment, new instances may be based on existing instances, rather than being based on the original scenes.

[0031] Within a service provider, worlds 204, scenes 206, and instances 208 may be represented as data structure objects, such as document objects, for example. The document objects are encoded in a descriptive language and define respective 3-D, 2.5-D or 2-D modeled objects or spaces. Service providers may include software modules to convert these objects to graphical or other representations. The software may be operated on the service provider to generate those representations. Additionally or alternately, client software operating on user devices may perform the translation of data structure objects to graphical representations.

[0032] In order for one or more users to interact in a virtual world, the system 100 instantiates, in memory, an instance of a scene defined by a collection of document objects. The service provider 102, or another system component, controls contents of the collection of document objects in response to signals received at one or more processors from one or more client devices of the users 104. The service provider 102, or another system component, receives, at one or more processors, data requesting changes to an instance of a scene including at least position and orientation of objects modeled in the scene. Data representing the changes is then provided to the one or more users so that an event playing out in an instance of a scene may be experienced in substantially real time by users 104.

[0033] Referring to FIG. 3, an example of a scene 300 that has been instantiated by the service provider 102 is shown. In this example, the scene 300 is a performance venue including a floor 306 and walls 308 and a piano 302 located on a stage 304 that is rendered for display at a user 104. Also rendered for display are: a performer avatar 310 is located on the stage 304 and an audience that includes avatars 312 that surround the stage to view the performance. FIG. 4 schematically shows the scene 300 as it changes over time from an initial state 400 at the beginning of a time period, to an intermediate state 402 and to a final state 404 at the end of the time period. As shown, an elapsed time at which each scene state occurs, measured from the initial state, is indicated on timeline 406. In another embodiment, the changes from the initial state 400 are stored as a series of frames, which are recorded at regular intervals, such as approximately every 0.01667 seconds, for example, which results in 60 frames per second being recorded. In one example of this embodiment, the frames are recorded regardless of whether changes to the scene have occurred.

[0034] Any VRU experience, such as the performance depicted in FIGS. 3 and 4, for example, may be experienced by users 104 in substantially real time and may be recorded to be experienced at a later time. A method of generating a recording of a VRU experience is shown in FIG. 5. The method may be carried out by software executed by one or more processors of the computer network system 100 and stored as computer-readable code in a computer-readable medium. Coding of software for carrying out such a method is within the scope of a person of ordinary skill in the art given the present description.

[0035] According to the method, a new scene is first instantiated at 500. Then, the service provider 102, or another component of the system 100, saves, to an initial scene file stored at the one or more memories, an initial scene state, at 502. The initial scene state includes at least a position and orientation of objects, including avatars, rendered in the scene at a beginning of a time period of a new recording. Changed scene states result after a change to the initial scene state or to a previous changed scene state has occurred. Changes to the initial scene state and subsequent changed scene states, are saved to a recorded experience file, at 504. The initial scene state and the changes may be saved as data structure objects, such as document objects, for example, that are encoded in a descriptive language to define the objects of the scene. In an embodiment, the initial scene file and the recorded experience file are the same file.

[0036] Every change may be saved along with a time at which the change occurred or only some changes may be saved. In an embodiment, changes may be saved at selected intervals over the time period so that a complete recording may be generated by interpolating between the changes at the selected intervals.

[0037] Changes may be saved in response to avatar movements and interactions between objects, for example. Changes may also be saved in response to initiation of electronic messaging or webcam interactions between users, for example. In this example, a chat interaction or a video interaction may be captured and stored as a separate file accessible when the saved changes are played back. In addition, alterations to object appearance and behavior resulting from plug-in applications, or images or video inserted into the scene may also be saved as changes or in a separate file accessible when the saved changes are played back.

[0038] The VRU experience may be recorded in its entirety or, alternatively, only selected portions may be recorded. For example, walls of a scene may not be recorded in order to allow for expansion of a room, for example, when the recorded experience is played back. Further, avatars that are friends of a user may be recorded and avatars that are not known to a user may not be recorded or a lecturer may be recorded for future playback but the audience may not be recorded. Selecting only a portion of an instance of a scene to record has several advantages including: conserving memory, avoiding potential copyright issues and preserving privacy, for example.

[0039] When a recorded experience, which may also be referred to as a video experience or vidience™ is played back, a user is able to navigate around recorded objects, as an observer or as an avatar, as if navigating through the original scene. Depending on the contents of the recorded experience file, the user may also see alterations to object appearance and behavior available from plug-in applications, listen to audio and view chats, video and images, for example. A method for playing back the recorded experience is shown in FIG. 6. The method may be carried out by software executed by one or more processors of the computer network system 100 and stored as computer-readable code in a computer-readable medium. Coding of software for carrying out such a method is within the scope of a person of ordinary skill in the art given the present description.

[0040] According to the method, an instance of a new scene is first instantiated, at 600, using one or more processors of the computer network system 100. The new instance is defined by data stored in memory. The new scene may include nothing at all with all objects being drawn into the blank space from the recorded experience. Alternatively, the new scene may be include at least some boundaries but be empty of objects when instantiated or, optionally, objects of the new scene may be hidden. The recorded experience file(s) is then retrieved from memory and played back within the new scene, at 602, by rendering, at the service provider 102 for display at a client device, objects of the initial scene state in the new instance and rendering updates to the initial scene state based on the subsequent changes over the time period. The objects of the recorded experience are displayed in the new scene such that, similar to when the recorded experience was experienced the first time, objects and boundaries of the recorded experience are present. A user, by controlling an avatar via a user interface at a client device, may navigate through the recorded experience and interact with other avatars also participating in the playback of the recorded experience. As indicated at 602, more than one recorded experience may be played back at the same time.

[0041] In an embodiment, a collection of document objects represents the new scene and the recorded experience file. The service provider 102, or another system component, controls contents of the collection of document objects in response to communications received at one or more processors from one or more client devices of the users 104. The service provider 102, or another system component, receives data requesting changes to the instance of the new scene including at least position and orientation of objects in the new scene. Data representing the changes is then provided to the one or more users so that an event playing out in the instance of the new scene may be experienced in substantially real time by users 104. The recorded experience is not modifiable by events occurring during playback.

[0042] In another embodiment, a first collection of document objects represents the new scene and a second collection of document objects represents the recorded experience. In this embodiment, the new scene and the recorded experience are layered upon one another by the service provider 102 or at the client device. Any conflicts between the first and second collections, such as objects occupying the same location, for example, are resolved at by the service provider 102 or at the client device. Data requesting changes to the instance of the new scene is received by the service provider 102 and data representing the changes is then provided to the one or more users so that an event playing out in the instance of the new scene may be experienced in substantially real time by users 104. Similar to the previous embodiment, the recorded experience is not modifiable by events occurring during playback.

[0043] A recorded experience may be replayed in response to a user request to a service provider 102, may be replayed periodically at selected time intervals or may be replayed in response to a number of users waiting to participate in the recorded experience reaching a threshold. Users may join a new scene at any point during playback of the recorded experience. In this way, users may join other users at their point in the playback of the recorded experience. Alternatively users may be able to instantiate their own new scene and playback the recorded experience.

[0044] In an embodiment, in order for users navigating through a recorded experience to feel as though they are experiencing the original scene, collisions are set to occur when objects such as avatars, for example, of the new scene attempt to occupy the same location as objects, such as avatars, for example, of the recorded experience. In an alternative embodiment, collisions between objects of the new scene and objects of the recorded experience may be turned off. In this way, objects of the new scene may pass through objects of the recorded experience. This may be useful when the recorded experience is crowded and the avatars of the new scene are attempting to move across a room, for example.

[0045] In an embodiment, objects including avatars may be marked for non-recording. The objects may be marked for non-recording in response to user opt-out instructions or copyright issues, for example. Objects that are not recorded may be substituted with another object during playback of the recorded experience. In a further embodiment, replacement objects, such as anonymous bot avatars, for example, may be substituted for non-recorded avatars during playback of the recorded experience. According to the example, bots may be included to fill empty seats of avatars that are not recorded, due to opt-out, for example, at a concert or other event. Replacement of the objects that have opted out may be based on selection or may be automated.

[0046] In another embodiment, private messages may be saved. Private messages may enhance playback of a recorded experience. For example, private message between a student and an assistant professor during a lecture, may remind the student of additional information obtained during the lecture. Alternatively, the sender and/or the recipient may unmark the messages as private. Private messages may be viewable by all users when a recorded experience is played back or by the sender and recipient of the private messages only. In an alternative embodiment, private messages may not be saved and instead, metadata may be substituted for private messages so that users viewing the recorded experience may be made aware that private messages were sent but are not able to view the contents of the private messages.

[0047] Also referring to FIG. 7, a single recorded experience is played back in a new scene 700 and visiting users, represented by avatars 702, are able enter the new scene 700 when the recorded experience is playing and navigate around the recorded experience. As shown, all of the recorded objects are present as the visitor avatars 702 move through the new scene 700. As such, the visitor avatars 702 are able to experience the scene as it was experienced by the users who were present during the original scene.

[0048] When a new scene is instantiated, the new scene may include nothing at all, may include some boundaries but be empty of objects or may include some objects. In the example of FIG. 7, the new scene 700 includes a plant object 704 that was not present in the recorded event. The visitor avatars 702 may move through the scene 700, interact with one another, interact with other objects belonging to the new scene 700 and communicate via electronic messages or webcam, for example. Further, when a new scene is instantiated, the new scene may include scripts that are layered at locations where scripts from the recorded experience were available. This allows the recorded experience to maintain functionality that was available in the original scene that was recorded.

[0049] Referring to FIG. 8, according to another embodiment, the new scene, in which the recorded experience is being played, may be recorded, as indicated at 800. After the new scene has been recorded and a new recorded experience file saved, the new recorded experience may be played back, as described with respect to FIG. 6. The new recorded experience will only include elements from the new scene. Recorded experiences may be “layered” on top of one another each time an original scene is re-played. For example, as shown in FIG. 9, the original performance event of scene 300 and the performance event of scene 700 are layered and a new scene 900 is instantiated according to the method of FIG. 7. Avatars 902 are visitors to the new scene 800 and may interact with one another. According to another embodiment, a single new recorded scene file may be generated that merges the original recorded experience with one or more subsequent recorded experiences. The method of FIG. 8 may be carried out by software executed by one or more processors of the computer network system 100 and stored as computer-readable code in a computer-readable medium. Coding of software for carrying out such a method is within the scope of a person of ordinary skill in the art given the present description.

[0050] In one example, the system and method described herein is used for film production. An environment of a film, such as waves crashing on a beach, may be recorded in a first recorded experience. Extra actors may be recorded in a second recorded experience while the environment scene is played back. Finally, main actors may be recorded in a third recorded experience that is layered on the first recorded experience and second recorded experience to provide an animated storyboard or even a final film.

[0051] In the figures of the present description, the avatars of recorded experiences are differentiated from one another in order to aid in the understanding of the subject matter described herein. The avatars may instead appear similar to one another so that it is not readily apparent which avatars are from a recorded experience and which avatars are participating in the current scene. It may, however, be advantageous to visually or otherwise distinguish avatars participating in a current scene from avatars in a recorded experience. According to one embodiment, the avatars of a current scene may be rendered with a different degree of shading, different color, different type of clothing or a different theme, for example, than the avatars of a recorded experience. Alternately, the avatars of a current scene may appear in outline, or semi-transparent so that they appear as ghosts. Avatars of a recorded experience may be shown as transparent and may become more and more transparent to indicate how long ago they visited the scene. By visually distinguishing between current users and recorded objects, current users are able to locate one another more easily so that they may interact with one another. Other methods for distinguishing objects that may be interacted with in a currently scene may be provided. For example, recorded objects may be ghosted, semi-transparent or haloed. Alternatively, objects of a recorded experience may be in black and white while avatars visiting a current scene may be in colour. Avatars participating in a current scene from avatars in a recorded experience may be distinguished from one another using audio, may appear the same and only be distinguished from one another in response to a mouse-over action by the user. In addition, the audio or visual distinguishing feature(s) may be toggled on and off in response to a user input.

[0052] In another embodiment, a recorded experience may display a timeline selectable to allow a user to jump to a selected time in the recorded experience, fast forward and rewind. When multiple users are viewing the recorded experience simultaneously, a new instance may be instantiated when playback of the recorded experience is modified by skipping a portion of the recorded experience, for example.

[0053] A recorded experience may be modified after it has been recorded. Objects may be deleted, an appearance of objects may be modified or a theme may be added to objects of the recorded experience, for example. The recorded experience may also be shortened or particular time sequences of the recorded experience may be removed.

[0054] As the number and type of events that are occurring in virtual worlds systems continues to increase, the desire to record and playback experiences will become more significant. The applications for such a method, system and apparatus are numerous. For example, a prospective buyer participating in a virtual house tour may wish to recall suggestions from a contractor who was present during the tour, participants of a wedding may wish to re-experience the event on their anniversary or students at a lecture may wish to attend the lecture a further time in order to refresh their knowledge.

[0055] There are several advantages to the method, system and apparatus described herein, some of which may have already been described. Unlike a video recording, which displays a successive series of images, the method of playing back a recorded experience described herein allows a user to navigate through the recorded experience rather than simply watch a video playback. In addition, the method of playing back a recorded experience allows for users to interact with other users who are also participating in the playback of the recorded experience. Another advantage of the method of playing back a recorded experience described herein is that, unlike video playback, the recorded experience is not limited by a point of view from which the recording was captured. Because a user is able to navigate through the recorded experience by changing their position and orientation within the recorded experience, they may view objects from any angle that they choose. As such playback of the same recorded experience may be different for all users.

[0056] The above-described embodiments are intended to be examples only. Alterations, modifications and variations can be effected to the particular embodiments by those of skill in the art without departing from the scope of the present application, which is defined solely by the claims appended hereto.