3D digital virtual character generation with facial feature preservation

[0001] The present application generally relates to videoconferencing, and more particularly relates to generating three-dimensional (3D) virtual characters for video conferencing.

[0023] Examples are described herein in the context of systems and methods for generating three-dimensional (3D) digital virtual characters for applications such as video conferencing. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. Reference will now be made in detail to implementations of examples as illustrated in the accompanying drawings. The same reference indicators will be used throughout the drawings and the following description to refer to the same or like items.

[0024] In the interest of clarity, not all of the routine features of the examples described herein are shown and described. It will, of course, be appreciated that in the development of any such actual implementation, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another.

[0025] Videoconferencing has become a common way for people to meet as a group, but without being at the same physical location. Participants can be invited to a videoconference meeting, join from their personal computers or telephones, and are able to see and hear each other and converse largely as they would during an in-person group meeting or event. The advent of user-friendly videoconferencing software has enabled teams to work collaboratively despite being dispersed around the country or the world. It has also enabled families and friends to engage with each other in more meaningful ways, despite being physically distant from each other.

[0026] To engage with meeting attendees, some participants may choose to replace their appearances with 3D virtual characters that reflect the head movements and facial expressions of the participants. However, it is time consuming to generate 3D virtual characters with different facial expressions as they are currently manually created by designers. As a result, the number of choices for the 3D virtual characters is limited and the 3D virtual characters generated do not accurately reflect the face shapes and expressions of the participants.

[0027] To solve the above problems, example systems and methods for generating 3D digital virtual characters for video conferencing are provided herein. As described herein, a 3D virtual character (“virtual character,” “3DVC,” or “VC” in short) can be created based on a facial image of a participant and be animated during the meeting according to the movement and facial expression of the participant. The virtual character can be represented as a 3D Morphable Face Models (3DMM) model, which includes a human base face model, a set of human face bases (also referred to as PCA models), and human expression bases. A deformation transfer can be applied to transfer the human face bases and expression bases from an existing human 3DMM model to a 3D virtual character. During the setup phase, an input image of a user can be used to create a virtual character neutral face and customized virtual character expressions. During the online meeting, the virtual character can be animated based on the participant's expressions and poses and be rendered into images and be included in the video stream of the participant.

[0028] For example, a virtual character base model (VCBM) generation system can access a source human face model S (e.g., the human base face model in the 3DMM), a target human face model {tilde over (S)} (e.g., a combination of the human base face model and a human face base in the 3DMM), and a source virtual character model A (e.g., the source virtual character model A₀). The VCBM generation system can determine a deformation transfer (A, S, {tilde over (S)}) such that the source virtual character model A can be deformed into a target virtual character model A in a way similar to how the source human face model S is deformed into the target human face model {tilde over (S)}, that is Ã=(A, S, {tilde over (S)}). For instance, the source human face model S can be the human base face S₀ in the 3DMM, and the target human face model {tilde over (S)} can be a combination of the human base face S₀ and a human face basis

[0029] $S_{\mathrm{id}}^j$
in the 3DMM. For a source virtual character model A, the deformation can generate Ã^j and the 3D virtual character basis (or virtual character PCA model)

[0030] $A_{\mathrm{id}}^j$
can be derived as

[0031] $A_{\mathrm{id}}^j={\tilde{A}}^j-A.$
Based on the 3D virtual character basis

[0032] $A_{\mathrm{id}}^j,$
the virtual character neutral face can be derived for a participant and for the virtual character model A. Virtual character expression bases can be derived similarly. Combining the virtual character expression bases according to the facial feature of the participant can lead to the customized virtual character expressions discussed above.

[0033] To derive the deformation transfer, correspondence between the human base face S₀ and the virtual character face model A₀ can be established. For instance, a set of points (referred to as “facial landmarks”) can be labelled on the human base face S₀ and the virtual character face model A₀ to indicate the vertex correspondence between the human base face and the virtual character face model. Based on the vertex correspondence, the virtual character face model A₀ can be deformed in a way similar to the human base face S₀ being deformed into the target human face {tilde over (S)}. For example, a loss function can be constructed and minimized so that the deformation transfer from the source virtual character model A to the target virtual character model à can be performed according to the deformation transfer from the source human face model S to the target human face model {tilde over (S)}. The minimization of the loss function can be performed by reformulating the loss function to a linear form and solving it through a linear solver. However, virtual character models are often different from the human face model and simply applying the deformation transfer from S to {tilde over (S)} to the source virtual character A may lead to artifacts in the target virtual character A. As such, various types of processing are described herein to reduce the artifacts in the target virtual character A.

[0034] For example, the source human face model S and the target human face model S may only contain a human face, whereas a virtual character model contains, in addition to the face part (referred to as the “face region”), other models (“non-face region”) such as an eye model and a mouth model. These models may contain internal areas that are not observable from the face of the virtual character. For example, an eye model may include an eye socket for holding the eyeball and the majority part of the eye socket is not observable from the face of the virtual character. Likewise, the mouth model of the virtual character also includes an interior portion holding the teeth and tongue of the virtual character that is not observable from the face of the virtual character. These non-face regions do not have corresponding portions in the human face model. As such, the deformation transfer from the source human face model S to the target human face model S does not provide information regarding the deformation of the non-face region. If directly applying the deformation transfer from the source human face model S to the target human face model {tilde over (S)} to the source virtual character A, the face region of the source virtual character A can be deformed properly, but the non-face region is not deformed. As a result, the non-face region may be dislocated relative to the face region leading to artifacts such as the eye socket or mouth part protruding outside the face.

[0035] To solve these problems, virtual triangles can be generated connecting the face region and non-face region. For instance, a vertex-triangle distance between each vertex of a surface in the face region and each triangle of a surface in the non-face region can be determined. If the distance is below a threshold distance value, a virtual triangle connecting a vertex of the surface in the face region and an edge of the surface in the non-face region can be added to the source virtual character model. Multiple virtual triangles can be added in a similar way. Deforming the source virtual character face model can be performed by introducing constraints to minimize the deformation to the virtual triangles. In this way, the deformation in the face region can be propagated into the non-face region so that the non-face region can be deformed along with the face region of the virtual character.

[0036] The deformation transfer can be performed in a two-stage process. In the first stage, the face region of the virtual character is deformed while the non-face region is fixed. Constraints related to the non-face region are disabled, such as the constraints defined based on the virtual triangles discussed above. After the face region is deformed, the virtual triangles can be added back to the virtual character and the second stage of the deformation is performed. In the second stage, the deformed face region is fixed, and the non-face region is deformed with constraints that are not related to the non-face region disabled. For example, the virtual triangles can be used to enforce the deformation of the virtual triangles to be minimized so that the non-face regions can be deformed according to the deformation of the face region.

[0037] In another example, the two-stage deformation process can be performed without introducing the virtual triangles. For example, correspondence can be established between pairs of surfaces in the face region and non-face region. In other words, for a triangle surface in the face region, a triangle surface in the non-face region that is close to the face region can be identified. These pairs of corresponding surfaces can be utilized when deforming the non-face region in the second stage to enforce the deformation of the surface in the non-face region to be close to the deformation of the corresponding surface in the face region. In this way, the non-face region can be deformed similarly as the face region, thereby reducing the artifacts in the deformed virtual character.

[0038] Further, virtual characters are unreal characters and typically have facial features that are unproportional to other features. For example, the virtual character may have larger eyes with a smaller mouth than those in the human face. Deformation transfer derived from human face models may not lead to the proper deformation for those facial features. To address this, facial feature enhancement can be introduced as constraints in the deformation transfer process to ensure the desired facial features of the deformed virtual character to be obtained. For example, the deformation can be performed such that a difference between the deformation for a facial feature of the virtual character and a scaled deformation for the corresponding facial feature in the human face is minimized. In this way, the facial feature of the virtual character can be enhanced (e.g., enlarged eyes or shrunk mouth) from the corresponding facial feature of the human face model. In some implementations, the difference between the deformation for the facial feature of the virtual character and the scaled deformation for the corresponding facial feature in the human face is introduced as a loss term in the loss function to be minimized. In some examples, the deformation is represented by a deformation gradient.

[0039] While some facial features of the virtual character need to be enhanced, there are certain facial features need to be preserved during the deformation transfer. For example, when performing deformation transfer to generate virtual character expression bases from human face expression bases, facial features, such as closed mouth, closed eye, should be preserved. In other words, if a human facial expression basis has the mouth or an eye closed, the corresponding virtual character expression basis should also have the mouth or eye closed. Other features, such as the shape of the mouth and eyes, the symmetry of the vertices, should also be preserved. In some implementations, each feature preservation is introduced as a loss term in the loss function to be minimized. For example, a set of virtual feature triangles can be generated based on vertices of a feature curve. A loss term can be defined based on the differences between deformation gradients of the set of virtual feature triangles and an identity deformation gradient so that the deformation of the region near the feature curve is minimized. For a closeness feature, the closeness of a facial feature can be defined by two feature curves being overlapped. The corresponding loss term can be defined based on a mapping distance between the two feature curves so that the mapping distance is minimized (thus keeping the facial feature closed as much as possible). The feature curves used in the feature preservation can be defined based on facial landmark points on the human face model.

[0040] As described herein, certain embodiments provide improvements to video conferencing by solving problems that are specific to online platforms. These improvements include more efficiently generating virtual character face bases and expression bases. These virtual character base models allow the virtual character for a video conference participant to be generated to match the shape and other features of the participant and to show expressions dynamically based on the facial expressions of the participant during the meeting. These improvements significantly increase the speed of generating the virtual character for each participant and allow the virtual character to be customized according to the participant. This allows the video conference to be conducted more efficiently and the participants stay more engaged in the meeting, thereby reducing the amount of time and the computing and network resources spent on the video conference.

[0041] This illustrative example is given to introduce the reader to the general subject matter discussed herein and the disclosure is not limited to this example. The following sections describe various additional non-limiting examples and examples of systems and methods for generating 3D digital virtual characters for video conferencing.

[0042] Referring now to FIG. 1, FIG. 1 shows an example system 100 that provides videoconferencing functionality to various client devices. The system 100 includes a chat and video conference provider 110 that is connected to multiple communication networks 120, 130, through which various client devices 140-180 can participate in video conferences hosted by the chat and video conference provider 110. For example, the chat and video conference provider 110 can be located within a private network to provide video conferencing services to devices within the private network, or it can be connected to a public network, e.g., the internet, so it may be accessed by anyone. Some examples may even provide a hybrid model in which a chat and video conference provider 110 may supply components to enable a private organization to host private internal video conferences or to connect its system to the chat and video conference provider 110 over a public network.

[0043] The system optionally also includes one or more authentication and authorization providers, e.g., authentication and authorization provider 115, which can provide authentication and authorization services to users of the client devices 140-160. Authentication and authorization provider 115 may authenticate users to the chat and video conference provider 110 and manage user authorization for the various services provided by chat and video conference provider 110. In this example, the authentication and authorization provider 115 is operated by a different entity than the chat and video conference provider 110, though in some examples, they may be the same entity.

[0044] Chat and video conference provider 110 allows clients to create videoconference meetings (or “meetings”) and invite others to participate in those meetings as well as perform other related functionality, such as recording the meetings, generating transcripts from meeting audio, generating summaries and translations from meeting audio, manage user functionality in the meetings, enable text messaging during the meetings, create and manage breakout rooms from the virtual meeting, etc. FIG. 2, described below, provides a more detailed description of the architecture and functionality of the chat and video conference provider 110. It should be understood that the term “meeting” encompasses the term “webinar” used herein.

[0045] Meetings in this example chat and video conference provider 110 are provided in virtual rooms to which participants are connected. The room in this context is a construct provided by a server that provides a common point at which the various video and audio data is received before being multiplexed and provided to the various participants. While a “room” is the label for this concept in this disclosure, any suitable functionality that enables multiple participants to participate in a common videoconference may be used.

[0046] To create a meeting with the chat and video conference provider 110, a user may contact the chat and video conference provider 110 using a client device 140-180 and select an option to create a new meeting. Such an option may be provided in a webpage accessed by a client device 140-160 or a client application executed by a client device 140-160. For telephony devices, the user may be presented with an audio menu that they may navigate by pressing numeric buttons on their telephony device. To create the meeting, the chat and video conference provider 110 may prompt the user for certain information, such as a date, time, and duration for the meeting, a number of participants, a type of encryption to use, whether the meeting is confidential or open to the public, etc. After receiving the various meeting settings, the chat and video conference provider may create a record for the meeting and generate a meeting identifier and, in some examples, a corresponding meeting password or passcode (or other authentication information), all of which meeting information is provided to the meeting host.

[0047] After receiving the meeting information, the user may distribute the meeting information to one or more users to invite them to the meeting. To begin the meeting at the scheduled time (or immediately, if the meeting was set for an immediate start), the host provides the meeting identifier and, if applicable, corresponding authentication information (e.g., a password or passcode). The video conference system then initiates the meeting and may admit users to the meeting. Depending on the options set for the meeting, the users may be admitted immediately upon providing the appropriate meeting identifier (and authentication information, as appropriate), even if the host has not yet arrived, or the users may be presented with information indicating that the meeting has not yet started, or the host may be required to specifically admit one or more of the users.

[0048] During the meeting, the participants may employ their client devices 140-180 to capture audio or video information and stream that information to the chat and video conference provider 110. They also receive audio or video information from the chat and video conference provider 110, which is displayed by the respective client device 140 to enable the various users to participate in the meeting.

[0049] At the end of the meeting, the host may select an option to terminate the meeting, or it may terminate automatically at a scheduled end time or after a predetermined duration. When the meeting terminates, the various participants are disconnected from the meeting, and they will no longer receive audio or video streams for the meeting (and will stop transmitting audio or video streams). The chat and video conference provider 110 may also invalidate the meeting information, such as the meeting identifier or password/passcode.

[0050] To provide such functionality, one or more client devices 140-180 may communicate with the chat and video conference provider 110 using one or more communication networks, such as network 120 or the public switched telephone network (“PSTN”) 130. The client devices 140-180 may be any suitable computing or communication devices that have audio or video capability. For example, client devices 140-160 may be conventional computing devices, such as desktop or laptop computers having processors and computer-readable media, connected to the chat and video conference provider 110 using the internet or other suitable computer network. Suitable networks include the internet, any local area network (“LAN”), metro area network (“MAN”), wide area network (“WAN”), cellular network (e.g., 3G, 4G, 4G LTE, 5G, etc.), or any combination of these. Other types of computing devices may be used instead or as well, such as tablets, smartphones, and dedicated video conferencing equipment. Each of these devices may provide both audio and video capabilities and may enable one or more users to participate in a video conference meeting hosted by the chat and video conference provider 110.

[0051] In addition to the computing devices discussed above, client devices 140-180 may also include one or more telephony devices, such as cellular telephones (e.g., cellular telephone 170), internet protocol (“IP”) phones (e.g., telephone 180), or conventional telephones. Such telephony devices may allow a user to make conventional telephone calls to other telephony devices using the PSTN, including the chat and video conference provider 110. It should be appreciated that certain computing devices may also provide telephony functionality and may operate as telephony devices. For example, smartphones typically provide cellular telephone capabilities and thus may operate as telephony devices in the example system 100 shown in FIG. 1. In addition, conventional computing devices may execute software to enable telephony functionality, which may allow the user to make and receive phone calls, e.g., using a headset and microphone. Such software may communicate with a PSTN gateway to route the call from a computer network to the PSTN. Thus, telephony devices encompass any devices that can make conventional telephone calls and are not limited solely to dedicated telephony devices like conventional telephones.

[0052] Referring again to client devices 140-160, these devices 140-160 contact the chat and video conference provider 110 using network 120 and may provide information to the chat and video conference provider 110 to access functionality provided by the chat and video conference provider 110, such as access to create new meetings or join existing meetings. To do so, the client devices 140-160 may provide user authentication information, meeting identifiers, meeting passwords or passcodes, etc. In examples that employ an authentication and authorization provider 115, a client device, e.g., client devices 140-160, may operate in conjunction with an authentication and authorization provider 115 to provide authentication and authorization information or other user information to the chat and video conference provider 110.

[0053] An authentication and authorization provider 115 may be any entity trusted by the chat and video conference provider 110 that can help authenticate a user to the chat and video conference provider 110 and authorize the user to access the services provided by the chat and video conference provider 110. For example, a trusted entity may be a server operated by a business or other organization with whom the user has created an account, including authentication and authorization information, such as an employer or trusted third-party. The user may sign into the authentication and authorization provider 115, such as by providing a username and password, to access their account information at the authentication and authorization provider 115. The account information includes information established and maintained at the authentication and authorization provider 115 that can be used to authenticate and facilitate authorization for a particular user, irrespective of the client device they may be using. An example of account information may be an email account established at the authentication and authorization provider 115 by the user and secured by a password or additional security features, such as single sign-on, hardware tokens, two-factor authentication, etc. However, such account information may be distinct from functionality such as email. For example, a health care provider may establish accounts for its patients. And while the related account information may have associated email accounts, the account information is distinct from those email accounts.

[0054] Thus, a user's account information relates to a secure, verified set of information that can be used to authenticate and provide authorization services for a particular user and should be accessible only by that user. By properly authenticating, the associated user may then verify themselves to other computing devices or services, such as the chat and video conference provider 110. The authentication and authorization provider 115 may require the explicit consent of the user before allowing the chat and video conference provider 110 to access the user's account information for authentication and authorization purposes.

[0055] Once the user is authenticated, the authentication and authorization provider 115 may provide the chat and video conference provider 110 with information about services the user is authorized to access. For instance, the authentication and authorization provider 115 may store information about user roles associated with the user. The user roles may include collections of services provided by the chat and video conference provider 110 that users assigned to those user roles are authorized to use. Alternatively, more or less granular approaches to user authorization may be used.

[0056] When the user accesses the chat and video conference provider 110 using a client device, the chat and video conference provider 110 communicates with the authentication and authorization provider 115 using information provided by the user to verify the user's account information. For example, the user may provide a username or cryptographic signature associated with an authentication and authorization provider 115. The authentication and authorization provider 115 then either confirms the information presented by the user or denies the request. Based on this response, the chat and video conference provider 110 either provides or denies access to its services, respectively.

[0057] For telephony devices, e.g., client devices 170-180, the user may place a telephone call to the chat and video conference provider 110 to access video conference services. After the call is answered, the user may provide information regarding a video conference meeting, e.g., a meeting identifier (“ID”), a passcode or password, etc., to allow the telephony device to join the meeting and participate using audio devices of the telephony device, e.g., microphone(s) and speaker(s), even if video capabilities are not provided by the telephony device.

[0058] Because telephony devices typically have more limited functionality than conventional computing devices, they may be unable to provide certain information to the chat and video conference provider 110. For example, telephony devices may be unable to provide authentication information to authenticate the telephony device or the user to the chat and video conference provider 110. Thus, the chat and video conference provider 110 may provide more limited functionality to such telephony devices. For example, the user may be permitted to join a meeting after providing meeting information, e.g., a meeting identifier and passcode, but only as an anonymous participant in the meeting. This may restrict their ability to interact with the meetings in some examples, such as by limiting their ability to speak in the meeting, hear or view certain content shared during the meeting, or access other meeting functionality, such as joining breakout rooms or engaging in text chat with other participants in the meeting.

[0059] It should be appreciated that users may choose to participate in meetings anonymously and decline to provide account information to the chat and video conference provider 110, even in cases where the user could authenticate and employs a client device capable of authenticating the user to the chat and video conference provider 110. The chat and video conference provider 110 may determine whether to allow such anonymous users to use services provided by the chat and video conference provider 110. Anonymous users, regardless of the reason for anonymity, may be restricted as discussed above with respect to users employing telephony devices, and in some cases may be prevented from accessing certain meetings or other services, or may be entirely prevented from accessing the chat and video conference provider 110.

[0060] Referring again to chat and video conference provider 110, in some examples, it may allow client devices 140-160 to encrypt their respective video and audio streams to help improve privacy in their meetings. Encryption may be provided between the client devices 140-160 and the chat and video conference provider 110 or it may be provided in an end-to-end configuration where multimedia streams (e.g., audio or video streams) transmitted by the client devices 140-160 are not decrypted until they are received by another client device 140-160 participating in the meeting. Encryption may also be provided during only a portion of a communication, for example encryption may be used for otherwise unencrypted communications that cross international borders.

[0061] Client-to-server encryption may be used to secure the communications between the client devices 140-160 and the chat and video conference provider 110, while allowing the chat and video conference provider 110 to access the decrypted multimedia streams to perform certain processing, such as recording the meeting for the participants or generating transcripts of the meeting for the participants. End-to-end encryption may be used to keep the meeting entirely private to the participants without any worry about a chat and video conference provider 110 having access to the substance of the meeting. Any suitable encryption methodology may be employed, including key-pair encryption of the streams. For example, to provide end-to-end encryption, the meeting host's client device may obtain public keys for each of the other client devices participating in the meeting and securely exchange a set of keys to encrypt and decrypt multimedia content transmitted during the meeting. Thus, the client devices 140-160 may securely communicate with each other during the meeting. Further, in some examples, certain types of encryption may be limited by the types of devices participating in the meeting. For example, telephony devices may lack the ability to encrypt and decrypt multimedia streams. Thus, while encrypting the multimedia streams may be desirable in many instances, it is not required as it may prevent some users from participating in a meeting.

[0062] By using the example system shown in FIG. 1, users can create and participate in meetings using their respective client devices 140-180 via the chat and video conference provider 110. Further, such a system enables users to use a wide variety of different client devices 140-180 from traditional standards-based video conferencing hardware to dedicated video conferencing equipment to laptop or desktop computers to handheld devices to legacy telephony devices. etc.

[0063] Referring now to FIG. 2, FIG. 2 shows an example system 200 in which a chat and video conference provider 210 provides videoconferencing functionality to various client devices 220-250. The client devices 220-250 include two conventional computing devices 220-230, dedicated equipment for a video conference room 240, and a telephony device 250. Each client device 220-250 communicates with the chat and video conference provider 210 over a communications network, such as the internet for client devices 220-240 or the PSTN for client device 250, generally as described above with respect to FIG. 1. The chat and video conference provider 210 is also in communication with one or more authentication and authorization providers 215, which can authenticate various users to the chat and video conference provider 210 generally as described above with respect to FIG. 1.

[0064] In this example, the chat and video conference provider 210 employs multiple different servers (or groups of servers) to provide different examples of video conference functionality, thereby enabling the various client devices to create and participate in video conference meetings. The chat and video conference provider 210 uses one or more real-time media servers 212, one or more network services servers 214, one or more video room gateways 216, one or more message and presence gateways 217, and one or more telephony gateways 218. Each of these servers 212-218 is connected to one or more communications networks to enable them to collectively provide access to and participation in one or more video conference meetings to the client devices 220-250.

[0065] The real-time media servers 212 provide multiplexed multimedia streams to meeting participants, such as the client devices 220-250 shown in FIG. 2. While video and audio streams typically originate at the respective client devices, they are transmitted from the client devices 220-250 to the chat and video conference provider 210 via one or more networks where they are received by the real-time media servers 212. The real-time media servers 212 determine which protocol is optimal based on, for example, proxy settings and the presence of firewalls, etc. For example, the client device might select among UDP, TCP, TLS, or HTTPS for audio and video and UDP for content screen sharing.

[0066] The real-time media servers 212 then multiplex the various video and audio streams based on the target client device and communicate multiplexed streams to each client device. For example, the real-time media servers 212 receive audio and video streams from client devices 220-240 and only an audio stream from client device 250. The real-time media servers 212 then multiplex the streams received from devices 230-250 and provide the multiplexed stream to client device 220. The real-time media servers 212 are adaptive, for example, reacting to real-time network and client changes, in how they provide these streams. For example, the real-time media servers 212 may monitor parameters such as a client's bandwidth CPU usage, memory and network I/O as well as network parameters such as packet loss, latency and jitter to determine how to modify the way in which streams are provided.

[0067] The client device 220 receives the stream, performs any decryption, decoding, and demultiplexing on the received streams, and then outputs the audio and video using the client device's video and audio devices. In this example, the real-time media servers do not multiplex client device 220's own video and audio feeds when transmitting streams to it. Instead, each client device 220-250 only receives multimedia streams from other client devices 220-250. For telephony devices that lack video capabilities, e.g., client device 250, the real-time media servers 212 only deliver multiplex audio streams. The client device 220 may receive multiple streams for a particular communication, allowing the client device 220 to switch between streams to provide a higher quality of service.

[0068] In addition to multiplexing multimedia streams, the real-time media servers 212 may also decrypt incoming multimedia stream in some examples. As discussed above, multimedia streams may be encrypted between the client devices 220-250 and the chat and video conference provider 210. In some such examples, the real-time media servers 212 may decrypt incoming multimedia streams, multiplex the multimedia streams appropriately for the various clients, and encrypt the multiplexed streams for transmission.

[0069] As mentioned above with respect to FIG. 1, the chat and video conference provider 210 may provide certain functionality with respect to unencrypted multimedia streams at a user's request. For example, the meeting host may be able to request that the meeting be recorded or that a transcript of the audio streams be prepared, which may then be performed by the real-time media servers 212 using the decrypted multimedia streams, or the recording or transcription functionality may be off-loaded to a dedicated server (or servers), e.g., cloud recording servers, for recording the audio and video streams. In some examples, the chat and video conference provider 210 may allow a meeting participant to notify it of inappropriate behavior or content in a meeting. Such a notification may trigger the real-time media servers to 212 record a portion of the meeting for review by the chat and video conference provider 210. Still other functionality may be implemented to take actions based on the decrypted multimedia streams at the chat and video conference provider, such as monitoring video or audio quality, adjusting or changing media encoding mechanisms, etc.

[0070] It should be appreciated that multiple real-time media servers 212 may be involved in communicating data for a single meeting and multimedia streams may be routed through multiple different real-time media servers 212. In addition, the various real-time media servers 212 may not be co-located, but instead may be located at multiple different geographic locations, which may enable high-quality communications between clients that are dispersed over wide geographic areas, such as being located in different countries or on different continents. Further, in some examples, one or more of these servers may be co-located on a client's premises, e.g., at a business or other organization. For example, different geographic regions may each have one or more real-time media servers 212 to enable client devices in the same geographic region to have a high-quality connection into the chat and video conference provider 210 via local servers 212 to send and receive multimedia streams, rather than connecting to a real-time media server located in a different country or on a different continent. The local real-time media servers 212 may then communicate with physically distant servers using high-speed network infrastructure, e.g., internet backbone network(s), that otherwise might not be directly available to client devices 220-250 themselves. Thus, routing multimedia streams may be distributed throughout the video conference system and across many different real-time media servers 212.

[0071] Turning to the network services servers 214, these servers 214 provide administrative functionality to enable client devices to create or participate in meetings, send meeting invitations, create or manage user accounts or subscriptions, and other related functionality. Further, these servers may be configured to perform different functionalities or to operate at different levels of a hierarchy, e.g., for specific regions or localities, to manage portions of the chat and video conference provider under a supervisory set of servers. When a client device 220-250 accesses the chat and video conference provider 210, it will typically communicate with one or more network services servers 214 to access their account or to participate in a meeting.

[0072] When a client device 220-250 first contacts the chat and video conference provider 210 in this example, it is routed to a network services server 214. The client device may then provide access credentials for a user, e.g., a username and password or single sign-on credentials, to gain authenticated access to the chat and video conference provider 210. This process may involve the network services servers 214 contacting an authentication and authorization provider 215 to verify the provided credentials. Once the user's credentials have been accepted, and the user has consented, the network services servers 214 may perform administrative functionality, like updating user account information, if the user has account information stored with the chat and video conference provider 210, or scheduling a new meeting, by interacting with the network services servers 214. Authentication and authorization provider 215 may be used to determine which administrative functionality a given user may access according to assigned roles, permissions, groups, etc.

[0073] In some examples, users may access the chat and video conference provider 210 anonymously. When communicating anonymously, a client device 220-250 may communicate with one or more network services servers 214 but only provide information to create or join a meeting, depending on what features the chat and video conference provider allows for anonymous users. For example, an anonymous user may access the chat and video conference provider using client device 220 and provide a meeting ID and passcode. The network services server 214 may use the meeting ID to identify an upcoming or on-going meeting and verify the passcode is correct for the meeting ID. After doing so, the network services server(s) 214 may then communicate information to the client device 220 to enable the client device 220 to join the meeting and communicate with appropriate real-time media servers 212.

[0074] In cases where a user wishes to schedule a meeting, the user (anonymous or authenticated) may select an option to schedule a new meeting and may then select various meeting options, such as the date and time for the meeting, the duration for the meeting, a type of encryption to be used, one or more users to invite, privacy controls (e.g., not allowing anonymous users, preventing screen sharing, manually authorize admission to the meeting, etc.), meeting recording options, etc. The network services servers 214 may then create and store a meeting record for the scheduled meeting. When the scheduled meeting time arrives (or within a threshold period of time in advance), the network services server(s) 214 may accept requests to join the meeting from various users.

[0075] To handle requests to join a meeting, the network services server(s) 214 may receive meeting information, such as a meeting ID and passcode, from one or more client devices 220-250. The network services server(s) 214 locate a meeting record corresponding to the provided meeting ID and then confirm whether the scheduled start time for the meeting has arrived, whether the meeting host has started the meeting, and whether the passcode matches the passcode in the meeting record. If the request is made by the host, the network services server(s) 214 activates the meeting and connects the host to a real-time media server 212 to enable the host to begin sending and receiving multimedia streams.

[0076] Once the host has started the meeting, subsequent users requesting access will be admitted to the meeting if the meeting record is located and the passcode matches the passcode supplied by the requesting client device 220-250. In some examples additional access controls may be used as well. But if the network services server(s) 214 determines to admit the requesting client device 220-250 to the meeting, the network services server 214 identifies a real-time media server 212 to handle multimedia streams to and from the requesting client device 220-250 and provides information to the client device 220-250 to connect to the identified real-time media server 212. Additional client devices 220-250 may be added to the meeting as they request access through the network services server(s) 214.

[0077] After joining a meeting, client devices will send and receive multimedia streams via the real-time media servers 212, but they may also communicate with the network services servers 214 as needed during meetings. For example, if the meeting host leaves the meeting, the network services server(s) 214 may appoint another user as the new meeting host and assign host administrative privileges to that user. Hosts may have administrative privileges to allow them to manage their meetings, such as by enabling or disabling screen sharing, muting or removing users from the meeting, assigning or moving users to the mainstage or a breakout room if present, recording meetings, etc. Such functionality may be managed by the network services server(s) 214.

[0078] For example, if a host wishes to remove a user from a meeting, they may select a user to remove and issue a command through a user interface on their client device. The command may be sent to a network services server 214, which may then disconnect the selected user from the corresponding real-time media server 212. If the host wishes to remove one or more participants from a meeting, such a command may also be handled by a network services server 214, which may terminate the authorization of the one or more participants for joining the meeting.

[0079] In addition to creating and administering on-going meetings, the network services server(s) 214 may also be responsible for closing and tearing-down meetings once they have been completed. For example, the meeting host may issue a command to end an on-going meeting, which is sent to a network services server 214. The network services server 214 may then remove any remaining participants from the meeting, communicate with one or more real time media servers 212 to stop streaming audio and video for the meeting, and deactivate, e.g., by deleting a corresponding passcode for the meeting from the meeting record, or delete the meeting record(s) corresponding to the meeting. Thus, if a user later attempts to access the meeting, the network services server(s) 214 may deny the request.

[0080] Depending on the functionality provided by the chat and video conference provider, the network services server(s) 214 may provide additional functionality, such as by providing private meeting capabilities for organizations, special types of meetings (e.g., webinars), etc. Such functionality may be provided according to various examples of video conferencing providers according to this description.

[0081] Referring now to the video room gateway servers 216, these servers 216 provide an interface between dedicated video conferencing hardware, such as may be used in dedicated video conferencing rooms. Such video conferencing hardware may include one or more cameras and microphones and a computing device designed to receive video and audio streams from each of the cameras and microphones and connect with the chat and video conference provider 210. For example, the video conferencing hardware may be provided by the chat and video conference provider to one or more of its subscribers, which may provide access credentials to the video conferencing hardware to use to connect to the chat and video conference provider 210.

[0082] The video room gateway servers 216 provide specialized authentication and communication with the dedicated video conferencing hardware that may not be available to other client devices 220-230, 250. For example, the video conferencing hardware may register with the chat and video conference provider when it is first installed and the video room gateway may authenticate the video conferencing hardware using such registration as well as information provided to the video room gateway server(s) 216 when dedicated video conferencing hardware connects to it, such as device ID information, subscriber information, hardware capabilities, hardware version information etc. Upon receiving such information and authenticating the dedicated video conferencing hardware, the video room gateway server(s) 216 may interact with the network services servers 214 and real-time media servers 212 to allow the video conferencing hardware to create or join meetings hosted by the chat and video conference provider 210.

[0083] Referring now to the telephony gateway servers 218, these servers 218 enable and facilitate telephony devices' participation in meetings hosted by the chat and video conference provider 210. Because telephony devices communicate using the PSTN and not using computer networking protocols, such as TCP/IP, the telephony gateway servers 218 act as an interface that converts between the PSTN, and the networking system used by the chat and video conference provider 210.

[0084] For example, if a user uses a telephony device to connect to a meeting, they may dial a phone number corresponding to one of the chat and video conference provider's telephony gateway servers 218. The telephony gateway server 218 will answer the call and generate audio messages requesting information from the user, such as a meeting ID and passcode. The user may enter such information using buttons on the telephony device, e.g., by sending dual-tone multi-frequency (“DTMF”) audio streams to the telephony gateway server 218. The telephony gateway server 218 determines the numbers or letters entered by the user and provides the meeting ID and passcode information to the network services servers 214, along with a request to join or start the meeting, generally as described above. Once the telephony client device 250 has been accepted into a meeting, the telephony gateway server is instead joined to the meeting on the telephony device's behalf.

[0085] After joining the meeting, the telephony gateway server 218 receives an audio stream from the telephony device and provides it to the corresponding real-time media server 212 and receives audio streams from the real-time media server 212, decodes them, and provides the decoded audio to the telephony device. Thus, the telephony gateway servers 218 operate essentially as client devices, while the telephony device operates largely as an input/output device, e.g., a microphone and speaker, for the corresponding telephony gateway server 218, thereby enabling the user of the telephony device to participate in the meeting despite not using a computing device or video.

[0086] It should be appreciated that the components of the chat and video conference provider 210 discussed above are merely examples of such devices and an example architecture. Some video conference providers may provide more or less functionality than described above and may not separate functionality into different types of servers as discussed above. Instead, any suitable servers and network architectures may be used according to different examples.

[0087] In some examples according to the present disclosure, a user may select an option to use one or more optional AI features available from the virtual conference provider. The use of these optional AI features may involve providing the user's personal information to the AI models underlying the AI features. The personal information may include the user's contacts, calendar, communication histories, video or audio streams, recordings of the video or audio streams, transcripts of audio or video conferences, or any other personal information available the virtual conference provider. Further, the audio or video feeds may include the user's speech, which includes the user's speaking patterns, cadence, diction, timbre, and pitch; the user's appearance and likeness, which may include facial movements, eye movements, arm or hand movements, and body movements, all of which may be employed to provide the optional AI features or to train the underlying AI models.

[0088] Before capturing and using any such information, whether to provide optional AI features or to providing training data for the underlying AI models, the user may be provided with an option to consent, or deny consent, to access and use some or all of the user's personal information. In general, the goal is to invest in AI-driven innovation that enhances user experience and productivity while prioritizing trust, safety, and privacy. Without the user's explicit, informed consent, the user's personal information will not be used with any AI functionality or as training data for any AI model. Additionally, these optional AI features are turned off by default account owners and administrators control whether to enable these AI features for their accounts, and if enabled, individual users may determine whether to provide consent to use their personal information.

[0089] As can be seen in FIG. 3, a user has engaged in a video conference and has selected an option to use an available optional AI feature. In response, the GUI has displayed a consent authorization window for the user to interact with. The consent authorization window informs the user that their request may involve the optional AI feature accessing multiple different types of information, which may be personal to the user. The user can then decide whether to grant permission or not to the optional AI feature generally, or only in a limited capacity. For example, the user may select an option to only allow the AI functionality to use the personal information to provide the AI functionality, but not for training of the underlying AI models. In addition, the user is presented with the option to select which types of information may be shared and for what purpose, such as to provide the AI functionality or to allow use for training underlying AI models.

[0090] Referring now to FIG. 4, FIG. 4 shows an example of an operating environment 400 for generating virtual characters for videoconferencing, according to certain aspects described herein. The operating environment 400 includes a chat and video conference provider 410 configured to host and provide various functionalities of video conferences, such as the chat and video conference provider 110 and the chat and video conference provider 210 described above with respect to FIGS. 1 and 2, respectively. The chat and video conference provider 410 is configured to receive and deliver videoconferencing streams to client computing devices 450A-450C (which may be referred to herein individually as a client computing device 450 or collectively as the client computing devices 450). The video conferencing streams can include video signals of the participants, audio signals captured at respective client computing devices associated with the participants, and other signals or streams regarding the participants. The client computing devices 450 may be the client devices 140-180 and 220-250 discussed above with respect to FIGS. 1 and 2.

[0091] The operating environment 400 further includes a virtual character base model generation system 412. The virtual character base model generation system 412 is configured to generate various base models for generating a virtual character customized for each user, such as virtual character face bases 434 and virtual character expression bases 436. The virtual character face bases 434 represent different face shapes and the virtual character expression bases 436 represent different facial expressions. The virtual character face bases A_id 434 and virtual character expression bases A_exp 436 for each virtual character face model 432 can be stored in a data store 430 along with other models used to generate these base models such as the virtual character face models 432, the human base face S₀ 442, the human face bases S_id 444, and the human expression bases S_exp 446. Additional details regarding generating the virtual character face bases A_id 434 and the virtual character expression bases A_exp 436 are provided later.

[0092] In order for the virtual character to be generated and customized for use in video conferencing, the virtual character base model generation system 412 can send the virtual character face bases 434 and the virtual character expression bases 436 to each client device 450 that supports the virtual character rendering for video conferencing. At a client device 450, the meeting client application 404 that is configured to allow a user to join a video conference can process the virtual character face bases 434 and the virtual character expression bases 436. For example, the meeting client application 404 can access an image of the user associated with the client device 450 to identify a facial feature vector α for the user. The facial feature vector α can represent the shape of the user face and can indicate the weights of respective virtual character face bases A_id434 in contributing to the user face shape. Based on the facial feature vector α, the meeting client application 404 can generate a virtual character neutral face A_N 454 by combining the virtual character face model A₀ 432 selected by the user with the virtual character face bases A_id 434 according to the facial feature vector α.

[0093] The meeting client application 404 can further generate a set of virtual character expressions 456 for the user associated with the client device 450 by combining the virtual character expression bases A_exp 436 based on the facial feature vector α of the user. The virtual character neutral face 454 and the virtual character expressions 456 can be generated before the video conference starts. During the video conference, the meeting client application 404 can detect the facial expression of the user based on the video of the user to determine an expression feature vector β 452. The expression feature vector 452 can be used to incorporate the proper facial expression into the generated virtual character based on the facial expression of the user in real time or near real time. The generated virtual character A can be rendered and the rendered virtual character 440 can be included in the video of the user and transmitted as the meeting content 456 along with other data such as audio signals to the chat and video conference provider 410. The above process can be repeated if the user chooses a different virtual character face model A₀.

[0094] Referring now to FIG. 5, FIG. 5 shows an example of generating a customized virtual character for a participant of a video conference based on a virtual character selection, according to certain aspects described herein. The example process shown in FIG. 5 involves actions performed in three phases: an offline one-time optimization, an offline one-time virtual character preparation, and an online virtual character customization. Assuming the participant has selected a virtual character represented by the virtual character face model A₀ 532. The offline one-time optimization may be performed by the virtual character base model generation system 412 and can involve a deformation transfer 502 to generate virtual character face bases A_id 534 and virtual character expression bases A_exp 536 for the virtual character face model A₀ 532.

[0095] The virtual character face bases A_id 534 and the virtual character expression bases A_exp 536 can be generated based on given models such as a human base face S₀ 542, a set of human face bases S_id 544, and a set of human expression bases S_exp 546. A human base face S₀ 542 is a 3D face model representing the basic features of a neutral human face in a standard shape, including the eyes, the mouth, the nose, etc. Each human face base in the set of human face bases S_id 544 represents a particular type of face shape, such as a round face shape, a rectangular face shape, a pear-shaped face, etc. Each human expression base in the set of human expression bases 546 represents a particular type of facial expression, such as eye blinking, mouth opening, smiling, anger, etc.

[0096] FIG. 6 shows an example of the human base face S₀, a set of human face bases S_id, and a set of human expression bases S_exp. As shown in FIG. 6, the human base face S₀ is a neutral face with standard face shape and size. The human face bases S_id in this example includes nine different human face bases, each of them representing different face shapes and sizes. The human expression bases S_exp in this example includes nine different human facial expressions, such as yawning, blinking left eye, blinking right eye, smiling, etc.

[0097] Similar to human faces, each virtual character face base in the set of virtual character face bases A_id 534 represents a particular type of face shape for a virtual character face model A₀. Each virtual character expression base in the set of virtual character expression bases A_exp 536 represents a particular type of virtual character facial expression. FIG. 6 also shows an example of the virtual character base face A₀, a set of virtual character face bases A_id, and a set of virtual character expression bases A_exp. As shown in FIG. 6, the virtual character face model A₀ is a neutral face with standard face shape and size. The virtual character face bases A_id 634 in this example includes nine different virtual character face bases, each of them representing different face shapes and sizes and corresponding to the nine different human face bases in S_id644. The virtual character expression bases A_exp 636 in this example includes 9×9 different virtual character expression bases organized into nine groups each group corresponding to one virtual character face base and having nine different facial expressions matching to the nine human facial expressions in S_exp 646. In other words, for each virtual character face basis in A_id 634, there is a group of nine virtual character expression bases. Because there are nine different virtual character face bases in A_id 634, there are nine group of virtual character expression bases in A_exp636.

[0098] Referring back to FIG. 5, for the selected virtual character face model 532, the virtual character base model generation system 412 can generate the set of virtual character face bases A_id 534 based on the human base face S₀ 542 and the set of human face bases S_id 544. By combining the human base face S₀ 542 with each of the set of the human face bases S_id 544, differently shaped human faces can be generated. To generate virtual character face bases that represent different virtual character face shapes, the virtual character base model generation system 412 can perform a deformation transfer 502 on the virtual character face model 532 according to a deformation transfer that deforms the human base face S₀ 542 into the respectively shaped human faces.

[0099] Denoting the deformation transfer as Ã=(A, S, {tilde over (S)}), where A is the source virtual character model; à is the target virtual character model; S is the source human face model and S is the target human face model. The deformation transfer Ã=(A, S, {tilde over (S)}) deforms the source virtual character model A into the target virtual character model à in a way similar to how the source human face model S is deformed into the target human face model {tilde over (S)}.

[0100] For instance, the source human face model S can be the human base face model S₀ in the 3DMM, and the target human face model S can be a combination of the human base face model S₀ and a human face basis

[0101] $S_{\mathrm{id}}^j$
in the 3DMM, i.e.,

[0102] $\tilde{S}=S_0+s_{id}^j.$
For a source virtual character model A=A₀, the deformation can generate a target virtual character Ã^j and the 3D virtual character face basis (or virtual character PCA model)

[0103] $A_{id}^j$
can be derived as

[0104] $A_{\mathrm{id}}^j={\tilde{A}}^j-A.$
As a result, the 3D virtual character face bases 534 (or virtual character PCA models)

[0105] $A_{\mathrm{id}}=\left[A_{\mathrm{id}}^1,A_{\mathrm{id}}^2,\dots ,A_{\mathrm{id}}^j,\dots ,A_{id}^{J_{\mathrm{id}}}\right],$
with J_id being the number of human face PCA models in S_id, can be derived.

[0106] To derive the virtual character expression bases A_exp 536, the source human face model S can be the combination of the human base face model S₀ and a human face basis

[0107] $S_{id}^j$
in the 3DMM, i.e.,

[0108] $S=S_0+S_{\mathrm{id}}^j.$
The target human face model S can be a combination of the human base face model S₀, the human face basis

[0109] $S_{id}^j,$
and a human expression basis

[0110] $S_{\exp }^i$
in the 3DMM, i.e.,

[0111] $\tilde{S}=S_0+S_{\mathrm{id}}^j+S_{\exp }^i.$
For a source virtual character model

[0112] $A=A_0+A_{\mathrm{id}}^j,$
the deformation can generate a target virtual character Ã^ij and the 3D virtual character expression basis

[0113] $A_{\exp }^{\mathrm{ij}}$
can be derived as

[0114] $A_{\exp }^{\mathrm{ij}}={\tilde{A}}^{\mathrm{ij}}-\left(A_{\mathrm{id}}^j+A_0\right).$
As a result, the 3D virtual character expression bases 536

[0115] $A_{\exp }=\left[A_{\exp }^1;A_{\exp }^2;\dots ;A_{\exp }^i;\dots ,A_{\exp }^I\right]=\left[\left[A_{\exp }^{11},A_{\exp }^{12},\dots ,A_{\exp }^{1J}\right];\left[A_{\exp }^{21},A_{\exp }^{22},\dots ,A_{\exp }^{2J_{\mathrm{id}}}\right];\dots \left[A_{\exp }^{I1},A_{\exp }^{I2},\dots ,A_{\exp }^{{\mathrm{IJ}}_{\mathrm{id}}}\right]\right]$
can be derived, where J_id is the number of human face PCA models and I is the number of human expression bases S_ex.

[0116] With the generated virtual character face bases A_id 534 and virtual character expression bases A_exp 536, the offline one-time virtual character preparation can be performed. This process may be performed by the client device 450 associated with the user and can involve generating the facial feature vector α for the participant and generating a virtual character neutral face A_N 554 and virtual character expressions A_exp(α) 556 for the virtual character face model A₀ 532 and the participant.

[0117] As discussed above, the facial feature vector α can represent the shape of the face of the participant and can be extracted from an image of the participant. An example of generating the facial feature vector is described in U.S. Patent Application Publication 2023/0260184A1, the entirety of which is incorporated herein by reference. In some examples, the dimension of the facial feature vector α 552 is the same as the number of the virtual character face bases A_id 534 and each value in the facial feature vector α 552 indicates the relative value of the corresponding virtual character face basis in contributing the face shape of the participant. As such, the facial feature vector α indicates the weights of respective virtual character face bases A_id 434 in contributing to the user face shape. Based on the facial feature vector α, the meeting client application 404 can generate a virtual character neutral face A_N 554 by combining the virtual character face model A₀ 532 with the virtual character face bases A_id 534 according to the facial feature vector α, such as

[0118] $\begin{array}{cc}{A}_N=A_0+\alpha A_{id}=A_0+\sum _{j=1}^{J_{id}}{\alpha }_j{A}_{id}^j.& \left(\mathrm{X1}\right)\end{array}$

[0119] The facial feature vector α can also be used to generate the virtual character expressions A_exp(α) 556 and each model in the A_exp(α) 556 can represent a facial expression for the face shape of the participant. For example,

[0120] $\begin{array}{cc}{A}_{exp}^i\left(\alpha \right)=\sum _{j=1}^{J_{\mathrm{id}}}{\alpha }_j{A}_{exp}^{ij}.& \left(\mathrm{X2}\right)\end{array}$

[0121] The generated virtual character neutral face A_N 554 and virtual character expressions A_exp(α) 556 can be used during online meeting to generate customized virtual character for the participant with the selected virtual character face model A₀. For example, as the meeting is conducted, the meeting video 504 of the participant is captured, for example, by a camera of the client device 450. The client device 450 can further extract an expression vector β 558 from the meeting video 504 by detecting the facial expression of the participant. An example of extracting an expression vector from a meeting video is described in U.S. Patent Application Publication 2023/0260184A1, the entirety of which is incorporated herein by reference. The client device 450 can perform the virtual character generation 506 using the expression feature vector β 558 to incorporate the facial expression into the generated virtual character face model based on the facial expression of the participant. For example, the client device 450 can generate a customized virtual character A_C during the virtual character generation 506 by:

[0122] $\begin{array}{cc}{A}_C=A_0+\alpha A_{id}+\beta A_{exp}\left(\alpha \right)=A_0+\sum _{j=1}^{J_{id}}{\alpha }_j{A}_{id}^j+\sum _{i=1}^I{\beta }_i{A}_{exp}^i\left(\alpha \right)& \left(\mathrm{X3}\right)\end{array}$
The generated virtual character A_C can be rendered during the virtual character rendering 508 to generate the video of the participant for transmission to other participants of the meeting. Generating the expression vector β 558, the virtual character generation 506, and the virtual character rendering 508 can be repeated periodically as the meeting is being conducted, such as every 40 milliseconds.

[0123] While the above example describes the process of generating the customized virtual character for one virtual character A₀, the same process can be performed for other virtual characters. For example, the virtual character base model generation system 412 can generate the virtual character face bases 434 and the virtual character expression bases 436 for different virtual character face bases 432 as described above. These different sets of virtual character face bases 434 and virtual character expression bases 436 can be transmitted to each client device 450. In this way, each user may choose to switch to a different virtual character face model A₀ during the meeting, and the corresponding customized virtual character can be generated and updated in real time or near real time.

[0124] Referring now to FIG. 7, FIG. 7 shows examples of components of a virtual character A₀, according to certain aspects described herein. Sub-figure (α) shows the main virtual character component of the virtual character (also referred to as the “main component” and denoted as W₁(A₀)) which includes the face, nose, mouth, ears, etc. Sub-figure (b) shows other components of the virtual character (denoted as W_j(A₀), j>1), such as the left eyeball, the right eyeball, upper gum and upper teeth, lower gum and lower teeth, etc. Sub-figure (c) shows the other components W_j(A₀), j>1 with the main virtual character component W₁(A₀) shown in a wire frame. Sub-figure (d) shows all components W (A₀), j>1 of the virtual character A₀.

[0125] The main component W₁(A₀) is the largest component overlapping with the face region. W₁(A₀) may contain interior mouth and eye regions as shown in the wireframe of sub-figure (c). FIG. 8 shows the interior region separation on the main virtual character component W₁(A₀). Sub-figure (α) shows the external region in a surface form with the interior regions shown in blue. Sub-figure (b) shows the external region in a wireframe form with the interior regions shown in blue. Sub-figure (c) shows the external region in a wireframe form with the interior regions shown in a surface form. Sub-figure (d) shows the interior regions in a surface form. Sub-figures (e) and (f) of FIG. 8 further show an example of dividing the mouth of the virtual character into two regions: the mouth upper region L_upper(A₀) in green and the mouth lower region L_lower(A₀) in orange. As will be discussed later, dividing the mouth of the virtual character into the upper and lower regions can facilitate the deformation to avoid preventing the mouth opening in the deformed virtual character.

[0126] Because W₁(A₀) may also contains interior structures such as the mouth and the eye socket as shown in FIG. 8, the main component W₁(A₀) cannot be used as the face region of the virtual character to be deformed directly along with the human face model. In other words, the human face model does not have the interior structures and the deformation transfer generated using the human face models does not provide information on how to deform the interior structures. If the main component W₁(A₀) is deformed directly by following the deformation of the human face model, only the face region will be deformed and the interior structures will remain unchanged, which may cause artifacts in the generated virtual character. As used herein, the face region of a virtual character face model (such as the virtual character face model 532) refers to a region including facial features such as noses and lips and excluding interior regions such as interior mouth and eye regions. The non-face regional refers to the remaining parts of the virtual character face model including the interior regions such as interior mouth and eye regions of the virtual character and other components W (A₀), j>1.

[0127] To allow the virtual character face model A₀ to be deformed properly, virtual triangles can be generated to connect the face region with the non-face region of the virtual character face model A₀ to propagate the deformation from the face region to the non-face region thereby allowing the non-face region to be deformed properly. Likewise, virtual triangles can also be generated between two components or regions within the non-face region of the virtual character to allow the deformation to be propagated to various components of the non-face region.

[0128] Denote the virtual triangles connecting one region W_i to another region W_j as

[0129] $V_t^1\left(W_i,W_j\right)$
and initialize

[0130] $V_t^1\left(W_i,W_j\right)=\varnothing .$
Determine a threshold δ(W_i, W_j) used for generating the virtual triangles as

[0131] $\begin{array}{cc}\delta \left(W_i,W_j\right)=d_{\min }\left(W_i,W_j\right)+{\delta }_1\left(d_{\max }\left(W_i,W_j\right)-d_{\min }\left(W_i,W_j\right)\right)& \left(\mathrm{X4}\right)\end{array}$
Here, δ₁ is a hyper parameter indicating the ratio for defining virtual triangles between two regions. d_min(W_i, W_j), d_max(W_i, W_j) are the min and max distances between surfaces of the two regions W_i, W_j and can be calculated by

[0132] $\begin{array}{cc}{d}_{\min }\left(W_i,W_j\right)=\underset{p\in V\left(W_i\right),F\in F\left(W_j\right)}{\min }{d}_{\min }\left(p,F\right)& \left(\mathrm{X5}\right)\end{array}$ $\begin{array}{cc}{d}_{\max }\left(W_i,W_j\right)=\underset{p\in V\left(W_i\right),F\in F\left(W_j\right)}{\max }{d}_{\max }\left(p,F\right).& \left(\mathrm{X6}\right)\end{array}$
Here, d_min(p, F) and d_max(p, F) are the minimum and maximum distances from one vertex p to a triangle Γ with three vertices (v₀, v₁, v₂), respectively, and can be defined as

[0133] $\begin{array}{cc}{d}_{\min }\left(p,F\right)=\underset{\begin{array}{c}\left(r_0,r_1,r_2\right)\\ r_0+r_1+r_2=1\\ r_i\ge 0\end{array}}{\min }p-\sum _{i=0}^2{r}_i{v}_i,& \left(\mathrm{X7}\right)\end{array}$ $\begin{array}{cc}{d}_{\max }\left(p,F\right)=\underset{\begin{array}{c}\left(r_0,r_1,r_2\right)\\ r_0+r_1+r_2=1\\ r_i\ge 0\end{array}}{\max }p-\sum _{i=0}^2{r}_i{v}_i,& \left(\mathrm{X8}\right)\end{array}$
where r₀, r₁, r₂ is the barycentric coordinates of a vertex on F. V(W_i) denotes the vertices of W_i and F(W_j) denotes the triangles of W_j.

[0134] For every vertex p∈V(W_i) and every triangle (v₀, v₁, v₂)∈F(W_j), if the vertex-triangle distance in Eq. (X7) is smaller than the threshold in Eq. (X4), i.e.

[0135] $\begin{array}{cc}{d}_{\min }\left(p,\left(v_0,v_1,v_2\right)\right)<\delta \left(W_i,W_j\right),& \left(\mathrm{X9}\right)\end{array}$
the following three triangles are added to

[0136] $V_t^1\left(W_i,W_j\right)$

[0137] $\begin{array}{cc}\left(p,v_1,v_2\right),\left(v_0,p,v_2\right),\left(v_0,v_1,p\right)& \left(\mathrm{X10}\right)\end{array}$
Note that the threshold in Eq. (X4) is adaptive to the size of the region. With W_i fixed, the threshold for a larger W_i is bigger. As a result, more virtual triangles will be generated. For example, if W₁ is the face region, W₂ is the pupil region, and W₃ is the interior mouth region, the number of virtual triangles from W₃ to W₁, i.e.

[0138] $V_t^1\left(W_3,W_1\right),$
will be larger than the number of virtual triangles from W₂ to W₁, i.e.

[0139] $V_t^1\left(W_2,W_1\right).$
FIG. 9 shows, in sub-figure (α), virtual triangles (in yellow) between the non-face region, the interior mouth region in this example (in green), and the face region (in blue) from the viewpoint on top of the mouth.

[0140] The generated virtual triangles can be classified into two groups G_0,0 and G_0,1: if a virtual triangle connects the mouth upper region L_upper(A₀) and mouth lower region L_lower(A₀), the virtual triangle is put into G_0,0, otherwise, it is put into G_0,1. Each virtual triangle in G_0,0 satisfies either one of the following conditions: has one vertex on L_upper(A₀) and two vertices on L_lower(A₀) or has two vertices on L_upper(A₀) and one vertex on L_lower(A₀). Sub-figure (b) of FIG. 9 shows examples of virtual triangles in G_0,0 and G_0,1. In sub-figure (b), virtual triangles shown in blue are between the mouth upper region and the mouth lower region and thus they are put into G_0,0. The remaining virtual triangles are shown in yellow and thus are included in G_0,1.

[0141] To generate a target deformed virtual character, a two-stage process can be employed. The two-stage deformation transfer algorithm is formulated as follows:

[0142] Stage 1: deform the face region of the virtual character Δ=δ(A₀) while other regions of A₀ are fixed:

[0143] $\begin{array}{cc}\begin{array}{cc}\hat{A}=ℱ\left(A,S,\tilde{S}\right)& s.t.\tilde{A}-\end{array}\Delta \left(\tilde{A}\right)=A-\Delta \left(A\right);& \left(\mathrm{X11}\right)\end{array}$

[0144] Stage 2: deform A while keeping its face region fixed as

[0145] $\begin{array}{cc}\tilde{A}=ℱ(\hat{A},S,\tilde{S})s.t.\Delta \left(\tilde{A}\right)=\Delta \left(\hat{A}\right)& \left(\mathrm{X12}\right)\end{array}$
In some examples, the deformation transfer in Eq.(X11) and Eq.(X12) can be formulated as

[0146] $\begin{array}{cc}\begin{array}{cc}\hat{A}=\underset{V\left(\hat{A}\right)}{\arg \min }& E\left(\hat{A}\right)=\sum _{i\in C_F}{\lambda }_i{E}_i\left(\hat{A}\right)\\ s.t.& V_i\left(\hat{A}\right)=V_i\left(A\right),\forall i\in L_v^{\mathrm{fix}}\left(A\right)\bigcup {\rho }_v\left(\Delta ,S_0,\tilde{S}\right)\bigcup {\stackrel{\_}{K}}_{\Delta ,v}.\end{array}& \left(\mathrm{X13}\right)\end{array}$ $\begin{array}{cc}\begin{array}{cc}\hat{A}=\underset{V\left(\hat{A}\right)}{\arg \min }& E\left(\hat{A}\right)=\sum _{i\in C_{\mathrm{NF}}}{\lambda }_i{E}_i\left(\hat{A}\right)\\ s.t.& V_i\left(\hat{A}\right)=V_i\left(A\right),\forall i\in L_v^{\mathrm{fix}}\left(A\right)\bigcup {\rho }_v\left(\Delta ,S_0,\tilde{S}\right)\bigcup K_{\Delta ,v}.\end{array}& \left(\mathrm{X14}\right)\end{array}$
where λ_i are hyper parameters for various terms in the loss function E(·) in Eq. (X13) and Eq. (X14). The loss function E(Â) and E(Ã) in Eq. (X13) and Eq. (X14) may also be referred to as objective functions and terms E_i(·) may be referred to as constraints. K_α,v, are the vertices' indices of the face region and K_α,v, are the vertices' indices of the non-face region with

[0147] $\begin{array}{cc}{\overline{K}}_{\Delta ,v}=\bigcup _j{\stackrel{\_}{K}}_{\Delta ,v,j}.& \left(\mathrm{X15}\right)\end{array}$
where K_α,v,j includes vertices' indices of the non-face region on the j-th component {W_j(A)}.

[0148] $L_v^{fix}\left(A\right)$
represents the fixed vertices indicating vertices in the non-face region to fixed during deformation transfer

[0149] $\begin{array}{cc}{L}_v^{\mathrm{fix}}\left(A\right)=\left\{L_{v,i}^{\mathrm{fix}}\left(A_0\right)\right\},& \left(\mathrm{X16}\right)\end{array}$
where

[0150] $L_{v,i}^{fix}\left(A_0\right)$
is the i-th fixed vertex. For example, vertices on the neck and back of the virtual character can be fixed vertices. These fixed vertices may be manually labelled. ρ_v(Δ, S, {tilde over (S)}) is the set of fixed vertices in the face region Δ of the virtual character to be fixed during the deformation transfer. C_Γ is a set containing the indices of constraints related to face region; and C_NF is a set containing the indices of constraints related to non-face region. As will be discussed in more detail below, the minimization problem in Eq. (X13) (or Eq. (X14)) can be solved by reformulating the loss function to a linear form and solving it through a linear solver.

[0151] The constraints related to face region can include, for example, enforcing the difference between the deformation gradient of triangles in the stage-one deformed virtual character T_j(A, Â) and the deformation gradient of corresponding triangles in the human face model T_j(S, {tilde over (S)}) to be as small as possible. This constraint can be used to make sure the deformation of A into à using the deformation gradient from S to {tilde over (S)}. The constraints can further include enforcing the triangles on A without any correspondence from the source model S to remain unchanged, i.e., having identity deformation gradient. Other constraints may also be used. Examples of the face region related constraints are discussed below in detail.

[0152] In some examples, the constraints related to non-face region C_NF can include two constraints defined based on the virtual triangles. For example, the following two constraints can be defined:

[0153] $\begin{array}{cc}{E}_{18}=\delta \left(L_{c,4}\left(\tilde{S}\right),L_{c,5}\left(\tilde{S}\right)\right)·\sum _{j\in G_{0,0}}{T_j\left(\hat{A},\tilde{A}\right)-I_3}^2& \left(\mathrm{X17}\right)\end{array}$ $\begin{array}{cc}{E}_{19}\left(\tilde{A}\right)=\sum _{j\in G_{0,1}}{T_j\left(\hat{A},\tilde{A}\right)-I_3}^2& \left(\mathrm{X18}\right)\end{array}$
Here, I₃ is an identify matrix defined as

[0154] $\begin{array}{cc}{I}_3=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right),& \left(\mathrm{X19}\right)\end{array}$
and T_j (Â, Ã) is the deformation gradient from  to à for triangle j.

[0155] As discussed above, G_0,0 contains virtual triangles connecting the mouth upper region L_upper(A₀) of the virtual triangle and the mouth lower region L_lower(A₀) of the virtual character. G_0,1 contains virtual triangles connecting two different regions other than those in G_0,0. δ(L_c,4({tilde over (S)}), L_c,5({tilde over (S)})) indicates whether the mouth is close or not with one indicating the mouth is closed and 0 otherwise. δ(L_c,4({tilde over (S)}),L_c,5({tilde over (S)})) will be discussed in detail later. During the second stage of the optimization process, E₁₈ (Ã) or E₁₉ (Ã) or both can be enabled by adjusting the value of corresponding hyperparameters λ₁₈ and λ₁₉.

[0156] FIG. 10 shows examples of the deformation results for a virtual character with different values of the hyperparameters λ₈ and λ₁₉. In FIG. 10, the virtual triangles between the eyeballs and the face region will drive the deformation of the eyeballs following the deformation of the face region. Without the terms in Eq.(X17) and Eq.(X18), the non-face regions connecting to the face region, such as the interior mouth region (the green region in (α) of FIG. 9), can deform without virtual triangles. In FIG. 9, the deformation gradient will propagate from the lip's regions to the interion mouth region via the red region. The interior mouth region may intersect with the face region (the blue region in FIG. 9). As such, a gap can be created between the face region and the non-face region before the virtual triangles are generated and before the deformation. As shown in FIG. 9, a gap is created between the green region and the blue region before the deformation. However, when the virtual character deforms without the terms in Eq.(X17) and Eq.(X18), the two regions may intersect with each other. With the terms in Eq.(X17) and Eq.(X18), the virtual triangles (yellow triangles in FIG. 9) maintain the gap between the two regions, thus eliminate the intersection between the two regions. The results in the first row of FIG. 10 are generated with a small weight λ₁₈=Δ₁₉=0.01. As can be seen from FIG. 10, a small value of λ₁₈ and λ₁₉ is unable to prevent the intersection (indicated as the black areas in the face region). As the weight λ₁₈=Δ₁₉ increases, the interior regions are pushed away from the face region (the second and the third rows of FIG. 10). Besides increasing weights λ₁₈=Δ₁₉, another way to prevent the self-intersection is increasing the hyper parameter δ₁ to increase the number of virtual triangles.

[0157] As shown in sub-figure (b) of FIG. 9, virtual triangles in G_0,0 connect the upper region of the mouth to the lower region of the mouth as defined in sub-figure (e) and sub-figure (f) of FIG. 8. Forcing these triangles to remain unchanged during deformation will prevent the mouth from opening. Thus, if the mouth is open on the source virtual character, i.e. δ(L_c,4({tilde over (S)}), L_c,5({tilde over (S)}))=0, virtual triangles in G_0,0 should be ignored (as formulated in Eq. (X17), and the correct result shown in (d) of FIG. 9 can be obtained. If δ(L_c,4({tilde over (S)}), L_c,5 ({tilde over (S)})) is removed and Eq.(X17) is replaced with the following:

[0158] $\begin{array}{cc}{E^{\prime }}_{18}\left(\tilde{A}\right)=\sum _{j\in G_{0,0}}{T_j\left(\hat{A},\tilde{A}\right)-I_3}^2.& \left(\mathrm{X20}\right)\end{array}$

[0159] Incorrect deformation results will be generated as shown in sub-figure (d) of FIG. 9. In sub-figure (d) of FIG. 9, virtual triangles in G_0,0 connecting upper teeth and lower teeth try to remain the original shape following Eq.(X20). As a result, the upper teeth are forced to connected to the lower teeth.

[0160] Alternative, or in addition to E₁₈(Ã) and E₁₉(Ã), other constraints may also be used in the second stage for the non-face region. For example, the two-stage deformation process can be performed without introducing the virtual triangles. Correspondence can be established between pairs of surfaces in the face region and non-face region. In other words, for a triangle surface in the face region, a triangle surface in the non-face region that is close to the face region can be identified. These pairs of corresponding surfaces can be utilized when deforming the non-face region in the second stage to enforce the deformation of the surface in the non-face region to be close to the deformation of the corresponding surface in the face region. In this way, the non-face region can be deformed similarly as the face region, thereby reducing the artifacts in the deformed virtual character. For example, a constraint can be added to the constraints related to non-face region C_NF and be introduced by adding E₂₀ to Eq. (X14) to enable deformation of the non-face region following the deformation of the face region without using virtual triangles:

[0161] $\begin{array}{cc}{E}_{20}\left(\tilde{A}\right)=\sum _{\left(i,j\right)\in G_3}{T_j\left(A,\hat{A}\right)-T_i\left(\hat{A},\tilde{A}\right)}^2& \left(\mathrm{X21}\right)\end{array}$
where G₃ contains pairs of the corresponding triangles between the face region and the non-face region. Additional examples of the non-face region related constraints are discussed below in detail.

[0162] Virtual characters are unreal characters and typically have facial features that are unproportional to other features. For example, the virtual character may have a wider face, larger eyes, a smaller mouth, and a longer nose than those in the human face. Deformation transfer derived from human face models may not lead to the proper deformation for those facial features. To address this problem, facial feature enhancement can be introduced as constraints in the deformation transfer process to ensure the desired facial features of the deformed virtual character to be obtained. To perform facial feature enhancement, feature points/vertices and feature curves on the human base face and the virtual character face model are utilized to define the various features of a face. FIG. 11 shows labelled input data including feature vertices, feature curves, and feature regions on the 3DMM base surface as shown in sub-figure (α) and the corresponding feature vertices, feature curves, and feature regions on the virtual character shown in sub-figure (b). Sub-figures (c) and (d) shows the details of the feature vertices and feature curves which include 78 feature vertices (also referred to as “facial landmarks”) and 9 feature curves defined on the facial landmarks of the 3DMM base surface. The nine feature curves include L_c,0 to L_c,8 defined over respective facial landmarks describing facial features such as right upper/lower eye lid, left upper/lower eye lid, upper mouth, lower mouth, and so on. For example, δ(L_c,4({tilde over (S)}), L_c,5 ({tilde over (S)})) in Eq. (X17) is defined using the feature curve L_c,4 describing the upper mouth and the feature curve L_c,5 describing the lower mouth.

[0163] The labelled feature vertices can be used to facilitate the feature enhancement by defining various auxiliary tools, such as virtual feature triangles. FIG. 12 shows examples of corresponding virtual feature triangles defined over feature vertices on the 3DMM model and a virtual character. The first row shows five virtual triangles on both the 3DMM based face and the virtual character base surface as

[0164] $\begin{array}{cc}\left(L_{v,16},L_{v,0},L_{v,8}\right),\left(L_{v,15},L_{v,1},L_{v,8}\right),& \left(\mathrm{X22}\right)\end{array}$ $\left(L_{v,14},L_{v,2},L_{v,8}\right),\left(L_{v,13},L_{v,3},L_{v,8}\right),\left(L_{v,12},L_{v,4},L_{v,8}\right)$
Similarly, virtual feature triangles to enhance face features such as eyes, nose, and lips can also be defined. The second row of FIG. 12 shows eight virtual feature triangles that can be used to enhance the global deformation of the eye regions.

[0165] $\begin{array}{cc}\left(L_{v,46},L_{v,44},L_{v,48}\right),\left(L_{v,46},L_{v,45},L_{v,47}\right),\left(L_{v,42},L_{v,48},L_{v,44}\right),\left(L_{v,42},L_{v,49},L_{v,43}\right)& \left(\mathrm{X23}\right)\end{array}$ $\left(L_{v,54},L_{v,52},L_{v,56}\right),\left(L_{v,54},L_{v,53},L_{v,55}\right),\left(L_{v,50},L_{v,56},L_{v,52}\right),\left(L_{v,50},L_{v,57},L_{v,51}\right)$
L_v,i=0, . . . , 77 are the facial landmarks shown in FIG. 11.

[0166] For example, if a facial feature is to be enlarged or reduced, the deformation can be performed such that a difference between the deformation for a facial feature of the virtual character and a scaled (up or down) deformation for the corresponding facial feature in the human face is minimized. In some implementations, a constraint can be added to face region related constraint set C_Γ to perform deformation gradient scaling by introducing E₁₂ to the objective function in Eq. (X13):

[0167] $\begin{array}{cc}{E}_{12}\left(\hat{A}\right)=\sum _{\left(i,j,k,s_1,s_2,s_3\right)\in M_g}{T_{\left(L_{v,i}\left(A\right),L_{v,j}\left(A\right),L_{v,k}\left(A\right)\right)}\left(A,\hat{A}\right)-E_{\left(s_1,s_2,s_3\right)}\left(T_{\left(L_{v,i}\left(S_0\right),L_{v,j}\left(S_0\right),L_{v,k}\left(S_0\right)\right)}\left(S_0,\tilde{S}\right)\right)}^2.& \left(\mathrm{X24}\right)\end{array}$
Here, M_g is the scaling triangle data defined as:
(i_l,j_l,k_l,η_l,x,η_l,y,η_l,z)∈Mg(A₀)
with i_l, j_l, k_l defining a virtual feature triangle using feature vertices (L_v,il, L_v,jl, L_v,kl) as corners, and η_l,x, η_l,y, η_l,z being hyper parameters. The deformation gradient for the triangle (L_v,il, L_v,jl, L_v,kl) will be scaled by η_l,x, η_l,y, η_l,z along x-, y-, z-axis, respectively. The virtual feature triangles in M_g may include the virtual feature triangles in Eq. (X22) which can be used to enhance the face of the virtual character or the virtual feature triangles in Eq. (X23) which can be used to enhance the eye area of the virtual character. T_{(Lv,iLv,jLv,k)} is the deformation gradient of a triangle by i-th, j-th, and k-th landmarks. E_(sI,s2,s3) is a scaling operation on the deformation gradient which enhances the scaling along the i-th axis by s_i with i being 1, 2, 3 representing x-, y-, or z-axis, respectively. Detail examples of E_(sI,s2,s3) will be presented later.

[0168] By setting the hyperparameter λ₁₂ for E₁₂ to be a non-zero positive value, the facial feature of the virtual character can be enhanced from the corresponding facial feature of the human face model. In some implementations, the hyperparameter λ₁₂ is set to be a non-zero positive value when generating the virtual character face bases 534 and is set to be zero when generating the virtual expression bases 536 because the deformation gradients for expressions do not need to be enhanced.

[0169] While some facial features of the virtual character need to be enhanced, there are certain facial features need to be preserved during the deformation transfer. For example, when performing deformation transfer to generate virtual character base expression models from human face expression models, facial features, such as closed mouth, closed eye, should be preserved. In other words, if the human facial expression has the mouth or an eye closed, the deformed virtual character base expression model should also have the mouth or eye closed. Other features, such as the shape of the mouth and eyes, the symmetry of the vertices, should also be preserved.

[0170] In some implementations, each feature preservation is introduced as a constraint or a loss term in the loss function to be minimized. For example, a set of virtual feature triangles can be generated based on vertices of a feature curve. A loss term can be defined based on the differences between deformation gradients of the set of virtual feature triangles and an identity deformation gradient so that the deformation of the region near the feature curve is minimized. For instance, a constraint can be added to the face region related constraints C_Γ and a loss term E₁₁ can be added to the loss function in Eq. (X13). A constraint can be added to the non-face region related constraints C_NF and a loss term E₁₆ can be added to the loss function in Eq. (X14). E₁₁ and E₁₆ can be defined as:

[0171] $\begin{array}{cc}{E}_{11}\left(\hat{A}\right)=\sum _{j\in G_{1,0}}{T_j\left(A,\hat{A}\right)-I_3}^2& \left(\mathrm{X25}\right)\end{array}$ $\begin{array}{cc}{E}_{16}\left(\tilde{A}\right)=\sum _{j\in G_{1,1}}{T_j\left(\hat{A},\tilde{A}\right)-I_3}^2.& \left(\mathrm{X26}\right)\end{array}$
Here, I₃ is the identity matrix defined in Eq. (X19). G_1,0 and G_1,1 are virtual feature triangles defined along the constraint curves

[0172] $L_c^{\mathrm{fix}}=\left\{L_{c,i}^{\mathrm{fix}}\right\},\mathrm{where}{L}_{c,i}^{\mathrm{fix}}$
is the i-th constraint curve. FIG. 13 shows examples of the constraint curves. In FIG. 13, there are two constraint curves

[0173] $L_{c,0}^{\mathrm{fix}}$
1302 and

[0174] $L_{c,1}^{\mathrm{fix}}$
1304 used to maintain the mouth shape. In some examples,

[0175] $L_{c,0}^{\mathrm{fix}}$
is set to be the feature curve L_c,4 and

[0176] $L_{c,1}^{\mathrm{fix}}$
is set to be the feature curve L_c,5 shown in FIG. 11. FIG. 13 also shows 3 constraint curves 1306-1310 and 1312-1316 for each pupil to keep the pupil in the circular shape.

[0177] The virtual feature triangles from a constraint curve can be generated as follows. Denote the constraint curve as

[0178] $\begin{array}{cc}L=\left[\begin{array}{cccccc}{v}_1& v_2& \dots & v_i& \dots & v_n\end{array}\right]& \left(\mathrm{X27}\right)\end{array}$
where v₁ is the index of the vertex v_vi on the mesh, n=|L|, and v_viv_vi is an edge of the mesh. For the curve Lc(n−2) virtual feature triangles can be constructed with the i-th triangle as (v₁ v₁₊₁ v₁₊₂) with i=1, . . . , n−2. When deforming a triangulated model from A to Ã, according to Eq.(X21), setting the deformation gradient for these virtual feature triangles as I₃, i.e.

[0179] $\begin{array}{cc}{T}_{(\begin{array}{ccc}{v}_i& v_{i+1}& v_{i+2})\end{array}}\left(A,\tilde{A}\right)=I_3,& \left(\mathrm{X28}\right)\end{array}$
indicates that these triangles do not deform. Thus, the curve L will fix during the deformation.

[0180] G_1,0 and G_1,1 can be generated by diving G₁. G₁ can be initialized as an empty set. For each constraint curve

[0181] $L_{c,i}^{\mathrm{fix}},$
virtual feature triangles

[0182] $V_t^2\left(L_{c,i}^{\mathrm{fix}}\right)$
can be generated as described above and be added to G₁ as

[0183] $\begin{array}{cc}{G}_1=G_1\bigcup V_t^2\left(L_{c,i}^{\mathrm{fix}}\right).& \left(\mathrm{X29}\right)\end{array}$
For each virtual feature triangle in G₁, if all the vertices of the virtual triangle belong to the face region Δ, this virtual feature triangle is added to G_1,0, otherwise, it is added to G_1,1.

[0184] For a closeness feature, the closeness of a facial feature can be defined by a first feature curve and a second feature curve being overlapped. The corresponding loss term can be defined based on a mapping distance between the first feature curve and the second feature curve so that the mapping distance is minimized (thus keeping the facial feature closed as much as possible). For example, a feature curve closing constraint can be added to the face region related constraints C_F. Loss term E₁₀ can be added to the loss function in Eq. (X13). E₁₀ can be defined as:

[0185] $\begin{array}{cc}{E}_{10}\left(\hat{A}\right)=\sum _{\left(i,j\right)\in \left\{\left(0,1\right),\left(2,3\right),\left(4,5\right)\right\}}\left(\sum _{\left(a,b,c,d,e\right)\in N\left(L_{c,i}\left(H\right),L_{c,j}\left(H\right)\right)}b·V_c\left(\hat{A}\right)+d·V_e\left(\hat{A}\right)-V_a\left(\hat{A}\right)\right)\delta \left(L_{c,i}\left(\tilde{S}\right),L_{c,j}\left(\tilde{S}\right)\right)& \left(\mathrm{X30}\right)\end{array}$
where N (L_c,i(A), L_c,j(A)) is the curve mapping, and δ(L_c,i({tilde over (S)}), L_c,j({tilde over (S)})) is a binary function indicating whether the curve L_c,i({tilde over (S)}) overlaps with the curve L_c,j({tilde over (S)}). E₁₀ adds constraints based on three curve pairs L_c,i, L_c,j in FIG. 11 with (i,j)∈{(0,1), (2,3), (4,5)}. Each curve pair controls the closing of a face feature: L_c,0, L_c,1 for the right eye closing; L_c,2, L_c,3 for the left eye closing; and L_c,4, L_c,5 for the mouth closing. Each closing constraint Eq. (X30) will be activated if and only if the corresponding feature is close on the source deformation, i.e., δ(L_c,i({tilde over (S)}), L_c,j({tilde over (S)}))=1. Additional details about Eq. (X30) are provided later.

[0186] In addition to facial features, there are other features of the virtual character that need to be preserved, such as the symmetry of the vertices. For example, a constraint (constraint #13) can be added to the face region related constraints C_Γ and another constraint (constraint #17) can be added to the non-face region related constraints C_NF to enforce the symmetry of the vertices along the horizontal axis. Corresponding loss terms E₁₃ and E₁₇ can be added to the loss function in Eq. (X13) and Eq. (X14), respectively. E₁₃ and E₁₇ can be defined as:

[0187] $\begin{array}{cc}{E}_{13}\left(\hat{A}\right)={\rho }_s\left(S_0,\tilde{S}\right)·\left(\sum _{\begin{array}{c}\left(i,j\right)\in M_s\left(A_0\right)\\ i,j\in K_{\Delta ,v}-L_v^{\mathrm{fix}}\left(A\right)\bigcup {\rho }_v\left(A,S_0,\tilde{S}\right))\end{array}}\left({E_x·V_i\left(\hat{A}\right)-V_j\left(\hat{A}\right)}^2\right)\right)& \left(\mathrm{X31}\right)\end{array}$ $\begin{array}{cc}{E}_{17}\left(\tilde{A}\right)={\rho }_s\left(S_0,\tilde{S}\right)·\left(\sum _{\begin{array}{c}\left(i,j\right)\in M_s\left(A_0\right)\\ i,j\in {\stackrel{\_}{K}}_{\Delta ,v}-L_v^{\mathrm{fix}}\left(A\right)\bigcup {\rho }_v\left(A,S_0,\tilde{S}\right)\end{array}}\left({E_x·V_i\left(\tilde{A}\right)-V_j\left(\tilde{A}\right)}^2\right)\right)& \left(\mathrm{X32}\right)\end{array}$
Here, ρ_s(S₀, {tilde over (S)}) is a binary function used to check whether {tilde over (S)} keeps the symmetric vertex pairs from S₀. ρ_s(S₀, {tilde over (S)}) is 1 if the symmetric vertex pairs are kept, and is 0 otherwise. Ms(A₀) includes all symmetric pair of vertices from A₀. E_x is a 3×3 symmetric matrix.

[0188] $L_v^{\mathrm{fix}}\left(A\right)\bigcup {\rho }_v\left(A,S_0,\tilde{S}\right)$
is the set of fixed vertices which include manually labelled vertices

[0189] $L_v^{\mathrm{fix}}\left(A\right)$
and fixed vertices identified from the deformation ρ_v(A, S₀, {tilde over (S)}). K_α,f includes indices of triangles in the face region and K_α,f includes indices of triangles in the non-face region. In some implementations, symmetric constraints are not applied for the expressions that do not preserve the vertex symmetry, such as raising left eyebrow, blinking right eye. Symmetric constraints can be applied for expressions that preserve the vertex symmetry, such as the jaw opening.

[0190] Referring now to FIG. 14, FIG. 14 shows a flowchart of an example process 1400 for generating a virtual character for a participant of a video conference, according to certain aspects of the present disclosure. The client device 450 can be configured to implement operations depicted in FIG. 14 by executing suitable program code (e.g., the meeting client application 404). The software or program code may be stored on a non-transitory storage medium (e.g., on a memory device). The process depicted in FIG. 14 and described below is intended to be illustrative and non-limiting. Although FIG. 14 depicts the various processing blocks occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the blocks may be performed in some different order, or some blocks may also be performed in parallel. For illustrative purposes, the process 1400 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0191] At block 1405, the process 1400 involves joining a video conference for a participant of the video conference. For example, the client device 450 associated with the participant can launch the meeting client application 404 to provide the functionality of the video conference as discussed above, including for example presenting a user interface to present a video of the participant. The participant may choose to use a virtual character to replace his/her face in the video. To do so, the meeting client application 404 may present a user interface allowing the participant to choose a virtual character face model from a set of available virtual character face models. The meeting client application 404 receives a user selection from the user interface and determines the selected virtual character face model as the virtual character face model 532. The meeting client application 404 may further identify the virtual character neutral face 554 and the virtual character expressions 556 for this participant that correspond to the selected virtual character face model. As discussed above, each of virtual character expressions 556 is customized for the participant and describes a facial expression of the virtual character customized for the participant. The virtual character expressions for the participant can be generated by combining a set of virtual character expression bases based on a facial feature vector of the participant extracted from an image of the participant. The virtual character neutral face 554 describes a neutral face of the virtual character customized for the participant. The virtual character neutral face model can be generated by combining a set of virtual character face bases according to the facial feature vector of the participant. The virtual character face bases are generated by applying a deformation transfer to the virtual character face model, a human base face, and a set of human face bases. The virtual character neutral face 554 and the virtual character expressions 556 can be generated offline before the video conference starts.

[0192] At block 1410, the process 1400 involves determining an expression parameter vector (such as the expression parameter vector β 558) from the video of the participant. As discussed above with respect to FIG. 5, the expression parameter vector can be detected by determining the facial expression of the participant from the video of the participant. At block 1415, the process 1400 involves generating a customized virtual character for the participant. As discussed in detail above, the client device 450 can apply the expression parameter vector to the set of virtual character expressions 556 customized for the participant and further incorporate the virtual character neutral face 554 customized for the participant to generate the customized virtual character for the participant. The customized virtual character for the participant will have similar facial features as the participant and reflect the facial expressions of the participant in real time or near real time. At block 1420, the process 1400 involves rendering the customized virtual character for the participant to get a video sequence which is sent to other participants of the video conference in replace of the video of the participant captured by the camera of the client device 450.

[0193] At block 1425, the process 1400 involves determining whether the client device 450 should exit the video conference, such as the participant decides to leave the video conference, or the host of the video conference has ended the meeting. If not, the process 1400 continues to generate the customized virtual character for the participant by performing operations in blocks 1410-1420 as discussed above. If it is determined that the client device 450 should exit the meeting, the process 1400 involves disconnecting, at block 1430, from the video conference.

[0194] Referring now to FIG. 15, FIG. 15 shows a flowchart of an example process 1500 for generating a virtual character using virtual triangles, according to certain aspects of the present disclosure. The virtual character base model generation system 412 or another computing device can be configured to implement operations depicted in FIG. 15 by executing suitable program code. The software or program code may be stored on a non-transitory storage medium (e.g., on a memory device). The process depicted in FIG. 15 and described below is intended to be illustrative and non-limiting. Although FIG. 15 depicts the various processing blocks occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the blocks may be performed in some different order, or some blocks may also be performed in parallel. For illustrative purposes, the process 1500 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0195] At block 1505, the process 1500 involves accessing a source human face model, a target human face model, and a source virtual character model. The source virtual character model can include a face region and a non-face region. The non-face region can include, for example, an interior mouth region, an interior eye region, or an accessory of the virtual character. At block 1510, which includes blocks 1515-1525, the process 1500 involves deforming the source virtual character model based on the source human face model and the target human face model to generate a target virtual character model.

[0196] At block 1515, the process 1500 involves generating virtual triangles connecting the face region and the non-face region of the source virtual character model. For example, the virtual character base model generation system 412 can determine a threshold distance value for a surface in the face region and a surface in the non-face region and determine a vertex-triangle distance between each vertex of the surface in the face region and each triangle of the surface in the non-face region. As discussed above and will be discussed below in more detail, the threshold distance value can be determined based on a minimum distance and a maximum distance between the surface in the face region and the surface in the non-face region. The virtual character base model generation system 412 can generate virtual triangles connecting the surface in the face region and the surface in the non-face region, if at least one of the vertex-triangle distances is below the threshold distance value. The generated virtual triangles can include virtual triangles formed by a vertex of the surface in the face region and an edge of a triangle of the surface in the non-face region if the vertex-triangle distance between the vertex and the triangle is below the threshold distance value. In some examples, prior to generating the virtual triangles connecting the face region and the non-face region, the virtual character base model generation system 412 may create a gap on the source virtual character model between the face region and the non-face region. This can help to prevent degenerated virtual triangles to be generated.

[0197] At block 1520, the process 1500 involves deforming the source virtual character model with minimized deformation to the one or more virtual triangles to generate the target virtual character model. For example, the virtual character base model generation system 412 can first deform the face region by fixing the non-face region and removing the virtual triangles from the source virtual character model. Based on the deformed source virtual character model, the virtual character base model generation system 412 can add the virtual triangles back to the mode and deform the non-face region by fixing the deformed face region. The deformation can be performed by imposing a constraint to minimize the deformation to the virtual triangles. In this way, the non-face region can be deformed by propagating the deformation gradients from the face region to the non-face region through the virtual triangles.

[0198] After the deformation, the virtual triangles can be removed from the deformed model at block 1525. As discussed above and will be discussed in more detail below, additional constraints or processing may be applied before, during, or after the deformation to generate the target virtual character. At block 1530, the virtual character base model generation system 412 can render the generated target virtual character into images, videos, and so on. In some examples, the target virtual character may be used to generate the virtual character bases and the virtual expression bases which are further used to generate customized virtual character for a participant of a video conference.

[0199] Referring now to FIG. 16, FIG. 16 shows a flowchart of an example process 1600 for generating a virtual character with facial feature preservation, according to certain aspects of the present disclosure. The virtual character base model generation system 412 or another computing device can be configured to implement operations depicted in FIG. 16 by executing suitable program code. The software or program code may be stored on a non-transitory storage medium (e.g., on a memory device). The process depicted in FIG. 16 and described below is intended to be illustrative and non-limiting. Although FIG. 16 depicts the various processing blocks occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the blocks may be performed in some different order, or some blocks may also be performed in parallel. For illustrative purposes, the process 1600 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0200] At block 1605, the process 1600 involves accessing a source human face model, a target human face model, and a source virtual character face model as well as feature curves. In some examples, the feature curves are marked on the source face model and the source virtual character face model. One feature curve on the source face model has a corresponding feature curve on the source virtual character face model. Each feature curve is used to define one or more features of the source virtual character face model and can be used to preserve the one or more features during the deformation. For example, the features can include a local curve feature defined by a feature curve. The local curve feature can be a mouth curve feature or an eye curve feature. In another example, the features can include a closeness feature defined by two feature curves. For example, if the closeness feature is for the mouth, the two feature curves can include a feature curve defined over the upper lip of the mouth and another feature curve defined over the lower lip of the mouth. Likewise, if the closeness feature is for an eye, the two feature curves can include a feature curve defined over the upper lid of the eye and another feature curve defined over the lower lid of the eye. In some examples, the feature curves are defined based on landmark points on the source human face model and the corresponding landmark points on the source virtual character face model.

[0201] At block 1610, which includes blocks 1615 and 1620, the process 1600 involves deforming the source virtual character model based on the source human face model and the target human face model to generate a target virtual character model. At block 1615, the process 1600 involves determining a loss function based on the feature curves to preserve features of the source virtual character face model on the target virtual character face model. For example, if the features include the local curve feature defined by a feature curve, the virtual character base model generation system 412 can generate a set of virtual feature triangles based on vertices of the feature curve, such as by connecting three adjacent vertices to form a virtual feature triangle A term defined based on the set of virtual feature triangles can be included in the loss function. For instance, the term can include differences between deformation gradients of the set of virtual feature triangles and an identity deformation gradient so that the deformation gradients of the set of virtual feature triangles can be enforced to be closed to identity deformation (i.e., no deformation) as much as possible thereby preserving the local curve feature. If the feature to be preserved is a closeness feature, a term defined based on a mapping distance between the two feature curves defining the closeness can be added to the loss function as described above and will be discussed in more detail later. In a further example, the feature may be a symmetric feature and the loss function can include a term defined based on pairs of vertices in the source virtual character face model.

[0202] At block 1620, the process 1600 involves deforming the source virtual character model by minimizing the loss function. As discussed above and will be discussed in more detail below, additional constraints or processing may be applied before, during, or after the deformation to generate the target virtual character. At block 1630, the virtual character base model generation system 412 can render the generated target virtual character into images, videos, and so on. In some examples, the target virtual character may be used to generate the virtual character bases and the virtual expression bases which are further used to generate customized virtual character for a participant of a video conference.

[0203] Referring now to FIG. 17, FIG. 17 shows a flowchart of an example process 1700 for generating a virtual character with feature enhancement, according to certain aspects of the present disclosure. The virtual character base model generation system 412 or another computing device can be configured to implement operations depicted in FIG. 17 by executing suitable program code. The software or program code may be stored on a non-transitory storage medium (e.g., on a memory device). The process depicted in FIG. 17 and described below is intended to be illustrative and non-limiting. Although FIG. 17 depicts the various processing blocks occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the blocks may be performed in some different order, or some blocks may also be performed in parallel. For illustrative purposes, the process 1700 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0204] At block 1705, the process 1700 involve accessing a source human face model, a target human face model, and a source virtual character face model. At block 1710, the process 1700 involves accessing a virtual feature triangle marked on the source human face model and a corresponding virtual feature triangle marked on the source virtual character face model. The virtual feature triangle on the source human face model can be generated by connecting a set of three landmark points on the source human face model. Likewise, the virtual feature triangle on the source virtual character face model can be generated by connecting a set of three landmark points on the source virtual character face model that correspond to the set of three landmark points on the source human face model.

[0205] At block 1715, which includes blocks 1720 and 1725, the process 1700 involves deforming the source virtual character model based on the source human face model and the target human face model to generate a target virtual character face model. At block 1720, the process 1700 involves determining a loss function based on the virtual feature triangles to enhance certain features of the target virtual character face model. As discussed above and will be discussed in more detail later, if the feature to be enhanced is the size of the eye and the enhancement is to increase the size of the eye on the target virtual character face model, a term defined based on a set of virtual feature triangles on the source virtual character face model and a corresponding set of virtual feature triangles on the target virtual character face model can be included in the loss function. The term can include, for each pair of corresponding virtual feature triangles, a difference, or a function thereof, between the deformation gradient of the virtual feature triangle on the target virtual character face model and a scaled-up deformation gradient of the corresponding virtual feature triangle on the source virtual character face model. In this way, the deformation gradients of the set of virtual feature triangles on the target model can be enforced to be close to a scaled-up deformation of the source model as much as possible thereby enhancing the size of the eye. Similar terms can be introduced to the loss function for other features to be enhanced, such as the other eye, the nose, the mouth, etc.

[0206] At block 1725, the process 1700 involves deforming the source virtual character face model by minimizing the loss function. As discussed above and will be discussed in more detail below, additional constraints or processing may be applied before, during, or after the deformation to generate the target virtual character. At block 1730, the virtual character base model generation system 412 can render the generated target virtual character into images, videos, and so on. In some examples, the target virtual character may be used to generate the virtual character bases and the virtual expression bases which are further used to generate customized virtual character for a participant of a video conference.

[0207] Referring now to FIG. 18, FIG. 18 shows a flowchart of an example process 1800 for generating a virtual character via a two-stage process, according to certain aspects of the present disclosure. The virtual character base model generation system 412 or another computing device can be configured to implement operations depicted in FIG. 18 by executing suitable program code. The software or program code may be stored on a non-transitory storage medium (e.g., on a memory device). The process depicted in FIG. 18 and described below is intended to be illustrative and non-limiting. Although FIG. 18 depicts the various processing blocks occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the blocks may be performed in some different order, or some blocks may also be performed in parallel. For illustrative purposes, the process 1800 is described with reference to certain examples depicted in the figures. Other implementations, however, are possible.

[0208] At block 1805, the process 1800 involve accessing a source human face model, a target human face model, and a source virtual character face model. At block 1810, which includes blocks 1815 and 1820, the process 1800 involves deforming the source virtual character model based on the source human face model and the target human face model to generate a target virtual character face model.

[0209] At block 1815, the process 1800 involves deforming the face region of the source virtual character face model by fixing the non-face region. As discussed above, a virtual character face model can include a face region, which has corresponding surfaces on the human face model, and a non-face region which does not have corresponding surfaces on the human face model. As such, the deformation at block 1815 can be performed for the face region of the virtual character face model according to the deformation from the source human face model to the target human face model. To do so, constraints or terms in the loss function that involve the non-face region can be disabled or removed from the loss function before minimizing the loss function to perform the deformation, such as Eqs. (17), (18), (20), (21), (26), and (32) as discussed above or will be discussed later in more detail.

[0210] At block 1820, the process 1800 involves deforming the non-face region of the source virtual character face model by fixing the face region. For example, constraints or terms in the loss function that do not involve the non-face region can be disabled or removed from the loss function before minimizing the loss function to perform the deformation, such as Eqs. (24), (25), (30), and (31) as discussed above or will be discussed later in more detail. At block 1830, the virtual character base model generation system 412 can render the generated target virtual character into images, videos, and so on. In some examples, the target virtual character may be used to generate the virtual character bases and the virtual expression bases which are further used to generate customized virtual character for a participant of a video conference.

[0211] While in the above examples, the deformation transfer is applied to the virtual character face model A₀, the same process can be applied to other types of virtual characters. For example, the system may allow a user to choose accessories, such as hair, beard, eye glasses, hat, and so on. The accessories can be attached to the virtual character face model to form a composite virtual character model H. This composite model can be deformed in a similar way as the virtual character face model A₀ (e.g., by treating the accessories as components of the virtual character face model and by building the virtual triangles) so that the accessories fit the deformed virtual character face.

An Example of Virtual Character Face Model Generation

[0212] In the following, a detailed example of generating a virtual character face model or a customized virtual character face model is described.

1 Introduction

[0213] This example describes details of a system for generating a virtual character face model and a customized virtual character face model. The virtual character is represented as a 3DMM model, which includes a base surface, face bases (PCAs), and expressions. The system uses a deformation transfer algorithm to transfer face bases (PCAs) and expressions from an existing human 3DMM model to the virtual character.

2 Preliminaries

2.1 Notations

Denote

• • [0214] |·|: the number of elements of a set or a vector.
  • [0215] I₃: the 3×3 identity matrix

[0216] $\begin{array}{cc}{I}_3=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right).& \left(1\right)\end{array}$

• • [0217] E_x: the 3×3 symmetric matrix

[0218] $\begin{array}{cc}{E}_x=\left(\begin{array}{ccc}-1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right).& \left(2\right)\end{array}$
Given a 3D triangulated model H, its elements are denoted as

• • [0219] A vertex: a vector with 3 values as its coordinates
  • [0220] A triangle: a vector with 3 vertices' indices
  • [0221] A curve: a vector of vertices' indices, where i-th and (i+1)-th vertices is an edge of a triangle
  • [0222] V(H): all vertices
    • [0223] V_i(H)∈V(H): the i-th vertex
    • [0224] |V(H)|: number of vertices
  • [0225] F(H): all triangles
    • [0226] F_i(H)∈F(H): the i-th triangle
    • [0227] IF(H)|: number of triangles
  • [0228] Θ_v,i(H): all adjacent triangles of the i-th vertex V_i(H)
  • [0229] Θ_f,i(H): all adjacent triangles of the i-th triangle F_i(H)
  • [0230] L(H): manually labelled data (detailed in section 4.2.1)
    • [0231] Lv(H)∈L (H): feature vertices
      • [0232] L_v,i(H)∈L_v(H): the i-th feature vertex
    • [0233] Lc(H)∈L(H): feature curves
      • [0234] L_c,i(H)∈L_c(H): the i-th feature curve
    • [0235] L_v(H)∈L(H): feature regions
      • [0236] L_r,i(H)∈L_r(H): the i-th feature region
  • [0237] W(H): automatically extracted components, where a component is a 3D mesh consisting of one simply connected region (detailed in section 4.1.4).
    • [0238] W_i(H)∈L(H): the i-th component
    • [0239] W₁(H)∈L (H): the main component, which is the component including the face region. It may not contain facial features, such as the nose which may be a separated component. It may contain interior mouth and eye regions.
  • [0240] Δ(H): the face region, which is a region including facial features such as noses and lips and excluding interior regions such as interior mouth and eye regions (detailed in section 4.2.2).
  • [0241] G(H): virtual triangles
  • [0242] K(H): sets of vertices or triangles
  • [0243] M(H)={(i,j)}: symmetric pairs of vertices. For (i,j)∈Ms(H), E_xV_i(H)=v_j(H). For (i, i)∈M_s(H), the x-coordinate of V_i(H) is zero (detailed in section 4.1.10)
  • [0244] Mg (H): manual defined virtual triangles based on feature vertices L_v(H),
    • [0245] (i_l, j_l, k_l, η_l,x, η_l,y, η_l,z)∈Mg (H): the 1-th virtual triangle where the deformation gradient for the triangle (L_v,il, L_v,jl, L_v,kl) will be scaled by η_l,x, η_l,y, η_l,z along x-,y-,z-axis (detailed in section 4.1.6).

[0246] Denote the following elements between two models as

• • [0247] T_j(H, {tilde over (H)}): the deformation gradient for the i-th triangle v_j(H) from the model H to its deformed model H (detailed in section 1).
  • [0248] M_c(H₁, H₂): the triangle correspondence between the model H₁ and the model H₂ (detailed in section 2.5.2).
    • [0249] |M_c(H₁, H₂)|: number of triangle correspondences in M_c(H₁, H₂).
  • [0250] ρ_f(H, {tilde over (H)}): the triangles that are fixed when deforming the model H to its deformed model {tilde over (H)}, i.e., v_j({tilde over (H)})=v_j(H) for i∈ρ_f(H, {tilde over (H)}), j∈F₁(H):

[0251] Given the source model S, its deformation {tilde over (S)}, and the target model A, a deformation transfer (A, S, {tilde over (S)}) generates a deformed model à for the model A as (detailed in section 2.5.3 and section 4.2.3)

[0252] $\begin{array}{cc}\tilde{A}=ℱ\left(A,S,\tilde{S}\right)& \left(3\right)\end{array}$
Given the source model S, its deformation {tilde over (S)}, the target model A, and the triangle correspondence M_c(S, A), the fixed vertices on A is defined as (implemented in section 4.1.8)

[0253] $\begin{array}{cc}{\rho }_v\left(A,S,\tilde{S}\right)& \left(4\right)\end{array}$
Given the source model S, its deformation {tilde over (S)}, the symmetric switch for {tilde over (S)} indicates whether {tilde over (S)} remains symmetric vertex pairs from S as (implemented in section 4.1.10)

[0254] $\begin{array}{cc}{\rho }_s\left(S,\tilde{S}\right)& \left(5\right)\end{array}$
Given a deformation gradient T_i, the enhancement of its scaling along i-th axis by s_i is deformed as (implemented in section 4.1.6)

[0255] $\begin{array}{cc}{E}_{\left(s_1,s_2,s_3\right)}\left(T_i\right)& \left(6\right)\end{array}$
Given 3D vertices, p₀, p₁, p₂, p₃, the following functions can be defined

[0256] $\begin{array}{cc}{N}_1\left(p_0,p_1,p_2\right)=p_0+\frac{\left(p_2-p_0\right)\times \left(p_1-p_0\right)}{\sqrt{\left(p_2-p_0\right)\times \left(p_1-p_0\right)}}& \left(7\right)\end{array}$ $\begin{array}{cc}{N}_2\left(p_0,p_1,p_2,p_3\right)={\left({\left[p_1-p_0,p_2-p_0,p_3-p_0\right]}^{-1}\right)}^T\left(\begin{array}{cccc}-1& 1& 0& 0\\ -1& 0& 1& 0\\ -1& 0& 0& 1\end{array}\right)& \left(8\right)\end{array}$
Given two curves L₁ and L₂, curve mapping and curve overlapping are denoted as (detailed in section 4.1.7)

[0257] $\begin{array}{cc}N\left(L_1,L_2\right),\delta \left(L_1,L_2\right)& \left(9\right)\end{array}$
Given virtual character data A₀, {H^i,j}, virtual character data pre-processing is denoted as (detailed in section 4.2.2)

[0258] $\begin{array}{cc}H={\Omega }_1\left(A_0,\left\{H^{i,j}\right\}\right)& \left(10\right)\end{array}$
A rigid transformation is used to transfer H to the same space as S₀

[0259] $\begin{array}{cc}\Gamma =\Gamma \left(H,S_0\right)& \left(11\right)\end{array}$
Given a deformed model H of the pre-processed model H, virtual character data post-processing is denoted as (detailed in section 4.2.2)

[0260] $\begin{array}{cc}{\tilde{A}}_0,\left\{{\tilde{H}}^{i,j}\right\}={\Omega }_2\left(\tilde{H}\right)& \left(12\right)\end{array}$
2.2 Barycentric Coordinates

[0261] Given a triangle Γ with three vertices (v₀, v₁, v₂), any vertex q on this triangle can be represented as

[0262] $\begin{array}{cc}q=\sum _{i=0}^2{r}_i{v}_i& \left(13\right)\end{array}$
where r₀, r₁, r₂ is the barycentric coordinates of q on F, with r₀+r₁+r₂=1, r_i≥0.
2.3 Curve Parameterization

[0263] On a mesh H with vertices V=V(H), a curve L consists of a set of indices as

[0264] $\begin{array}{cc}L=\left[\begin{array}{cccccc}{v}_1& v_2& \dots & v_i& \dots & v_n\end{array}\right]& \left(14\right)\end{array}$
where v_i is the index of the vertex on the mesh, n=|L|, and V_viV_vi+1 is an edge of the mesh. The curve can be parameterized as

[0265] $\begin{array}{cc}L\left(t\right)=\left[\begin{array}{cccccc}{t}_1& t_2& \dots & t_i& \dots & t_n\end{array}\right]=\{\begin{array}{cc}{V}_{v_i}& t=t_i\\ \frac{t_{i+1}-t}{t_{i+1}-t_i}{V}_{v_i}+\frac{t-t_i}{t_{i+1}-t_i}{V}_{v_{i+1}}& t\in \left[t_i,t_{i+1}\right]\end{array}& \left(15\right)\end{array}$
where each vertex V_vi is parameterized as L(t_i)=V_vi with

[0266] $\begin{array}{cc}{t}_i=\{\begin{array}{cc}0& i=1\\ \frac{{\sum }_{j=1}^{i-1}V_{v_j}{V}_{v_{j+1}}}{{\sum }_{j=1}^{n-1}V_{v_j}{V}_{v_{j+1}}}& 1<i<n\\ 1& i=n\end{array}& \left(16\right)\end{array}$
2.4 3DMM Models

[0267] A neural network, such as a FaceTrackNet described in in U.S. Patent Application Publication 2023/0260184A1, the entirety of which is incorporated herein by reference, derives the 3DMM parameters (α, β, R, T, s) from an input human facial image. 3DMM parameters include a pose vector (R, T, s), a facial feature vector α, and an expression vector β. Given a facial feature vector α and expression l, the human facial model in 3DMM form is

[0268] $\begin{array}{cc}S=S_0+S_{\mathrm{id}}·\alpha +S_{\exp }·\beta & \left(17\right)\end{array}$
where S₀ is the base surface, S_id is the PCA shape basis, and S_exp is the expression basis. A pose, denoted as (R, T, s)=(R, t_x, t_y, s), is defined as

• • [0269] R: rotation
  • [0270] t_x, t_y: translation, with T=(t_x, t_y, 0)^T
  • [0271] s: scaling

[0272] Given a viewport of height h and width w, the projection matrix corresponding to the pose (R, T, s) is

[0273] $\begin{array}{cc}\Phi =\left[\begin{array}{cccc}\frac{w}{2}& 0& 0& \frac{w}{2}\\ 0& -\frac{h}{2}& 0& \frac{h}{2}\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}\frac{2s}{w}& 0& 0& -1\\ 0& \frac{2s}{h}& 0& -1\\ 0& 0& -1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cc}R& T\\ 0& 1\end{array}\right]=\left[\begin{array}{cccc}s& 0& 0& 0\\ 0& -s& 0& h\\ 0& 0& -1& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cc}R& T\\ 0& 1\end{array}\right]& \left(18\right)\end{array}$
The projection matrix projects a 3D point, P, into the 3D viewport as

[0274] $\begin{array}{cc}\left[\begin{array}{c}s\left(R_{0}P+t_x\right)\\ h-s\left(R_{1}P+t_y\right)\\ -R_{2}P\end{array}\right]& \left(19\right)\end{array}$
Given a pose (R, T, s), the SOP (scaled orthographic projection), Π, projects a 3D point, P, into a 2D point linearly as

[0275] $\begin{array}{cc}\begin{array}{c}\prod \left(P\right)={\prod }_{R,T,s}\left(P\right)=\left[\begin{array}{cccc}s& 0& 0& 0\\ 0& -s& 0& h\end{array}\right]\left[\begin{array}{cc}R& T\\ 0& 1\end{array}\right]\left[\begin{array}{c}P\\ 1\end{array}\right]\\ =\left[\begin{array}{cc}{\mathrm{sR}}_0& {\mathrm{st}}_x\\ -{\mathrm{sR}}_1& h-{\mathrm{st}}_y\end{array}\right]\left[\begin{array}{c}P\\ 1\end{array}\right]\\ =\left[\begin{array}{c}s\left(R_{0}P+t_x\right)\\ h-s\left(R_{1}P+t_y\right)\end{array}\right]\end{array}& \left(20\right)\end{array}$
2.5 Deformation Transfer Algorithm
2.5.1. Deformation Gradient

[0276] Given a 3D triangulated model H and its deformed model H, the i-th vertex of each model is

[0277] $v_i=V_i\left(H\right),{\tilde{v}}_i=V_i\left(\tilde{H}\right).$
For the i-th triangle of each model,

[0278] $F_i\left(H\right)=F_i\left(\tilde{H}\right)=\left(i_0,i_1,i_2\right),$
add the fourth vertex as

[0279] $v_{i_3}=N_1\left(v_{i_0},v_{i_1},v_{i_2}\right),{\tilde{v}}_{i_3}=N_1\left({\tilde{v}}_{i_0},{\tilde{v}}_{i_1},{\tilde{v}}_{i_2}\right)$
where N₁ is defined in Eq.(7). The deformation gradient for the i-th triangle F_i from H to {tilde over (H)} is

[0280] $\begin{array}{cc}{T}_i\left(H,\tilde{H}\right)=T_{\left(i_0,i_1,i_2\right)}\left(H,\tilde{H}\right)=\left[{\tilde{v}}_{i_1}-{\tilde{v}}_{i_0},{\tilde{v}}_{i_2}-{\tilde{v}}_{i_0},{\tilde{v}}_{i_3}-{\tilde{v}}_{i_0}\right]·{\left[v_{i_1}-v_{i_0},v_{i_2}-v_{i_0},v_{i_3}-v_{i_0}\right]}^{-1},& \left(21\right)\end{array}$
which can be rewritten in matrix form as

[0281] ${\left(T_i\left(H,\tilde{H}\right)\right)}^T={\left({\left[v_{i_1}-v_{i_0},v_{i_2}-v_{i_0},v_{i_3}-v_{i_0}\right]}^{-1}\right)}^T·{\left[{\tilde{v}}_{i_1}-{\tilde{v}}_{i_0},{\tilde{v}}_{i_2}-{\tilde{v}}_{i_0},{\tilde{v}}_{i_3}-{\tilde{v}}_{i_0}\right]}^T={\left({\left[v_{i_1}-v_{i_0},v_{i_2}-v_{i_0},v_{i_3}-v_{i_0}\right]}^{-1}\right)}^T·\left(\begin{array}{c}{\tilde{v}}_{i_1}^T-{\tilde{v}}_{i_0}^T\\ {\tilde{v}}_{i_2}^T-{\tilde{v}}_{i_0}^T\\ {\tilde{v}}_{i_3}^T-{\tilde{v}}_{i_0}^T\end{array}\right)={\left({\left[v_{i_1}-v_{i_0},v_{i_2}-v_{i_0},v_{i_3}-v_{i_0}\right]}^{-1}\right)}^T\left(\begin{array}{cccc}-1& 1& 0& 0\\ -1& 0& 1& 0\\ -1& 0& 0& 1\end{array}\right)\left(\begin{array}{c}{\tilde{v}}_{i_0}^T\\ {\tilde{v}}_{i_1}^T\\ {\tilde{v}}_{i_2}^T\\ {\tilde{v}}_{i_3}^T\end{array}\right)=N_2\left(v_{i_0},v_{i_1},v_{i_2},v_{i_3}\right)\left(\begin{array}{c}{\tilde{v}}_{i_0}^T\\ {\tilde{v}}_{i_1}^T\\ {\tilde{v}}_{i_2}^T\\ {\tilde{v}}_{i_3}^T\end{array}\right)$
where 2 is defined in q.
2.5.2. Triangle Correspondence Using Non-Rigid Registration

[0282] Given source model S and target model A, non-rigid registration is to deform S into {tilde over (S)}, such that all vertices V({tilde over (S)}) are close to the model A. The non-rigid registration is achieved via solving the following optimization problem

[0283] $\begin{array}{cc}\underset{V\left(\tilde{S}\right)}{\min }E\left(\tilde{S}\right)=E\left(V\left(\tilde{S}\right)\right)={\lambda }_0{E}_0+{\lambda }_1{E}_1+{\lambda }_2{E}_2& \left(22\right)\end{array}$ $s.t.V_k\left(\tilde{S}\right)=m_k,k\in K.$
where

• • [0284] K is a set of markers on the target model A
  • [0285] λ₀, λ₁, λ₂ are hyper parameters
  • [0286] E₀ is the smoothness term defined as

[0287] $\begin{array}{cc}{E}_0\left(\tilde{S}\right)=\sum _{i=1}^{❘F\left(S\right)❘}\sum _{j\in {\Theta }_{f,i}\left(S\right)}{T_i\left(S,\tilde{S}\right)-T_j\left(S,\tilde{S}\right)}^2& \left(23\right)\end{array}$

• • [0288] where Θ_f,i(S) is the set of adjacent triangles to the i-th triangle.
  • [0289] E₁ is the registration identity term defined as

[0290] $\begin{array}{cc}{E}_1\left(\tilde{S}\right)=\sum _{i=1}^{❘F\left(S\right)❘}{T_i\left(S,\tilde{S}\right)-I_3}^2.& \left(24\right)\end{array}$

• • [0291] E₂ is the closest valid point term defined as

[0292] $\begin{array}{cc}{E}_2\left(\tilde{S}\right)=\sum _{i=1}^{❘V\left(S\right)❘}{V_i\left(\tilde{S}\right)-c_i}^2& \left(25\right)\end{array}$

• • [0293] where c_i is the closest valid point for V_i({tilde over (S)}) on A.
    After non-rigid registration, the triangle correspondence between S and A can be set up by comparing the centroids of F({tilde over (S)}) and F(A). A triangle pair (i,j) is selected if the centroid of the i-th triangle F_i({tilde over (S)}) and the centroid of the j-th triangle F_j(A) are close enough (such as closer than a pre-determined threshold) and the angle between their normals is less than 90°. The triangle correspondence between S and A can be denoted as