Drivable implicit three-dimensional human body representation method

[0001] The present application is a continuation of International Application No. PCT/CN2021/088226, filed on Apr. 19, 2021, the content of which is incorporated herein by reference in their entirety.

[0002] The present application belongs to the field of three-dimensional reconstruction, in particular to a drivable implicit three-dimensional human body representation method.

[0003] The present application proposes a drivable implicit function to represent a dynamic human body, and a drivable three-dimensional model can be reconstructed from a multi-view video for generation of a three-dimensional model and free-view synthesis. In the related art, it often takes a high cost and a lot of time to reconstruct a drivable three-dimensional model with traditional methods. These methods need complex hardware devices to reconstruct the model, and in the later stage, the designers need to bind the skeleton to the three-dimensional model for control, and design skinning blend weights, which takes a lot of time. Recently, in some works, the dynamic human body is expressed as a hidden function based on a neural network. However, it is difficult for these methods to obtain an optimal implicit function only from the video, and additional constraints are required. Moreover, it is not possible to obtain a drivable three-dimensional model with these methods. The present application aims to propose a new dynamic human body representation method, which enables leaning of an implicit function representation from the video to be solvable, and can output a drivable three-dimensional model.

[0004] The present application aims to provide a drivable implicit three-dimensional human body representation method aiming at the deficiency of the prior art. The drivable implicit function is used for representing the geometry and appearance of a human body, and the dynamic human body representation is reconstructed through differential rendering.

[0005] The object of the present application is achieved through the following technical solution: a drivable implicit three-dimensional human body representation method includes the following steps of:

[0006] (1) Constructing a neural network implicit function in a standard coordinate system to represent a geometry and an appearance of a human body, generating a skinning blend weight of any three-dimensional point in each video frame coordinate system by using the neural network implicit function, constructing a neural skinning blend weight field, and transforming the three-dimensional points of a video frame back to the standard coordinate system to represent a dynamic human body.

[0007] (2) Learning the neural skinning blend weight field in the standard coordinate system and optimizing the neural network implicit function, so that the neural network implicit function in the standard coordinate system can be driven to generate a three-dimensional human body in a new state; the specific learning process is as follows:

[0008] (2.1) Rendering the neural network implicit function into a two-dimensional image based on a differentiable volume renderer; optimizing a representation of the neural network implicit function by minimizing an error between the rendered two-dimensional image and a corresponding image in a multi-view video.

[0009] (2.2) Minimizing a difference between the skinning blend weights of corresponding three-dimensional points in the standard coordinate system and the video frame coordinate system, and optimizing the representation of the neural network implicit function of the skinning blend weight.

[0010] (3) Performing generation of a three-dimensional model of the human body and view synthesis based on the optimized neural network implicit function in step (2).

[0011] Further, the constructed neural skinning blend weight field is combined with a set of three-dimensional key points of the human body, and the three-dimensional points in the video frame coordinate system are transformed back to the standard coordinate system by a linear blend skinning algorithm.

[0012] Further, after learning the neural skinning blend weight field in the standard coordinate system, when a new set of three-dimensional key points of the human body are given, the neural skinning blend weight field in the standard coordinate system can be combined with the key points, and the three-dimensional points in the standard coordinate system are transformed by a linear blend skinning algorithm, thereby generating the three-dimensional human body in the new state.

[0013] Further, the step of rendering the neural network implicit function into a two-dimensional image based on a differentiable volume renderer comprises: sampling a set of three-dimensional points along light projected to a pixel by a camera, transforming the three-dimensional points back to the standard coordinate system by a linear blend skinning algorithm, calculating voxel densities and colors of the three-dimensional points by using the neural network implicit function, and accumulating volume densities and colors on the light to obtain a pixel color.

[0014] Further, the corresponding three-dimensional points in the standard coordinate system and the video frame coordinate system are sampled, and the corresponding skinning blend weights are calculated to minimize the difference of the skinning blend weights of two three-dimensional points, thereby optimizing the neural network implicit function of the skinning blend weight.

[0015] Further, during the generation of the three-dimensional model, a Marching cubes algorithm is used to extract a three-dimensional mesh model from the optimized neural network implicit function, and the three-dimensional mesh model is driven according to a linear blend skinning algorithm.

[0016] Further, during the view synthesis, the differentiable volume renderer is used to obtain the two-dimensional image.

[0017] The method has the beneficial effects that a neural skinning blend weight field represented by a neural network is proposed, so that a drivable hidden function is obtained by learning from the video, and the method is the first method for obtaining the drivable three-dimensional model from the video by optimization. According to the present application, a neural skinning blend weight field is learned in each video frame, and the three-dimensional points of each video frame are transformed back to a standard coordinate system, so that the time sequence information of the video is integrated, the observation of the target is improved, and the three-dimensional model in the standard coordinate system is optimized; at the same time, according to the present application, a neural skinning blend weight field is learned in the standard coordinate system, so that the three-dimensional model can be driven to generate a three-dimensional model in a new posture. The present application can obtain high-quality three-dimensional reconstruction and view synthesis effects from multi-view videos.

[0020] The technical details and principles of the present application will be further described with reference to the accompanying drawings:

[0021] The present application provides a drivable implicit three-dimensional human body representation method. As shown in FIG. 1, the present application optimizes and reconstructs a drivable implicit function from an input multi-view video. The reconstructed implicit function can be used for generation of a three-dimensional model of a human body and view synthesis. The specific process is as follows.

[0022] (1) A neural network implicit function is constructed in a standard coordinate system to represent a geometry and an appearance of a human body, a skinning blend weight of any three-dimensional point is generated in each video frame coordinate system by using the neural network implicit function, a neural skinning blend weight field is constructed, the constructed neural skinning blend weight field is combined with a set of three-dimensional key points of the human body, and the three-dimensional points in the video frame coordinate system are transformed back to the standard coordinate system by a linear blend skinning algorithm for representing a dynamic human body.

[0023] (2) The neural skinning blend weight field is learned in the standard coordinate system and the neural network implicit function is optimized, so that the neural network implicit function in the standard coordinate system can be driven to generate a three-dimensional human body in a new state; the specific learning process is as follows: when a new set of three-dimensional key points of the human body are given, the neural skinning blend weight field in the standard coordinate system can be combined with the key points, and the three-dimensional points in the standard coordinate system are transformed by a linear blend skinning algorithm, thereby generating the three-dimensional human body in a new state.

[0024] The specific learning process is as follows:

[0025] (2.1) The neural network implicit function into a two-dimensional image is rendered based on a differentiable volume renderer, including the following steps: sampling a set of three-dimensional points along light projected to a pixel by a camera, transforming the three-dimensional points back to the standard coordinate system by a linear blend skinning algorithm, calculating voxel densities and colors of the three-dimensional points by using the neural network implicit function, and accumulating volume densities and colors on the light to obtain a pixel color.

[0026] The representation of the neural network implicit function is optimized by minimizing an error between the rendered two-dimensional image and a corresponding image in a multi-view video.

[0027] (2.2) The corresponding three-dimensional points in the standard coordinate system and the video frame coordinate system are sampled, and the corresponding skinning blend weights are calculated to minimize the difference of the skinning blend weights of two three-dimensional points, thereby optimizing the neural network implicit function of the skinning blend weight.

[0028] (3) Generation of a three-dimensional model of the human body and view synthesis are performed based on the optimized neural network implicit function in step (2). Specifically, during the generation of the three-dimensional model, a Marching cubes algorithm is used to extract a three-dimensional mesh model from the optimized neural network implicit function, and the three-dimensional mesh model is driven according to a linear blend skinning algorithm. During the view synthesis, the differentiable volume renderer is used to obtain the two-dimensional image.

[0029] As shown in FIG. 2, in the drivable implicit three-dimensional human body representation method proposed by the present application, the specific steps of constructing a drivable implicit function for representing a dynamic human body are as follows:

[0030] 1. The present application represents a dynamic human body based on a model in a standard coordinate system and a skinning blend weight field in a video frame coordinate system. The model in the standard coordinate system is represented by continuous voxel densities and colors, in which the voxel density field and color field are realized by a multi-layer perceptron network. For a specific frame i in a multi-view video, the deformation field from the video frame coordinate system to the standard coordinate system is T_i, and the specific construction method will be introduced below. The present application expresses the voxel density prediction of a spatial three-dimensional point x in the video frame coordinate system as the following function:
σ_i(x),z_i(x)=F_σ(γ_x(T_i(x)))

• • [0031] where F_σ represents a multi-layer perceptron network with four fully connected layers, γ_x represents a coding function acting on the position of the three-dimensional point x, σ_i(x) and z_i(x) represent the voxel density and shape feature of the three-dimensional point x in the specific frame i, respectively.

[0032] Regarding the color function, the present application takes the shape feature z_i(x) and the viewing angle direction d as the input of the function. In addition, the present application defines a global hidden variable _i as an input for each frame, which is used to encode the state of the human appearance in the frame. The color function is defined as follows:
c_i(x)=F_c(z_i(x),γ_d(d),_i)

• • [0033] where F_c represents a multi-layer perceptron network with two fully connected layers, γ_d represents a position coding function acting on the viewing angle direction d, and c_i(x) represents the color of the three-dimensional point x in the specific frame i.

[0034] As shown in FIG. 2, in the dynamic hidden variable proposed by the present application, the specific steps of constructing a deformation field are as follows:

[0035] 1. The human body is driven by the human skeleton, there are K body parts and k transformation matrices can be generated. In the linear blend skinning algorithm, a three-dimensional point v in the standard coordinate system can be transformed into a three-dimensional point v′ in the coordinate system of a certain video frame by the following formula:

[0036] $v^{\prime }=\left({\sum }_{k=1}^K{w\left(v\right)}_k{G}_k\right)v$

• • [0037] where w(v)_k represents the skin mixing weight of the k^th body part, G_k represents a transformation matrix of the k^th body part. Similarly, for a three-dimensional point x in the coordinate system of a certain video frame, if the skinning blend weight corresponding to the three-dimensional point is known in the present application, the present application can transform the three-dimensional point x into a three-dimensional point x′ in the standard coordinate system by the following formula:

[0038] $x^{\prime }={\left({\sum }_{k=1}^K{w^o\left(x\right)}_k{G}_k\right)}^{-1}x$

• • [0039] where w^o(x)_k represents the skinning blend weight defined in the video frame coordinate system, and G_k represents the transformation matrix of the k^th body part.

[0040] 2. The present application uses a neural network to generate the skinning blend weight of any three-dimensional point in the video frame coordinate system. A direct way is to use a fully connected network to map any three-dimensional points into a skinning blend weight. Another way is to first calculate the initial skinning blend weight of each three-dimensional point according to a statistical driving model, then generate a residual weight using the fully connected network, and finally obtain a final skinning blend weight from the two weights. The skinning blend weight generated in this way is defined as:
w_i(x)=norm(F_Δw(x,ψ_i)+w^s(x,S_i))

• • [0041] where S_i represents a statistical driving model, w^s represents an initial skinning blend weight calculated based on the statistical driving model S_i, F_Δw(x, ψ_i) represents a fully connected network for calculating the residual weight, ψ_i represents the weight hidden variable shown in FIG. 2, and norm is the normalized function. An example of implementation is that the initial skinning blend weight w^s of any three-dimensional point can be generated according to a SMPL model.

[0042] 3. The present application also generates a skinning blend weight field w^can in the standard coordinate system. For any three-dimensional point, the initial skinning blend weight is calculated based on the statistical driving model in a standard state. At the same time, the present application defines a weight hidden variable ψ^can in the standard coordinate system as the input of the fully connected network F_Δw(x, ψ_i) for calculating the residual weight.

[0043] In the drivable implicit three-dimensional human body representation method provided by the present application, the drivable neural network implicit function representation is optimized through differential rendering. The specific steps are as follows:

[0044] 1. Differentiable volume rendering: for a given viewing angle, the representation of a neural network implicit function is converted into a two-dimensional RGB image by using a differentiable volume renderer. For each pixel of the image, the differential volume renderer accumulates the volume densities and colors on the camera ray through an integral equation to obtain the pixel color. In actual implementation, the present application uses numerical integration for approximation. Firstly, the camera parameters are used to calculate the corresponding camera ray r, and then N_k three-dimensional points are sampled between the nearest point and the farthest point. The nearest point and farthest point here can be calculated according to the SMPL model. Then, the present application uses the neural network implicit function to calculate the voxel density and color of the three-dimensional point at each three-dimensional point. For a specific frame i in a multi-view video, the rendered color {tilde over (c)}_i(r) of a pixel is defined as:

[0045] ${\tilde{C}}_i\left(r\right)={\sum }_{k=1}^{N_k}{T}_k\left(1-\exp \left({\sigma }_i\left(x_k\right){\delta }_k\right)\right)c_i\left(x_k\right)$ $T_k=\exp \left(-{\sum }_{j=1}^{k-1}{\sigma }_i\left(x_j\right){\delta }_j\right)$

• • [0046] where δ_k=∥x_k+1−x_k∥₂ represents the distance between adjacent sampling points, σ_i(x_k) and c_i(x_k) represent the colors of the three-dimensional point x_k of a specific frame i, N_k represents the number of three-dimensional points sampled on the camera ray r. By adopting differentiable volume rendering, the present application optimizes the representation of the neural network implicit function based on structured hidden variables by minimizing the error between the rendered image of each frame and the corresponding frame and the corresponding view image in the multi-view video.

[0047] 2. The representation of the implicit function based on a drivable neural network is optimized, specifically: for the input multi-view video, all cameras are calibrated in advance and the camera parameters are known. For each picture, the present application uses a segmentation method to segment the foreground person and sets the background pixel value to 0. The present application jointly optimizes parameters F_σ, F_c, F_Δw, {_i}, {ψ_i} to minimize the following objective function:
L_rgb=∥{tilde over (C)}_i(r)−C_i(r)∥₂

• • [0048] where represents the set of camera rays that pass through the picture pixels and C_i(r) represents a true pixel value.

[0049] In addition, in order to learn the skinning blend weight field w^can in the standard coordinate system, the present application introduces a consistency loss function of the skinning blend weight field, that is, the corresponding points in the standard coordinate system and the video frame coordinate system should have the same skinning blend weight. For the three-dimensional point x in the coordinate system of the i^th frame, the above formula can be used to map to the point T_i(x) in the standard coordinate system, and the consistency loss function is as follows:
L_nsf=Σ_x∈Xi∥w_i(X)−w^can(T_i(x))∥₁

• • [0050] where X_i represents a set of three-dimensional points sampled in the three-dimensional human body bounding box in the coordinate system of the i^th frame. In addition, the coefficients of L_rgb and L_nsf are both set to be 1.

[0051] The dynamic three-dimensional reconstruction and view synthesis provided by the present application specifically includes the following steps:

[0052] 1. View synthesis: in order to generate different human bodies according to given key points and synthesize pictures, the present application constructs a deformation field according to the input key points, and transforms the three-dimensional points under the key points into a standard coordinate system. A new set of key points of human body is given, the present application firstly updates the statistical driving model S^new and calculates the initial skin mixing weight w^s of each three-dimensional point accordingly. Then the final skinning blend weight is calculated according to the following formula:
w^new(x,ψ^new)=norm(F_Δw(x,ψ^new)+w^s(x,S^new))

• • [0053] where ψ^new represents a weight hidden variable in the new state. The parameters of ψ^new are optimized according to the following formula:
    L_new=∥w^new(x)−w^can(T^new(x))∥₁
  • [0054] where T^new represents a deformation field from the coordinate system in the new state to the standard coordinate system, ^new represents a set of three-dimensional points sampled in the three-dimensional box of the human body. It should be noted that the parameters of w^can are fixed in training.

[0055] 2. Generation of a three-dimensional model: based on the optimized neural network implicit function, the present application firstly needs to discretize the continuous space into uniform voxels. Firstly, the spatial range of human body is determined, and the space is divided into a set of voxels with a size of 5 mm×5 mm×5 mm. According to the present application, the voxel density is calculated for each voxel, and finally a human body mesh model is extracted by Marching cubes algorithm. Then the skinning blend weight is calculated on each human body mesh node. Finally, the key points are given, the present application drives the grid nodes to obtain a new three-dimensional model.

[0056] The above-mentioned embodiments are used to explain, rather than to limit the present application. Any modification and change made to the present application within the scope of protection of the spirit and claims of the present application shall fall within the scope of protection of the present application.