Content generation device, content generation method, and program

[0001] The present application is a continuation of and claims the benefit of priority to International Application No. PCT/JP2022/042847, filed Nov. 18, 2022, which is based upon and claims the benefit of priority to Japanese Application No. 2021-188791, filed Nov. 19, 2021. The entire contents of these applications are incorporated herein by reference.

[0002] The present invention relates to a content generation device, a content generation method, and a program.

[0003] For example, JP H11-312160 A describes a technique that uses an independent and moving character as an avatar to promote communication relating to a document between users. The entire contents of this publication are incorporated herein by reference.

[0004] According to one aspect of the present invention, a content generation device includes a control unit including circuitry that acquires text data representing a first text, generate, using a voice generation model, a synthesized voice in which the first text is read out in a voice of the user, and generate synthesized content by synthesizing the synthesized voice and a personal image of the user. The circuitry of the control unit learns a way of reading out a second text in the voice of the user based on the reading out of the second text by the user and generates the voice generation model based on the way of reading out the second text in the voice of the user.

[0005] According to another aspect of the present invention, a content generation method includes acquiring text data representing a first text, generating, using a voice generation model generated based on a voice in which a user has read out a second text by learning a way of reading out the second text in a voice of the user, a synthesized voice in which the first text represented by the text data is read out in the voice of the user, and generating synthesized content by synthesizing the generated synthesized voice and a personal image of the user.

[0006] According to yet another aspect of the present invention, a non-transitory computer-readable medium is stored therein a program that when executed by a computer, a content generation device executes a content generation method. The content generation method includes acquiring text data representing a first text, generating, using a voice generation model generated based on a voice in which a user has read out a second text by learning a way of reading out the second text in a voice of the user, a synthesized voice in which the first text represented by the text data is read out in the voice of the user, and generating synthesized content by synthesizing the generated synthesized voice and a personal image of the user.

[0021] Embodiments will now be described with reference to the accompanying drawings, wherein like reference numerals designate corresponding or identical elements throughout the various drawings.

1. Configuration of Content Generation System

[0022] A configuration of a content generation system according to an embodiment of the present invention will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the configuration of the content generation system according to the present embodiment.

[0023] As shown in FIG. 1, the content generation system 1 includes a user terminal 10 and a content generation device 20. The user terminal 10 and the content generation device 20 are communicably connected via a network NW. Note that the user terminal 10 and the content generation device 20 may be connected to each other by wired communication or wireless communication.

(1) Content Generation System

[0024] The content generation system 1 is a system that generates content in which a digital clone of a user explains material on behalf of the user. Examples of the content include image content, web content, three-dimensional (3D) content, and 3D hologram content. The image content is content that displays a digital clone using an image such as a still image or video (footage). The web content is content that displays a digital clone in a 3D space displayed on a web browser. The 3D content is content that causes a 3D digital clone to speak. The 3D hologram is content that projects a digital clone using a 3D hologram.

[0025] A digital clone is a digitized replica of a user. A digital clone is represented by a user's personal image (hereinafter, also referred to as “personal image”), and reads out the text of the material using the user's personal voice (hereinafter, also referred to as “personal voice”).

[0026] The content generation system 1 generates content by synthesizing the user's voice, the user's image, an image of the material, and the like. The content generated by synthesis is also referred to below as “synthesized content”. The synthesized content is generated by synthesizing image content, web content, 3D content, 3D hologram content, and the like according to the content used by the user.

[0027] In the following, an example will be described in which the content used by the user is image content, and the content generation system 1 synthesizes the user's voice, the user's image, the image of the material, and the like, and then generates a video as synthesized content that is used by a digital clone to explain the material (hereinafter referred to as “synthesized video”).

[0028] The content generation system 1 generates content based on the data of the material. The data of the material is data that at least includes data representing a first text (hereinafter, also referred to as “text data”), being a reading target, and data to be displayed so as to correspond to the content of the text data (hereinafter, also referred to as “display data”).

[0029] In the following, as an example, an example will be described in which the material is material used for a presentation. Furthermore, as an example, an example will be described in which the data of the material (hereinafter, also referred to as “presentation data”) is data that has been created using Microsoft PowerPoint. The PowerPoint data is data that includes both text data and display data. Specifically, the text data is text that has been input into a note portion. Further, the display data is mainly slides, and may also include videos and animations attached to the slides.

[0030] The content generation system 1 generates, based on the text data, a voice (hereinafter, also referred to as “synthesized voice”) that reads out the first text in the user's personal voice. The content generation system 1 generates the synthesized voice using a learned model that has been learned by machine learning. The learned model (hereinafter, also referred to as “voice generation model”) that generates the synthesized voice is a model that has learned, based on the user's voice when reading out a second text, being the learning target, a way of reading out the second text in the user's voice.

[0031] The way of reading by the user, being the learning target, is for example, an intonation, an accent, and a reading speed unique to the user.

[0032] When text data is input, the voice generation model is capable of generating and outputting a synthesized voice that reads out the first text represented by the text data in the user's personal voice and way of reading. Consequently, as a result of the content generation system 1 inputting the text data of the presentation data into the voice generation model, it is possible to acquire a synthesized voice in which the first text represented by the text data is read out in the user's personal voice and reading style.

[0033] Furthermore, the content generation system 1 generates a personal image for the digital clone (hereinafter, also referred to as “personal image for synthesis”) based on the user's personal image. The personal image for synthesis may be either a still image or video (footage). The content generation system 1 generates the personal image for synthesis using a learned model that has been learned by machine learning. The learned model that generates the personal image for synthesis (hereinafter, also referred to as “image generation model”) is a model that has learned the user's motions based on the user's personal image.

[0034] The user's motions that become the learning target are, for example, the user's facial movements and gestures. The user's facial movements are, for example, the movements of the mouth or changes in facial expression according to the reading. The gestures are, for example, movements of the head or hand, and body gestures according to the reading.

[0035] When the user's voice (such as a synthesized voice) is input, the image generation model is capable of generating a personal image for synthesis in which the user's motions change according to the voice. As a result, by inputting a synthesized voice, that has been generated based on the text data of the presentation data, into the image generation model, the content generation system 1 is capable of acquiring, as the personal image for synthesis, a personal image in which the motions change according to the synthesized voice.

[0036] Then, by synthesizing the generated synthesized voice and personal image for synthesis, the content generation system 1 generates data representing the user's digital clone (hereinafter, also referred to as “clone data”).

[0037] In the clone data, the first text, being the reading target, is read out in the user's personal voice, and the personal image changes as though the user is performing motions according to the content of the first text. For example, in the personal image, the user's mouth and the user's facial expressions change according to the first text that is being read out (the personal voice being output), and the personal image changes as though the user is moving the head and performing hand and body gestures.

[0038] In this way, as a result of the user's personal image changing according to the user's personal voice, mismatch that occurs between the voice and the image is reduced, and the sense of awkwardness felt by the listeners can be reduced.

[0039] The clone data is generated in a data format according to the content used by the user, such as image content, web content, 3D content, 3D hologram content, and the like.

[0040] In the following, an example will be described in which the content used by the user is image content, and video representing the user's digital clone (hereinafter, also referred to as “clone video”) is generated as clone data.

[0041] The content generation system 1 generates, based on the display data, an image (hereinafter, also referred to as “display image”) that is displayed so as to correspond to the synthesized voice. Furthermore, the content generation system 1 generates, based on the text data, data in a text format that is displayed as subtitles (hereinafter, referred to as “subtitle text”). In addition, by synthesizing the clone video, the display image, and the subtitle text, the content generation system 1 generates a video in which the user's digital clone is explaining the content of the material as a synthesized video. The synthesized video is an example of content generated by the content generation system 1.

[0042] In the synthesized video, the user's digital clone reads out the first text according to the content of the displayed material. As a result, it can be made to appear as though the user's digital clone is explaining the material on behalf of the user.

(2) User Terminal

[0043] The user terminal 10 is a terminal used by the user. The user terminal 10 includes input devices (such as a mouse, a keyboard, and a touch panel), an output device (such as a display and speakers), and a central processing unit. For example, the user terminal 10 may be any terminal such as a personal computer (PC), a smartphone, or a tablet.

[0044] The user operates the user terminal 10, and uploads to the content generation device 20 the information that is required to generate (learn) the voice generation model and the image generation model, and the information that is required to generate the synthesized video.

[0045] The information required to generate the voice generation model is the voice of the user reading out the second text (hereinafter, also referred to as “voice for learning”), which is the learning target. The voice for learning is generated, for example, by having the user actually read out approximately 200 pieces of second text. The information required to generate the image generation model is the user's personal image for learning (hereinafter, also referred to as “personal image for learning”). The personal image for learning may be either a still image or video (footage), but changes in the user's motions can be more accurately learned by the image generation model in the case of video.

[0046] The information required to generate the synthesized video is presentation data. The content generation device 20 generates, based on the uploaded presentation data, synthesized video using the voice generation model and the image generation model.

[0047] As a result of the user operating the user terminal 10, and downloading the synthesized video from the content generation device 20 and playing the synthesized video, it is possible for the user to have the digital clone give the presentation on their behalf.

(3) Content Generation Device

[0048] The content generation device 20 is a device that generates the synthesized video (an example of content). The content generation device 20 includes input devices (such as a mouse, a keyboard, and a touch panel), an output device (such as a display and speakers), and a central processing unit. The content generation device 20 is, for example, a server device implemented by a personal computer (PC).

[0049] The content generation device 20 generates the voice generation model, the image generation model, and the synthesized video based on the various information uploaded from the user terminal 10.

[0050] Specifically, the content generation device 20 generates the voice generation model based on the voice for learning that has been uploaded from the user terminal 10. Furthermore, the content generation device 20 generates the image generation model based on the personal image for learning that has been uploaded from the user terminal 10. Moreover, the content generation device 20 generates the display image based on the display data of the presentation data that has been uploaded from the user terminal 10. In addition, the content generation device 20 generates the subtitle text based on the text data of the presentation data that has been uploaded from the user terminal 10.

[0051] Also, the content generation device 20 generates the synthesized voice by inputting the text data of the presentation data that has been uploaded from the user terminal 10 to the voice generation model, generates the personal image for synthesis by inputting the generated synthesized voice to the image generation model, and generates the clone video by synthesizing the synthesized voice and the personal image for synthesis. Then, the content generation device 20 generates the synthesized video by synthesizing the generated display image, subtitle text, and clone video.

2. Functional Configuration of Content Generation Device

[0052] A configuration of the content generation system 1 according to the present embodiment has been described above. Next, a functional configuration of the content generation device 20 according to the present embodiment will be described with reference to FIGS. 1 to 7.

[0053] As shown in FIG. 1, the content generation device 20 includes a communication unit 210, an input unit 220, a storage unit 230, a control unit 240, and an output unit 250.

(1) Communication Unit

[0054] The communication unit 210 has a function of transmitting and receiving various information. For example, the communication unit 210 communicates with the user terminal 10 via the network NW. The communication unit 210 receives the voice for learning, which is information that is required to generate the voice generation model, during communication with the user terminal 10. Furthermore, the communication unit 210 receives the personal image for learning, which is information that is required to generate the image generation model, during communication with the user terminal 10.

[0055] Moreover, the communication unit 210 receives the presentation data, which is information that is required to generate the synthesized video, during communication with the user terminal 10. In addition, the communication unit 210 transmits the synthesized video in the communication with the user terminal 10.

(2) Input Unit

[0056] The input unit 220 has a function of accepting inputs. For example, the input unit 220 accepts information inputs that have been input using an input device, such as a mouse, a keyboard, or a touch panel, that are provided as hardware in the content generation device 20.

(3) Storage Unit

[0057] The storage unit 230 has a function of storing various information. The storage unit 230 is constituted by a storage medium provided as hardware in the content generation device 20, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access memory (RAM), and a read only memory (ROM), or any combination of these storage media.

[0058] As shown in FIG. 1, the storage unit 230 stores a voice generation model 231 and an image generation model 232. Furthermore, the storage unit 230 may also store the voice for learning, the personal image for learning, the presentation data, and the like, that the communication unit 210 has received from the user terminal 10. Moreover, the storage 230 may store the display image, the subtitle text, the synthesized voice, the personal image for synthesis, the clone video, the synthesized video, and the like, that have been generated by the content generation device 20.

(4) Control Unit

[0059] The control unit 240 has a function of controlling the overall operation of the content generation device 20. The control unit 240 is implemented, for example, by causing a central processing unit (CPU) provided as hardware in the content generation device 20 to execute a program.

[0060] As shown in FIG. 1, the control unit 240 includes an acquisition unit 241, a learning unit 242, a division unit 243, a playback time determination unit 244, a subtitle generation unit 245, a voice generation unit 246, an image generation unit 247, a synthesis unit 248, and an output processing unit 249.

(4-1) Acquisition Unit

[0061] The acquisition unit 241 has a function of acquiring various information. For example, the acquisition unit 241 may acquire the voice for learning, the personal image for learning, and the presentation data that the communication unit 210 has received from the user terminal 10. The acquisition unit 241 acquires, from the presentation data, the text data, which is the reading target, and the display data that is displayed so as to correspond to the content of the text data.

[0062] Here, an example of the presentation data acquired by the acquisition unit 241 will be described with reference to FIGS. 2 to 4. FIGS. 2 to 4 are diagrams showing an example of the presentation data according to the present embodiment. FIGS. 2 to 4 illustrate presentation data 30 including n slides 31-1 to 31-n (where n is a natural number). A display area DA1 of the presentation data 30 displays the slides 31-1 to 31-n. A display area DA2 of the presentation data 30 displays the slide that has been selected from among the slides 31-1 to 31-n. A display area DA3 of the presentation data 30 displays the first text corresponding to the slide that has been selected from among the slides 31-1 to 31-n.

[0063] FIG. 2 is a diagram showing a first slide. As shown in FIG. 2, the first slide 31-1 is displayed in the display area DA2, and the first text 32-1 corresponding to the slide 31-1 is displayed in the display area DA3.

[0064] FIG. 3 is a diagram showing a second slide. As shown in FIG. 3, the second slide 31-2 is displayed in the display area DA2, and the first text 32-2 corresponding to the slide 31-2 is displayed in the display area DA3.

[0065] FIG. 4 is a diagram showing an nth slide. As shown in FIG. 4, the nth slide 31-n is displayed in the display area DA2, and the first text 32-n corresponding to the slide 31-n is displayed in the display area DA3.

[0066] The acquisition unit 241 acquires, from the presentation data 30, the slides 31-1 to 31-n as display data, and the first text 32-1 to 32-n as text data.

(4-2) Learning Unit

[0067] The learning unit 242 has a function of generating a learned model. For example, the learning unit 242 generates a learned model by performing machine learning using learning data that has been acquired by the acquisition unit 241.

[0068] Specifically, the learning unit 242 uses the voice for learning that has been acquired by the acquisition unit 241 as teaching data to perform machine learning of a way of reading the text in the user's voice. As a result of the machine learning, the learning unit 242 generates the voice generation model 231, which is capable of generating and outputting, when text data is input, a synthesized voice that reads out the first text represented by the text data in the user's personal voice.

[0069] In the present embodiment, the learning unit 242 generates the personally unique voice generation model 231 by performing transfer learning of the voice for learning (teaching data) with respect to an existing learned model that has learned the text and a way of reading the text in advance. Note that the existing learned model for generating the voice generation model 231 is stored in advance in the storage unit 230. Furthermore, the existing learned model for generating the voice generation model 231 includes a dictionary representing general intonations and accents and is capable of reproducing the general intonations and accents.

[0070] When multiple existing learned models have been prepared, the learning unit 242 is capable of generating the personally unique voice generation model 231 for multiple users by performing learning using a single user's voice for learning with respect to a single learned model. In addition, when an existing learned model has been prepared for each language, the learning unit 242 is capable of generating the personally unique voice generation model 231 for each language by performing transfer learning of the user's voice for learning with respect to the learned model in each language.

[0071] The learning unit 242 writes and stores the generated voice generation model 231 in the storage unit 230.

[0072] Furthermore, the learning unit 242 performs machine learning of the user's motions using the personal image for learning that has been acquired by the acquisition unit 241. As a result of the machine learning, the learning unit 242 generates, as the learned model, the image generation model 232 which, when a synthesized voice is input, is capable of generating and outputting the personal image for synthesis in which the user's motions change according to the voice.

[0073] In the present embodiment, the learning unit 242 generates the personally unique image generation model 232 by performing transfer learning of the personal image for learning (teacher data) with respect to an existing learned model that has learned the changes in a person's motions in advance. Note that the existing learned model for generating the image generation model 232 is stored in advance in the storage unit 230. Furthermore, the existing learned model for generating the image generation model 232 is, for example, a model that has learned mouth movements synchronized with the voice in advance using a generative adversarial network (GAN).

[0074] When multiple existing learned models have been prepared, the learning unit 242 is capable of generating the personally unique image generation model 232 for multiple users by performing learning using the personal image for learning of a single user with respect to a single learned model.

[0075] The learning unit 242 writes and stores the generated image generation model 232 in the storage unit 230.

(4-3) Division Unit

[0076] The division unit 243 has a function of dividing the first text (text data). For example, the division unit 243 divides the first text into multiple pieces based on an input indicating the dividing locations of the first text. The input indicating the dividing locations is, for example, a line break. The division unit 243 divides the first text into multiple sentences at each line break. As a result of the division unit 243 dividing the first text, the first text can be read out in divided units of sentences. As a result, because the voice is interrupted each time a single divided sentence is read out, it is possible to provide pauses when reading out the first text.

[0077] For example, in the case of the first text 32-1 shown in FIG. 2, a line break is input after “Now, . . . as announced several days ago”. Although it appears that there is a line break after “Based on this medium-term management plan, . . . and focus on”, this is due to a line wrap display based on the display area of the note portion, and there is no line break input in this section. Therefore, the division unit 243 divides the first text 32-1 into two sentences, namely the sentence “Now, . . . as announced several days ago”, and the sentence “Based on this medium-term management plan, . . . strengthening the management base”.

[0078] Furthermore, in the case of the first text 32-2 shown in FIG. 3, a line break is input after “The following measures . . . as a key concept”. Therefore, the division unit 243 divides the first text 32-2 into two sentences, namely the sentence “The following measures . . . as a key concept.”, and the sentence “We will solve . . . raise our company valuation”.

[0079] Furthermore, in the case of the first text 32-n shown in FIG. 4, a line break has not been input. As a result, the division unit 243 does not divide the first text 32-n.

(4-4) Playback Time Determination Unit

[0080] The playback time determination unit 244 has a function of determining a playback time of the synthesized video. For example, the playback time determination unit 244 determines the playback time of the synthesized video based on the number of characters in the first text. Specifically, the playback time determination unit 244 determines, for each piece of acquired text data, a playback time of the display image corresponding to the text data by converting the number of characters in the first text represented by the text into a time.

[0081] Furthermore, the playback time determination unit 244 may determine the playback time of the synthesized video based on the reading speed of the synthesized voice. Specifically, the playback time determination unit 244 determines, for each piece of acquired text data, a playback time of the display image corresponding to the text data by calculating the time the reading ends based on the reading speed.

[0082] When there are multiple display images, the playback time determination unit 244 calculates a playback time of the synthesized video by adding together the playback time of each display image.

[0083] Note that the playback time determination unit 244 may determine the playback time of the synthesized video based on both the number of characters in the first text and the reading speed.

[0084] Furthermore, the playback time determination unit 244 may determine the playback time of the synthesized video by considering the pauses used while reading out the first text. For example, the playback time determination unit 244 calculates the time required to express the pauses according to the number of sentences that have been divided by the division unit 243 and add then the time to the playback time calculated based on the number of characters and the reading speed mentioned above.

(4-5) Subtitle Generation Unit

[0085] The subtitle generation unit 245 has a function of generating subtitle text. For example, the subtitle generation unit 245 generates the subtitle text based on the text data that has been acquired by the acquisition unit 241. Specifically, the subtitle generation unit 245 generates the subtitle text in units of sentences that have been divided by the division unit 243.

[0086] Note that, when the language used in the text data acquired by the acquisition unit 241 and the language corresponding to the voice generation model 231 are different, the subtitle generation unit 245 may generate subtitle text in which the text data has been translated to the language corresponding to the voice generation model 231.

(4-6) Voice Generation Unit

[0087] The voice generation unit 246 has a function of generating the synthesized voice. For example, the voice generation unit 246 uses the user's personally unique voice generation model 231 to generate a synthesized voice in which a first voice representing the text data that has been acquired by the acquisition unit 241 is read out in the user's voice.

[0088] The voice generation unit 246 generates a synthesized voice for each piece of text data (sentence) that has been divided by the division unit 243. As a result, because the voice is interrupted each time one of the synthesized voices that have been generated is read out, it is possible to provide pauses when reading out the first text represented by the text data.

[0089] Further, it is assumed that it is necessary to generate, from text data using a first language (such as Japanese), a synthesized voice that performs the reading in a second language (such as English) that is different from the first language. In this case, the voice generation unit 246 uses the voice generation model 231 corresponding to the second language, which is obtained by performing transfer learning based on the voice for learning (teacher data) in the first language, which has been used for learning by the voice generation model 231 corresponding to the first language. As a result, the voice generation unit 246 is capable of generating a synthesized voice in which the first text representing the text data in the first language is read out in the user's voice using the second language.

(4-7) Image Generation Unit

[0090] The image generation unit 247 has a function of generating various images for generating the synthesized video. For example, the image generation unit 247 generates a display image that is displayed so as to correspond to the synthesized voice based on the display data acquired by the acquisition unit 241. Specifically, the image generation unit 247 generates the display image by converting the display data into an image.

[0091] Furthermore, the image generation unit 247 uses the user's personally unique image generation model 232 to generate a personal image for synthesis in which the user's motions change according to the synthesized voice generated by the voice generation model 231.

[0092] For example, the image generation unit 247 generates a personal image for synthesis in which the user's facial movements change according to the reading by the synthesized voice. Specifically, the image generation unit 247 generates the personal image for synthesis in which the movements of the user's mouth and the user's facial expressions change according to the reading by the synthesized voice.

[0093] In addition, the image generation unit 247 may generate a personal image for synthesis which changes as though the user is making gestures according to the reading by the synthesized voice. Specifically, the image generation unit 247 generates a personal image for synthesis in which the user is moving the head and the user is performing hand and body gestures according to the reading by the synthesized voice.

[0094] In this way, the image generation unit 247 generates a personal image for synthesis that reproduces the user's movements when the user is reading out text or when giving a presentation. As a result, the clone shown in the clone video is capable of reading out text or giving a presentation while moving more naturally like the user.

(4-8) Synthesis Unit

[0095] The synthesis unit 248 has a function of performing various synthesis. For example, the synthesis unit 248 generates a synthesized video by synthesizing at least the synthesized voice generated by the voice generation unit 246, the display image generated by the image generation unit 247, and the user's personal image. Specifically, the synthesis unit 248 generates the clone video by synthesizing the synthesized voice generated by the voice generation unit 246 and the personal image for synthesis generated by the image generation unit 247. Then, the synthesis unit 248 generates the synthesized video by synthesizing the display image and the generated clone video.

[0096] Note that, when there are multiple slides (display data) and multiple first text (text data) as in the presentation data shown in FIGS. 2 to 4, a display image, a synthesized voice, and a personal image for synthesis are generated for each pair including a slide and first text. Consequently, the synthesis unit 248 generates a synthesized video by generating the clone video for each pair including a slide and first text. As a result, the same number of synthesized videos as number of slides is generated from a single piece of presentation data. Then, the synthesis unit 248 synthesizes the synthesized videos generated from the single presentation data to generate one final synthesized video.

[0097] Furthermore, when a single first text is divided into multiple sentences by the division unit 243, multiple synthesized voices are generated from a single first text by the voice generation unit 246. As a result, the image generation unit 247 generates a personal image for synthesis with respect to each of the synthesized voices. Consequently, the synthesis unit 248 generates a single clone video by synthesizing the synthesized voices and the personal images for synthesis such that they are associated with each other.

[0098] In addition, the synthesis unit 248 may generate the synthesized video by also synthesizing the subtitle text generated by the subtitle generation unit 245 in addition to the display image and the clone video. When synthesizing the subtitle text, the synthesis unit 248 may perform the synthesis such that only one subtitle text is displayed at once, or may perform the synthesis such that multiple subtitle text items are displayed at the same time.

[0099] Here, an example of a synthesized video generated by the synthesis unit 248 will be described with reference to FIGS. 5 to 7. FIGS. 5 to 7 are diagrams showing an example of a synthesized video according to the present embodiment. In FIGS. 5 to 7, portions of the synthesized video generated based on the presentation data 30 shown in FIGS. 2 to 4 are being respectively displayed.

[0100] In FIG. 5, as a portion of the synthesized video generated based on the presentation data 30, a synthesized video 40-1 that has been generated based on the slide 31-1 and the first text 32-1 shown in FIG. 2 is displayed. As shown in FIG. 5, the synthesized video 40-1 is constituted by a display image 41-1, a clone video 42-1, and subtitle text 43-1. When section of the synthesized video corresponding to the synthesized video 40-1 is played back, the user's clone being displayed in the clone video 42-1 reads out the displayed subtitle text 43-1 using the user's personal voice in the user's reading style. When the reading of the displayed subtitle text 43-1 is completed, the next subtitle text 43-1 is displayed. When the reading of all of the subtitle text 43-1 included in the synthesized video 40-1 has been completed, the display image 41-1 switches to the next image (that is, the next section of the synthesized video is played back).

[0101] In FIG. 6, as a portion of the synthesized video generated based on the presentation data 30, a synthesized video 40-2 that has been generated based on the slide 31-2 and the first text 32-2 shown in FIG. 3 is displayed. As shown in FIG. 6, the synthesized video 40-2 is constituted by a display image 41-2, a clone video 42-2, and subtitle text 43-2. When the playback of the section of the synthesized video corresponding to the synthesized video 40-1 is completed, the section corresponding to the synthesized video 40-2 is played back in the same manner as the section corresponding to the synthesized video 40-1.

[0102] In FIG. 7, as a portion of the synthesized video generated based on the presentation data 30, a synthesized video 40-n that has been generated based on the slide 31-n and the first text 32-n shown in FIG. 4 is displayed. As shown in FIG. 7, the synthesized video 40-n is constituted by a display image 41-n, a clone video 42-n, and subtitle text 43-n. The synthesized video 40-n is the last section of the synthesized video. The section of the synthesized video corresponding to the synthesized video 40-n is played back in the same manner as the sections corresponding to the synthesized video 40-1 and synthesized video 40-2, and when the playback of the section corresponding to the synthesized video 40-n is completed, the playback of the entire synthesized video is completed.

(4-9) Output Processing Unit

[0103] The output processing unit 249 has a function of controlling various outputs. For example, the output processing unit 249 transmits the synthesized video generated by the synthesis unit 248 to the user terminal 10. Furthermore, the output processing unit 249 may play back the synthesized video generated by the synthesis unit 248, transmit the video and voice being played back to the user terminal 10, and cause the user terminal 10 to output the footage and voice.

(5) Output Unit

[0104] The output unit 250 has a function of outputting various information. The output unit 250 is implemented by, for example, a display device such as a display or touch panel included as hardware in the content generation device 20, or a voice output device such as a speaker. The output unit 250 outputs, for example, a screen or a voice in response to an input from the output processing unit 249.

3. Processing Flow

[0105] A functional configuration of the content generation device 20 according to the present embodiment has been described above. Next, the processing flow performed by the content generation device 20 according to the present embodiment will be described with reference to FIGS. 8 to 10.

(1) Processing for Generating Voice Generation Model

[0106] The processing by which the content generation device 20 generates the voice generation model 231 will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of a processing flow in the generation of the voice generation model 231 according to the present embodiment. The content generation device 20 generates and prepares the voice generation model 231 in advance in order to generate the synthesized video.

[0107] First, as shown in FIG. 8, the acquisition unit 241 of the content generation device 20 acquires a voice for learning (S101). Specifically, the acquisition unit 241 acquires the voice for learning that the communication unit 210 of the content generation device 20 has received from the user terminal 10.

[0108] Then, the learning unit 242 of the content generation device 20 generates the voice generation model 231 (S102). Specifically, the learning unit 242 generates the user's personally unique voice generation model 231 by transfer learning of the voice for learning (teacher data) acquired from the acquisition unit 241 with respect to an existing learned model that has learned the second text, which is the learning target, and a text reading style in advance. Note that the existing learned model for generating the voice generation model 231 is stored in advance in the storage unit 230 of the content generation device 20.

[0109] Then, the learning unit 242 writes and stores the generated voice generation model 231 in the storage unit 230 (S103).

(2) Processing for Generating Image Generation Model

[0110] The processing by which the content generation device 20 generates the image generation model 232 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing an example of a processing flow in the generation of the image generation model 232 according to the present embodiment. The content generation device 20 generates and prepares the image generation model 232 in advance in order to generate the synthesized video.

[0111] First, as shown in FIG. 9, the acquisition unit 241 acquires a personal image for learning (S201). Specifically, the acquisition unit 241 acquires the personal image for learning that the communication unit 210 has received from the user terminal 10.

[0112] Then, the learning unit 242 generates the image generation model 232 (S202). Specifically, the learning unit 242 generates the user's personally unique image generation model 232 by transfer learning of the personal image for learning (teacher data) acquired from the acquisition unit 241 with respect to an existing learned model that has learned the changes in a person's motions in advance. Note that the existing learned model for generating the image generation model 232 is stored in advance in the storage unit 230 of the content generation device 20.

[0113] Further, the learning unit 242 writes and stores the generated image generation model 232 in the storage unit 230 (S203).

(3) Processing for Generating Synthesized Video

[0114] The processing by which the content generation device 20 generates the synthesized video will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of a processing flow in the generation of a synthesized video according to the present embodiment.

[0115] First, as shown in FIG. 10, the acquisition unit 241 acquires presentation data (S301). Specifically, the acquisition unit 241 acquires the presentation data that the communication unit 210 has received from the user terminal 10.

[0116] Then, the acquisition unit 241 acquires display data (S302). Specifically, the acquisition unit 241 acquires the slides included in the acquired presentation data as display data.

[0117] Next, the image generation unit 247 of the content generation device 20 generates a display image (S303). Specifically, the image generation unit 247 generates the display image by converting the display data acquired by the acquisition unit 241 into an image.

[0118] Then, the acquisition unit 241 acquires text data (S304). Specifically, the acquisition unit 241 acquires the first text in the note portion included in the acquired presentation data as text data.

[0119] Next, the division unit 243 of the content generation device 20 performs division processing (S305). Specifically, the division unit 243 divides a single first text into multiple sentences by dividing the first text acquired by the acquisition unit 241 at each line break.

[0120] Then, the playback time determination unit 244 of the content generation device 20 determines the playback time of the synthesized video (S306). Specifically, the playback time determination unit 244 determines the playback time of the synthesized video based on the number of characters in the first text, the reading speed, and the pauses between the divided first text.

[0121] Next, the subtitle generation unit 245 of the content generation device 20 generates subtitle text (S307). Specifically, the subtitle generation unit 245 generates the subtitle text in units of the divided first text.

[0122] Then, the voice generation unit 246 of the content generation device 20 generates a synthesized voice (S308). Specifically, the voice generation unit 246 inputs the sentences that have been divided by the division unit 243 one at a time to the voice generation model 231 stored in the storage unit 230. As a result, the synthesized voices are generated by the voice generation model 231. Further, the voice generation unit 246 acquires the synthesized voices that has been output from the voice generation model 231.

[0123] Next, the image generation unit 247 generates a personal image for synthesis (S309). Specifically, the image generation unit 247 inputs the synthesized voices that have been generated by the voice generation unit 246 one at a time to the image generation model 232 stored in the storage unit 230. As a result, the personal image for synthesis is generated by the image generation model 232. Further, the image generation unit 247 acquires the personal image for synthesis that has been output from the image generation model 232.

[0124] Then, the synthesis unit 248 of the content generation device 20 generates a clone video (S310). Specifically, the synthesis unit 248 generates the clone video by synthesizing, for each synthesized voice generated by the voice generation unit 246, the personal image for synthesis that has been generated by the image generation unit 247.

[0125] Next, the synthesis unit 248 generates a synthesized video (S311). Specifically, the synthesis unit 248 generates a clone video in which the display image generated by the image generation unit 247 and the subtitle text generated by the subtitle generation unit 245 have been synthesized and generates the synthesized video. After generating the synthesized video, the content generation device 20 ends the processing. Note that the content generation device 20 may write and store the generated synthesized video in the storage unit 230 or transmit the generated synthesized video to the user terminal 10 as required.

[0126] As described above, the content generation device 20 according to the present embodiment includes the acquisition unit 241, the voice generation unit 246, the image generation unit 247, and the synthesis unit 248.

[0127] The acquisition unit 241 acquires the text data representing the first text, which is the reading target, and the display data that is displayed so as to correspond to the content of the text data.

[0128] The voice generation unit 246 generates the synthesized voice that reads out the first text representing the acquired text data in the user's voice by using the voice generation model 231, which has learned, based on the user's voice when reading out the second text, which is the learning target, the way of reading out the second text in the user's voice.

[0129] The image generation unit 247 generates the display image to be displayed so as to correspond to the synthesized voice based on the acquired display data.

[0130] The synthesis unit 248 generates synthesized content by synthesizing the generated synthesized voice and display image, and the user's personal image.

[0131] With this configuration, when the synthesized content that has been generated is played back, the first text representing the text data, which is the reading target, is read out in the user's personal voice in the way of reading that more fully expresses the user's likeness.

[0132] Therefore, the content generation device 20 according to an embodiment of the present invention reduces the sense of awkwardness felt by the listeners when the text is read out and can more fully express the speaker's likeness.

4. Modifications

[0133] An embodiment of the present invention has been described. Next, modifications of an embodiment of the present invention will be described. Each of the modifications described below may be independently applied to an embodiment of the present invention or may be combined when applied to an embodiment of the present invention. Furthermore, each of the modifications may be applied so as to replace the configuration described in an embodiment of the present invention or may be additionally applied to the configuration described in an embodiment of the present invention.

[0134] For example, the voice generation unit 246 may express the user's facial expressions with the synthesized voice by adjusting parameters. The voice generation unit 246 sets parameters such as 80% joy, 20% surprise according to the content of the text data. Note that the types of emotions may be a combination of various emotions other than joy and surprise, such as sadness or anger. As a result, the voice generation unit 246 can generate a synthesized voice that is capable of expressing, in addition to a way of reading that is authentic to the user, various emotions according to the content of the text data.

[0135] Furthermore, the content generation device 20 may have a function that enables the synthesized content that has been generated by the synthesis unit 248 to be edited. The user inputs the edit content to the user terminal 10. The content generation device 20 edits the synthesized content according to the user inputs made with respect to the user terminal 10. In this function, for example, editing of the slides (display image), the subtitle text, the voice of the clone data, and the personal image is possible. When editing the slides, it is possible to edit the content of the slides, switch the display order of the slides, add new slides, delete existing slides, and the like.

[0136] In addition, in the embodiment described above, although an example has been described in which a clone video, in which multiple synthesized voices and multiple personal images for synthesis have been synthesized, is synthesized with respect to a single display image, the present invention is not limited to such an example. For example, multiple display images may be synthesized with respect to a clone video in which a single synthesized voice and a single personal image for synthesis have been synthesized. In this case, the display images are switched and displayed while the reading of the single synthesized voice is completed.

[0137] Moreover, in the embodiment described above, an example has been described in which the data of the material is PowerPoint data, but the present invention is not limited to such an example. For example, as long as the data of the material includes text data and display data, the data may also be data created using Microsoft Word, data created using Microsoft Excel, portable document format (PDF) data, and the like. In addition, the data of the material may also be a combination of data containing only text data and data containing only display data.

[0138] Also, in the embodiment described above, an example has been described in which the content generation system 1 is configured by the user terminal 10 (client) and the content generation device 20 (server), but the present invention is not limited to such an example. For example, the content generation system 1 may be configured by only a content generation device 20 that can be directly operated by the user. That is, the content generation device 20 may also play the role of the user terminal 10. In this case, the user is capable of generating and using the synthesized content without connecting a terminal to the network NW.

[0139] Furthermore, the functions of the content generation device 20 may be implemented by multiple devices. For example, the function of generating the voice generation model 231 and the image generation model 232 may be implemented by another device. In this case, the content generation device 20 can generate synthesized content in the same manner as in the embodiment above by storing the voice generation model 231 and the image generation model 232 that have been generated by another device in the storage unit 230.

[0140] In addition, in the embodiment described above, although an example has been described in which the acquisition unit 241 acquires the slides included in the presentation data as the display data, and the image generation unit 247 generates the display image by converting the display data acquired by the acquisition unit 241 into an image, the present invention is not limited to such an example.

[0141] A configuration is also possible in which the acquisition unit 241 acquires the text data to be read out by the digital clone without acquiring the display data, and the image generation unit 247 generates the personal image for synthesis (digital clone), in which the user's motions change according to the generated synthesized voice, and does not generate a display image that is displayed so as to correspond to the synthesized voice.

[0142] In the present modification, as shown in FIG. 11, the displayed synthesized video 50 is a video in which a synthesized voice and a digital clone 51 have been synthesized (clone video 52), and a display image generated by the image generation unit 247, such as the slides, is not included in the synthesized video 50.

[0143] In the present modification, subtitle text may be displayed in the clone video 52. On the other hand, when subtitle text is not needed, the subtitle generation unit 245 may be omitted from the constituent elements of the content generation device 20 in the embodiment described above.

[0144] Next, a processing flow performed when generating a synthesized video in the present modification will be described with reference to FIG. 12. As shown in FIG. 12, the content generation device 20 of the present modification executes the processes in the embodiment above in the order of S304, S305, S306, S308, S309, and S310. The aspects that are different from the embodiment above will be described below.

[0145] In S304, the acquisition unit 241 at least acquires the text data to be read out by the digital clone. Although the division processing in S305 and the playback time determination in S306 are both executed, it is possible to omit S305 and the division unit 243 that executes this process when the text data is already divided, and to omit S306 and the playback time determination unit 244 that execute this process if it is not necessary to determine the playback time.

[0146] In S309, although the image generation unit 247 generates the personal image for synthesis (digital clone), in which the user's motions change according to the generated synthesized voice, it does not generate the display image such as the slides that are displayed so as to correspond to the synthesized voice.

[0147] In the present modification, because the generated synthesized video includes a synthesized voice and a personal image for synthesis, but does not include a display image, the generation of the synthesized video is completed after generating the clone video in S310. That is, in the present modification, S311 in the embodiment described above is not essential. However, when subtitle text is included in the synthesized video, the clone video and the subtitle text may be synthesized into a synthesized video in S311.

[0148] Furthermore, in the embodiment described above, as shown in FIGS. 5 to 7, although an example has been described in which the display images 41-1 to 41-n such as the slides are two-dimensionally displayed, the present invention is not limited to such an example.

[0149] For example, as shown in FIG. 13, the content generation device 20 may generate 3D content in which the display image such as the slides and the digital clone (3D avatar) are arranged in a 3D space that is displayed in 3D. As shown in FIG. 13, in the synthesized video 60 generated by the content generation device 20 in the present modification, two display images 62 and 63 displayed in 3D, and a 3D avatar 64 are arranged in a 3D space 61. Each of the two display images 62 and 63 are diagonally expressed such that one end portion in the left-right direction of the synthesized video 60 (the end portion nearer the peripheral edge portion of the synthesized video 60) is positioned closer to the viewer side of the synthesized video 60 than the other end portion. However, the display of the 3D space is not limited to the example shown in FIG. 13, and the number, size, arrangement position, and direction of the incline and the like of the display images may be appropriately changed. The two display images 62 and 63 display the slides, still images, videos, subtitle text, and the like. A configuration is also possible in which comments and the like that have been uploaded by the viewers of the synthesized video 60 to a social networking service (SNS) are displayed in real-time in the two display images 62 and 63. In FIG. 13, although the 3D avatar 64 is expressed so as to be positioned further toward the viewer side than the two display images 62 and 63, the present invention is not limited to this, and the 3D avatar 64 may be made to move around in the 3D space 61 according to the explanation using the display image, the synthesized voice, and the like, and the 3D avatar 64 may also change the facial expressions, or move body parts such as the mouth, head, hands, feet, and torso. In addition, in the embodiment described above, although the clone video is firstly created from the synthesized voice and the user's personal image, and the synthesized video is then created using the clone video, in the case of 3D content using a 3D space, the synthesized video may be produced by creating a 3D avatar corresponding to the user by using the personal image, and then extracting a status such as the 3D avatar moving the mouth according to the synthesized voice in the 3D space. That is, the generation of the clone video and the generation of the synthesized video may be performed at the same time, and this is the same in the embodiment described above.

[0150] Furthermore, in the embodiment above, as shown in FIGS. 5 to 7, although an example has been described in which each of the synthesized videos 40-1 to 40-n display the display images 41-1 to 41-n, such as slides, still images, and videos, the clone videos 42-1 to 42-n, and the subtitle text 43-1 and 43-n at the same time, the present invention is not limited such an example.

[0151] For example, the synthesized video of the present modification may be a video in which the display image, such as slides, still images, and videos, the clone video, the subtitle text, and the like are consecutively combined, such that the display form changes with time. In other words, the synthesized video may be one in which at least one of the slides, still images, videos, clone videos, subtitle text, and the like, are displayed at a certain point in time of the synthesized video being played back.

[0152] Moreover, in the embodiment above, although an example has been described in which the image generation unit 247 generates a personal image for synthesis (personal image) in which the user's motions change according to the synthesized voice, and the synthesis unit 248 generates a synthesized video using the synthesized voice and the generated personal image for synthesis, the present invention is not limited to such an example.

[0153] For example, the synthesis unit 248 may generate the synthesized video using a personal image of a user that is independent of the synthesized voice (hereinafter, sometimes referred to as independent personal image). The independent personal image is a personal image in which the user's motions do not change according to the synthesized voice, and examples include a user's still image, and a video in which the mouth of the user's image (digital clone) is moving irrespective of the synthesized voice. The independent personal image may be an image acquired from the user terminal 10 via the communication unit 210 and may also be an image generated by the image generation unit 247 from the personal image for learning. When the independent personal image is acquired from the user terminal 10 via the communication unit 210, the image generation unit 247 of the content generation device 20 may be omitted.

[0154] The modifications of an embodiment of the present invention have been described above.

[0155] Note that all or part of the content generation device 20 in the embodiment described above may be implemented by a computer. The computer may include at least one processor and one memory. In this case, a program that achieves the functions may be recorded on a computer-readable recording medium, such that a computer system can read and execute the program recorded on the recording medium.

[0156] Note that the “computer system” referred to here includes an operating system (OS) and hardware such as peripheral devices.

[0157] Furthermore, the “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magnetooptical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built into the computer system. In addition, the “computer-readable recording medium” may include a medium that dynamically retains a program for a short period of time, such as a communication line that transmits a program through a network such as the Internet or a telecommunication line such as a telephone line, or a medium that retains the program for a given period of time in that case, such as a volatile memory of a computer system that serves as a server or a client.

[0158] Also, the program mentioned above may be a program for implementing some of the functions described above or may be a program that implements the functions described above in combination with a program recorded in advance in a computer system or may be a program that implements the functions using a programmable logic device such as a field programmable gate array (FPGA).

[0159] One aspect of the present invention is a recording medium that non-temporarily stores a program, wherein the program causes a computer to function as: an acquisition unit that acquires text data representing a first text, being a reading target; a voice generation unit that, using a voice generation model that based on a voice in which a user has read out a second text, being a learning target, has learned a way of reading out the second text in the voice of the user, generates a synthesized voice in which the first text represented by the acquired text data is read out in the voice of the user; and a synthesis unit that generates synthesized content by synthesizing a generated synthesized voice and a personal image of the user.

[0160] An embodiment of the present invention has been described above with reference to the drawings. However, specific configurations are not limited to those described above, and may include various design changes and the like within a scope not departing from the spirit of the present invention.

[0161] Various techniques have been proposed for communicating between users by using characters and the like that represent the users as avatars.

[0162] For example, in JP H11-312160 A below, a technique is described that uses an independent and moving character as an avatar to promote communication relating to a document between users. In this technique, voice data is prepared in advance by recording a user's voice, and by defining a facial expression of the avatar to change when the voice data is played back, the avatar can be made to appear as though it is reading out the text corresponding to the voice data. As a result, for example, it is possible to have the avatar give a presentation on behalf of the user, who is the speaker.

[0163] However, simply using a speaker's avatar and a speaker's voice data in the manner of the technique of JP H11-312160 A above does not sufficiently reproduce the speaker's likeness and can cause the listeners of the presentation to feel a sense of awkwardness. This sense of awkwardness hinders the concentration of the listeners on the presentation.

[0164] A content generation device, a content generation method, and a program according to embodiments of the present invention can reduce the sense of awkwardness felt by listeners when text is read out and can more fully express the speaker's likeness.

[0165] A content generation device according to an aspect of the present invention includes: an acquisition unit that acquires text data representing a first text, being a reading target; a voice generation unit that, using a voice generation model that based on a voice in which a user has read out a second text, being a learning target, has learned a way of reading out the second text in a voice of the user, generates a synthesized voice in which the first text represented by the acquired text data is read out in the voice of the user; and a synthesis unit that generates synthesized content by synthesizing a generated synthesized voice and a personal image of the user.

[0166] A content generation method according to another aspect of the present invention includes: an acquisition process for causing an acquisition unit to acquire text data representing a first text, being a reading target; a voice generation process for causing a voice generation unit to generate, using a voice generation model that based on a voice in which a user has read out a second text, being a learning target, has learned a way of reading out the second text in a voice of the user, a synthesized voice in which the first text represented by the acquired text data is read out in the voice of the user; and a synthesis process for causing a synthesis unit to generate synthesized content by synthesizing a generated synthesized voice and a personal image of the user.

[0167] A program according to yet another aspect of the present invention causes a computer to function as: an acquisition unit that acquires text data representing a first text, being a reading target; a voice generation unit that, using a voice generation model that based on a voice in which a user has read out a second text, being a learning target, has learned a way of reading out the second text in a voice of the user, generates a synthesized voice in which the first text represented by the acquired text data is read out in the voice of the user; and a synthesis unit that generates synthesized content by synthesizing a generated synthesized voice and a personal image of the user.

[0168] According to an embodiment of the present invention, it is possible to reduce the sense of awkwardness felt by listeners when text is read out, and to more fully express the likeness of a speaker.

[0169] Obviously, numerous modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.