System and method for personalizing electronic mail messages by rendering the messages in the voice of a predetermined speaker

[0002] The present invention relates generally to the field of electronic messaging and, more particularly, to a system and method of personalizing received electronic messages through text-to-speech conversion. Electronic messaging, especially Internet e-mail using protocols such as Post Office Protocol (POP3) or Messaging Application Programming Interface (MAPI) has become extremely popular over the past several years. Besides being able to send plain text messages, users sending e-mail today are able to format messages in Hypertext Markup Language (HTML), attach practically any kind and number of supplementary documents or files, including images, executable programs, documents created by a word processor, sound files, video files, and many others. Attachments can be sent in compressed or uncompressed formats. The capability of sending audio files as e-mail attachments has led some users to use e-mail in a manner similar to voice mail.

[0003] The present invention provides a system and method for personalizing electronic messages by rendering them to recipients in the voice of a predetermined human speaker. The invention may be implemented by associating with a message payload a set of basis vectors comprising speech parameters derived from the predetermined human speaker.

[0004] FIG. 1 is a circuit block diagram of an environment of the invention.

[0005] FIG. 2 is a flowchart illustrating the operation of one aspect of the invention.

[0006] FIG. 3 is a simplified block diagram of a preliminary training process through which a set of basis vectors representing the speech parameters of a human speaker are derived.

[0007] FIG. 4 is a flowchart illustrating the operation of yet another aspect of the invention.

[0008] FIG. 5 is a flowchart illustrating the operation of a further aspect of the invention.

[0009] FIG. 6 is a circuit block diagram of an environment of the present invention.

[0010] FIG. 7A is a flowchart illustrating the operation of the present invention within the environment shown in the circuit block diagram of FIG. 6.

[0011] FIG. 7B is a flowchart illustrating the operation of another embodiment capable of operating within the environment shown in the circuit block diagram of FIG. 6.

[0012] FIG. 8A illustrates the relationship between the message payload and the attachment indicator (AI) referenced in the flowchart of FIG. 7A.

[0013] FIG. 8B illustrates the relationship between the recipient's message retrieval request and the rendering indicator (RI) referenced in the flowchart of FIG. 7B.

[0014] As illustrated in FIG. 1, the present invention will be described assuming that there is at least one human user communicating electronically over a communications network 100. Each user has some kind of communication device, 110 and 120 respectively, such as a desktop computer, laptop computer, palmtop computer, Personal Digital Assistant (PDA), cellular phone, PCS phone, cordless phone, satellite phone, or similar device. The term “electronic message” as used hereafter refers to a data object from a messaging protocol such as electronic mail or e-mail, Short Message Service (SMS), or other electronic messaging schemes.

[0015] Referring now to FIG. 2, a process 200 is illustrated. A message payload is created at 210, either automatically by a communication device or by a human sender using a communication device. The message payload may comprise simply text, or it may comprise a combination of a text message and one or more attached documents or files of various types (e.g., word processor files, image files, audio files, spreadsheets, executable programs, or others), some of which may also be, at least in part, text. In some instances, the message payload may comprise a null (blank) text message with one or more attachments. At 220, the set of basis vectors associated with a predetermined human speaker is additionally attached to the message payload, whether there are other attachments already present or not. The combined message payload and attached set of basis vectors, which may comprise the electronic message, are then transmitted through a communications network at 230. The electronic message is received at the remote end at 240. The recipient's communication device 120 processes the electronic message to separate the basis vectors from the message payload and, optionally, to synthesize as computer generated speech the text contained in the message payload, including the primary text message and any text attachments or text portions thereof, based on a voice model defined by the received set of basis vectors at 250. At 260, the recipient's communication device renders the text to the recipient as audible speech. One important application of the invention is allowing the recipient of an electronic message to hear the text portions thereof read in the voice of the sender. However, a sender may attach basis vectors corresponding to a speaker other than him or herself.

[0016] FIG. 3 illustrates a preliminary training process 300 by which a set of parameters representing a specific human speaker may be derived for use in text-to-speech applications. The training process 300 is performed at least once for each human speaker whose speech parameters are to be used with the invention. At 310, sample speech from the desired speaker is input to a training algorithm 320, from which speech parameters such as Linear Predictive Coding (LPC) coefficients are produced as output at 330. With sufficient training, a particular speaker's voice can be modeled for text-to-speech applications using a relatively small amount of coefficient data compared to an actual audio recording of the speech. The speech parameters associated with a particular speaker will hereafter be referred to as “a set of basis vectors.” Those skilled in the art will also realize that other methods of speech synthesis such as formant synthesis may also be used with the invention. Many speech analysis and synthesis methods can be used in the present invention as long as the speaker's voice may be modeled by a set of speech parameters that can be used subsequently to perform text-to-speech conversion.

[0017] FIG. 4 illustrates the operation of another process 400 to avoid unnecessarily sending the set of basis vectors, further saving network bandwidth. As shown in FIG. 4, the set of basis vectors is attached to the message payload by the sender's communication device only if it has not been sent previously 410 to a given recipient. The sender's communication device 110 maintains a record of which particular set of basis vectors has been sent to which recipients, and proceeds accordingly when new messages are created and sent. Those skilled in the relevant art are aware that it is easy to add such record keeping to the address book portion of an e-mail client software program, for example. If the specified set of basis vectors has already been sent, only the message payload is transmitted to the recipient. Otherwise, the set of basis vectors is attached and sent along with the message payload. In either case, the data object sent through the network by the sender is called the “electronic message.” Upon receiving the electronic message, the recipient's communication device determines at 420 whether or not a set of basis vectors is attached. If so, any text portions of the message payload are, optionally, synthesized as speech (250 and 260) using the attached set of basis vectors as input to the speech synthesizer. If, however, a set of basis vectors is not attached, the recipient's communication device 120 examines an identifier within the message payload at 430. The identifier may comprise the electronic return address of the message payload (e.g., [email protected]), a domain name (e.g., “mynetwork.net”), an alias or nickname, an e-mail account user name (e.g., “john.doe85”), an IP address, or the name of a person from the “name” field in the message header (e.g., “John Doe”). Data elements such as the sender's name and return address may be automatically inserted in the message header by outgoing email protocols such as Simple Mail Transfer Protocol (SMTP). At 430, an attempt is made to match the identifier with a set of basis vectors stored at the recipient's communication device. If a match is found, processing proceeds to 250 and 260 to render the text as audible speech. If no match is found, the text payload is simply delivered to the recipient in the conventional fashion at 440. The recipient's communication device may select the appropriate set of basis vectors matching the identifier of the message payload from among a plurality of such sets of basis vectors, depending on from how many users the recipient has received such sets of basis vectors.

[0018] Referring now to FIG. 5, yet another process 500 is illustrated that provides greater flexibility in associating a set of basis vectors with the specific human speaker from which it was derived. In some cases, different individuals send e-mail messages from the same e-mail account, meaning the same return address and sender's name are associated with each sent message, regardless of who the actual sender was at the time. This is true if an entire family, for example, shares a single e-mail address. To differentiate the intended speaker from the sender of the message, a speaker identifier is added to the electronic message at 510. In one implementation, the speaker identifier may be added to the message header. The speaker identifier is an electronic label that corresponds to a specific set of basis vectors. In one implementation, it may correspond to the name of the speaker and comprise ASCII text added to the message header as a separate field. The speaker identifier tells the recipient's communication device 120 which set of stored basis vectors to associate with the received message payload when a set of basis vectors is not attached thereto, allowing multiple senders to share the same return address. At 520, the sender's communication device determines whether a set of basis vectors corresponding to the speaker identifier has been sent previously. If not, the appropriate set of basis vectors is attached at 530. If so, the message payload proceeds to the remote end (240) without an attached set of basis vectors. At 540, the recipient's communication device uses the included speaker identifier to associate a specific set of stored basis vectors with the message payload.

[0019] Process 500 thus enables, for example, a brother and sister to have sets of basis vectors corresponding to their respective voices stored on their grandmother's communication device from previous messages, to send new text messages to their grandmother without attaching the sets of basis vectors, and to share the same return e-mail address. When the grandmother receives a message from the granddaughter, for example, the grandmother's communication device automatically uses the granddaughter's speaker identifier included in the message payload to associate the granddaughter's pre-stored set of basis vectors with the message and to render the text portions as synthesized speech.

[0020] Referring now to FIG. 6, an exemplary environment 600 is illustrated. In two embodiments assocated with the environment of FIG. 6, the invention makes use of one or more mail servers associated with the sender, the recipient, or both. The environment 600 comprises a sender communciation device 610, a cellular base station 620, an outgoing e-mail server 630, a communications network 640, an incoming e-mail server 650, a cellular base station 660 and a recipient communication device 670. The sender uses the communication device 610 to communicate with the communications network 640, such as the public switched telephone network (PSTN), via the cellular base station 620. An outgoing e-mail server 630 associated with the sender uses a protocol such as SMTP to send e-mail messages over the Internet, or any such network, to the recipient's incoming e-mail server 650. Those skilled in the art, however, will realize that the sender and receiver may share the same mail server so that, in reality, only one mail server may be involved. The recipient's incoming e-mail server 650 uses a protocol such as Post Office Protocol (POP3) to retrieve the received e-mail message and forward it to the recipient's communication device 670 via cellular base station 660.

[0021] Referring now to FIG. 7A, a process 700 operating within the environment shown in FIG. 6 is illustrated. While creating the message payload at 210, the sender has the option of specifying that a set of basis vectors corresponding to a chosen speaker be attached to the message payload at the sender's outgoing e-mail server 640. This is done by means of an attachment indicator (AI) added at 710 to the message payload, such as to the message header. In one implementation, the AI is simply a bit (or ASCII digit) equal to 1 if the set of basis vectors is to be attached and 0 if the set of basis vectors is not to be attached to the message. In either case, only the message payload is uploaded from the sender's communication device to the sender's outgoing e-mail server 640 at 720. When the message payload is received at mail server 640, the mail server 640 checks the status of the AI at 730. If the AI is asserted, the mail server 640 attaches the set of basis vectors specified by the identifier to the message payload and forwards the combined electronic message to the next network node. Processing then proceeds at the recipient's end as explained above. The sets of basis vectors are stored at the outgoing e-mail server 640, not in either the sender's or recipient's communication device. Optionally, the set of basis vectors may be stored along with the sender's account profile at mail server 640.

[0022] FIG. 8A illustrates the relationship between the message payload and the AI. The message payload 810 contains the added AI 820, preferably in the header portion of the message 830, which contains other fields 840.

[0023] Referring now to FIG. 7B, another process 705 operating within the environment shown in FIG. 6 is llustrated. It is assumed that the sender has not attached a set of basis vectors to the message payload. The message is subsequently received by the recipient's incoming e-mail server 650 at 750. The recipient, at 760, requests the message from the server 650. When the recipient sends the request to download new messages from mail server 650, the recipient also sets an optional rendering indicator (RI) within the message retrieval request. This data element is very similar to the AI explained previously and may be a bit or single ASCII digit that controls whether or not the incoming e-mail server 650 attaches a set of basis vectors for text-to-speech processing to the received message payload. If the RI is asserted at 770, the mail server 650 attaches the set of basis vectors indicated by the identifier included in the message payload at 780. The various sets of basis vectors corresponding to different speakers are stored at the incoming e-mail server 650 rather than being stored at either the sender's or the recipient's communication device. It will be appreciated by those skilled in the art that the processes 700, 705 support the sending of different sets of basis vectors corresponding to different speakers from a single e-mail address.

[0024] FIG. 8B illustrates the relationship between the message retrieval request and the RI. The RI 830 is included within the message retrieval request 840, which also contains other message data 850, at a suitable location.

[0025] It will be appreciated by those of ordinary skill in the art that the present invention can be embodied in other specific forms without departing from the spirit or essential character thereof. The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive.