Video-voicemail solution for wireless communication devices

[0001] This invention relates to the operation of mobile phone communication systems such as those including cellular phones or any form of mobile wireless communication device capable of voice communication, and in particular, enhancements to conventional realtime voice communications and voicemail storage and retrieval systems for mobile phones allowing the integration of still images and video clips.

[0002] Today, voicemail for mobile phones is simply that, storing voice messages when the receiving party is not available, to be retrieved at a later time. Essentially, voice mail for mobile phones operates in a very similar manner to voicemail for conventional office phones. To date, there has been no attempt to integrate images or video with voicemail for mobile phones or conventional office phones.

[0003] Communications between two parties where both voice and video are utilized is well known and is commonly referred to as video conferencing or teleconferencing. Sometimes this capability is also known as a “Net Meeting”. Here, a group of individuals, each having a computing device including microphone, speaker, video camera, and a connection to the World Wide Web, are able to communicate in a real-time manner through both video and voice mediums. The ultimate goal of a video conferencing system is to transmit a continuous stream of audio and video to and from each participant in the meeting or conversation, and to emulate as much as possible the interaction that would occur if all participants were in the same room. To do this, there will be compromises and limitations for many years to come relative to the desired functionality for video conferencing due to bandwidth limitations. In contrast to this goal and these compromises, the invention described herein uses existing bandwidth capabilities to selectively integrate images and video with voice communication in order to solve very specific and valuable problems. The manner in which this integration occurs has not been offered before.

[0004] The term “videomail” is often used in the industry, but in contrast to voice mail, does not refer to the storage of messages for later retrieval. Instead, it refers to attaching video clips to e-mails and a similar manner to the common practice of attaching files containing digital photographs to e-mails.

[0005] Cameras have, at times, been incorporated into mobile phones for surveillance purposes. Instead of broadcasting digital video via a data link with the mobile/cellular service provider, these phones broadcast an RF signal to a viewing receiver in the same manner as any other covert surveillance camera system.

[0006] Digital cameras are available as attachments to some PDAs (Personal Digital Assistants), some of which also are available with wireless connection to a service provider allowing accessibility to the World Wide Web. Some PDAs with wireless Web-interface capability can also function as mobile or cellular phones. Some mobile phones have added PDA capabilities, again with wireless Web-interface capability. Also, some mobile phones now have digital cameras available as options to allow pictures to be captured and attached to emails. However, none such devices offer the integration of still images or video with cellular voice communication or with voicemail for cellular phones following the existing paradigm of phone calls and voicemail messages.

[0007] A very successful capability offered by one cellular service provider (the Nextel Direct Connect® digital two-way radio service), allows frequent communications among a group of individuals who work closely together to be more convenient and less costly. As a result, this capability has been adopted by the majority of businesses that require frequent communications with individuals working at different locations in a local area, for the most part, businesses in the construction industry. This allows a manager, foreman, or responsible person to more easily keep track of the progress at a variety of locations, and more readily communicate to affect swift problem resolution. Unfortunately, these communications rely on the ability of the individuals involved to clearly describe situations and problems they observe in terms that the responsible person can understand in order to best make decisions and guide the remote workers. The ability for the responsible person to see the subject or problem area would significantly enhance the value of these communications.

[0008] A solution is needed that, given the bandwidth limitations of current and next generation cellular data transmission capability, provides an easy way for persons to communicate image and video information, while maintaining a communication paradigm that is familiar, basically the paradigm of phone calls and voicemails. Such a system would allow high resolution images to be transmitted when a high level of detailed is required, and alternately, video clips (which may be at lower resolutions) where spatial relationships and or motion information is required.

[0009] An enhanced communication and voicemail solution for mobile phones is described where still images and/or video clips are injected into the voice stream creating a “video-voice” call. When a receiving party is not available to take a video-voice call, this combined stream of voice and image information is stored at the mobile service provider in a manner similar to voice mail today. Then, stored video-voicemails may be retrieved at a later time by the receiving party. While the sending party may use a normal size mobile phone containing a miniature digital camera, the receiving party may view video-voicemail images on a variety of devices including a wireless mobile phone or PDA, or alternately a conventional PC connected to the World Wide Web.

[0010] Compared with continuous, full motion video, occasionally injecting a still image or video clip into the voice stream allows much higher resolution images to be sent given bandwidth limitations, allowing the receiving party to view a subject or situation in much greater detail. For use in business applications, conveniently viewing this more detailed information, synchronized with voice explanations, enables better decisions thereby saving time and money.

[0019] A better way to communicate between persons operating at remote locations and a manager, advisor, or person in authority, would include interspersing images and/or video clips within the voice communication stream, whether or not it is real time or a voicemail message left for future retrieval. Such a solution utilizes video clips where they are most effective (even with reduced resolution)—conveying motion information, or alternately conveying spatial information of the subject area by way of a “Pan” motion with the camera. In addition, high resolution still images can be injected into the voicestream to allow a very detailed view of a particular subject or problem area, synchronized with a verbal description and other related discussion.

[0020] Many applications will benefit from this new capability, including the construction industry, the medical and care industry, field service and repair, building inspection, insurance adjusters, or any application where people need a convenient way to document situations at remote locations and make this information available to others. Another specific application that will benefit from video-voice communications is that of emergency situations. When someone calling 911 to report an emergency can also provide video clips and high resolution still images, a dispatcher or paramedic receiving the call can much better understand the situation and even instruct the caller in a way that may save lives.

[0021] In general, integrating this capability with the familiar paradigm of the mobile phone call and voicemail is most convenient and useful. As discussed earlier, it is known to attach digital photos and digital video clips to emails. Emails are inherently digital, so this attachment is natural. Emails are also not a realtime communications medium. However, it is not known to attach digital photos and digital video clips to voice communications, whether realtime or as stored messages.

[0022] Note that, throughout this specification, the terms “mobile phone”, “cellular phone”, and “wireless phone” are synonymous and refer to any mobile communications device capable of bi-directional voice communication.

[0023] FIG. 1 shows a flow chart for voice and image information within the scope of the present invention. Here, calling party 1 transmits a stream of voice information with still images and/or video clips interspersed throughout. These may be part of a real-time conversation with receiving party 4 where the receiving party interactively communicates (voice) with the calling party while receiving images that are displayed on the receiving device. Scenario 7, describing the receiving party's communication during a realtime conversation, indicates that the receiving party may communicate via some form of wireless phone device with digital capability communicating directly via the service provider, or alternately may communicate through the World Wide Web using a conventional PC with multimedia and voice communication capability (i.e., speaker and microphone or some form of headset).

[0024] If the receiving party is not available for realtime communications, a message may be stored at service provider 2 in database 3 designed to store voice, image, and video clip information while retaining the time relationships between the three. Per scenario 6, the retrieving party accesses previously stored voice, image, and video clip information directly from the service provider via some form of wireless phone device. Alternately, per scenario 5, the receiving party may access stored voice, image, and video clip information via the World Wide Web through either wired or wireless Web access. Although FIG. 1 only shows one mobile service provider, it is possible that there is more than one service provider in the link shown between the calling party and the receiving party. For simplicity, only one is shown here. Throughout this specification and the attached claims, the term “mobile service provider” will refer to one or more service providers who support mobile (or cellular or wireless) communication.

[0025] For cellular voice communications today, real-time conversations are never recorded. Only voicemail messages are recorded. As relates to video-voice conversations, it is however useful to record real-time communications that contain either still images or video clips injected into the voice stream. This may be desired in order to document the visual information being conveyed as part of the conversation for later retrieval. It may also be desired when the receiving party answers a call and, as part of the conversation, subsequently realizes that images or video clips are being transferred, but can't properly or safely view them.

[0026] With the ability to have voicemail or video-voicemail messages stored at the mobile service provider retrieved via the World Wide Web, users can archive conversations, voicemail messages, and video-voicemail conversations and messages on their personal or business computer system. Today, there is not a convenient way to archive mobile voicemail messages.

[0027] If the receiving party is not in a position to view these images real-time, such as when driving a car, having the conversation recorded including all images will allow them to review the visual and audio information by retrieving the recorded message at a later time. Alternately, if the receiving party is driving or otherwise in a situation where viewing is inconvenient or impossible, it may be useful to have the ability to transfer a real-time conversation into video-voicemail, if the conversation reaches a point where it is meaningless to continue real-time without the receiving party being able to view the images or video clips. This capability may be implemented by always recording realtime video-voice conversations at the mobile service provider, and discarding the information at the end of the conversation if the receiving (or calling) party has not taken some action (like pressing some button) to initiate the saving of the video-voice conversation. Alternately, the user could set the default mode to be that of automatically saving conversations, deleting them later if not needed or deleting them after they have been downloaded via the Web and archived. A variation on this these would include automatically saving conversations from a particular calling party, and deleting them later if not needed or deleting them after they have been downloaded via the Web and archived. The accumulation of information resulting from these recording scenarios would require a much larger amount of memory for storing messages at the service provider, but if this is valuable, it is a service that users would be willing to pay extra for.

[0028] The scenarios just described can also be applied to recording realtime voice conversations for later retrieval and archive. Even archiving voice alone can be a powerful tool within business applications. For recording realtime conversations, the issue of permissions and privacy arises. One easy to handle the granting of “permission to be recorded” would be that permission is deemed to be given to record conversations from or to a particular phone number by calling from that particular phone number and taking a prescribed action which could include entering a specified code. It may instead be desired to record only video-voice conversations and not voice-only conversations, again with permissions having been given.

[0029] Although the essence and value of a video-voice conversation as described herein is bi-directional for voice communication, but mostly or solely unidirectional with regard to the transmission of video clips and still images, the methods described may in certain circumstances, be valuable in a bi-directional manner. For instance, a worker at a jobsite may send images to a supervisor at a remote location, and the supervisor may, in return, send an image of a portion of a blueprint while pointing out some specific details to resolve certain issues. Of course, this requires that the worker have a device with a display capable of presenting the blueprint image with enough resolution to properly discriminate the necessary information.

[0030] Another purpose for recording video-voice messages involves a reverse scenario where the supervisor/foreman/responsible party may wish to record information and instructions concerning a particular job site, such that the information and instructions may be viewed by workers at a later time before commencing work or as part of problem resolution during the job.

[0031] FIG. 2 shows one variation of a camera integrated according to this invention with a conventional cellular phone. While cameras are sometimes offered as external options to mobile phones, such solutions may not be rugged enough for business use, especially in the construction industry. Incorporating the camera within the phone is a simpler and more rugged solution. While the variation shown in FIG. 2 does not allow the user to photograph themselves while viewing the display, such as would be required for a mobile video conference or net meeting, this capability is not required where the purpose of the video-voice conversation is remote viewing. FIG. 2a is a front view of the same phone.

[0032] Notice in FIG. 2a that button 8 located on left side is positioned such that the index finger of the right hand may easily press this button to activate the “shutter” for the integral digital camera. Alternately, having the button on the right side would allow operation by the index finger if the user's right hand covers the keypad. These alignments would be reversed for a left-handed person. Either way, having the “shutter button” on the side allows a more firm grip on the camera body, allowing a steadier picture, when pushing the “shutter” button.

[0033] If the integral camera and associated electronics are designed to allow either still images or video clips to be captured and transmitted, there are at least two methods of utilizing shutter button 8 to easily support both. In the first method, the button may be pressed briefly and released to record a still image, or pressed and held for a longer duration to enable a video clip to be transmitted during the duration of time for which the button is held. In the second method, button 8 may be partially depressed to capture a still image or fully depressed to capture a video clip. Alternately for each of the above methods, which button action captures still images and which captures video clips may be reversed. In addition to the methods just described for determining when video-clips are captured as opposed to still images, a more conventional method can always be utilized where the desired mode of capture is first selected through a key or combination of key presses on the phone's keypad, followed by pressing a “shutter” button.

[0034] Although this specification refers to transmitting both still images and video clips, an implementation may only deal with one of the two. If the bandwidth limitations are severe, occasionally injecting high resolution still images into the voicestream is probably more valuable than video clips. Eventually, when the available bandwidth is at a level where high resolution video clips can be easily sent via mobile communications, sending only video clips will be appropriate. In the interim, the combination of high resolution still images and lower resolution video clips may be the best overall compromise.

[0035] FIG. 2b shows a rear view of the cell phone revealing battery cover 9 and an integral sliding protective cover 10 that protects the lens for the integral digital camera. The sliding protective cover allows the phone to have a normal conformation when the cover is closed and provides maximum protection for the integral camera. Alternately, an integral protective cover might be hinged at one end and “flip-open” to expose the camera lens. In any case, for a robust solution for business use, an integral protective cover must be always attached to the main body of the phone so that it can be easily restored to its protective position after the user is finished using the camera feature, and so that the cover will never be misplaced or lost.

[0036] FIG. 2c shows another rear view of the same phone where protective cover 10 has been withdrawn to reveal camera lens 11. Note that, as for conventional digital cameras and video cameras, an optical zoom capability may be added.

[0037] FIG. 3 shows an application example where a construction worker (the sending party), in this case a mason, is communicating with his supervisor regarding a problem with a brick column that has just been constructed. As shown in image 12 and enlargement 13, the sending party is holding the phone in front of them like a camera in order to capture image and/or video information. In this mode, the display on the phone should temporarily act like the viewfinder display on a digital camera. This mode can be activated by a button on the keypad, a push of the shutter button 8, or some other mechanism. When button 8 is released, normal phone display information usually consisting of digits and icons can be optionally superimposed over the camera display, in a black or white (reversed) format.

[0038] Since the user of the phone/camera will be holding the device in front of them, they will not be able to talk directly into a normal microphone. Hence, it is necessary to have either a speaker phone capability, or some form of wired or wireless headset to allow bi-directional voice communication while the camera function is being utilized.

[0039] Looking again at FIG. 3, voice, image, and video clip information is either transferred in real time by way of the service provider 15 to the receiving party, or alternately is stored as a video-voicemail message at the mobile service provider for retrieval at a later time. In addition, as mentioned previously, a real-time conversation with image and video clips added may also be stored at the service provider to further document a situation for later retrieval. Note that during any conversation where image and video clip information is being transferred, bi-directional voice communication will occur for real-time conversations.

[0040] In FIG. 3, the camera is incorporated into a conventional-looking cell phone 14 and is observed by the receiving party on a cell phone/PDA combination unit 16 allowing a larger, more detailed view of images enabling the receiving party to make better decisions.

[0041] FIG. 4 shows how images are positioned in voice stream 17 of a video voicemail conversation with the initial position in time of the images relative to the voicestream being maintained for all such information transmitted. Here three high-resolution still images, image 18, image, and image 20, are injected into the voice stream at different times, a voice description typically coordinated with each image to explain any issues. To allow this discussion to continue while the receiving party is viewing a related image, the last still image transmitted or the last frame of a video clip will typically be maintained on the screen of the receiving device until superseded by another image or video clip, or until otherwise terminated by an action of the receiving party.

[0042] FIG. 5 shows both a video clip sequence 21 and a high resolution still image 22 injected into the voice stream 17, such that the initial position in time of images and video clips relative to the voicestream are maintained for all such information transmitted. Notice that the video clip is able to convey spatial orientation by sweeping (panning) from left to right, thereby positioning a specific focal point properly within its surrounding environment. In this case, a construction crew has uncovered a pipe and in the process has broken it in one specific place. The pan video clip sequence sweeps along the length of the pipe, ending the sequence at the specific location of interest, where high resolution still image 22 provides a close-up detail of the break itself.

[0043] Where, according to this invention, video clips or high-resolution still images are injected into the voice stream, the last image to be captured according to the action of shutter button 8 will typically be maintained on the display of the capturing phone for a predetermined time period after button 8 is released, or until otherwise terminated by a subsequent action of the sending party. Also, when button 8 is released, normal phone display information may optionally again be superimposed on the displayed image if desired.

[0044] Since it is desired that the party capturing and sending images can clearly and easily observe what images are being captured, it may be necessary to add some form of sun-shade to allow clear observation of an LCD “viewfinder” display on the phone/camera. Alternately, or in addition, it may be useful to add a polarizing filter over an LCD display for better viewing in bright sunlight. Other display technologies, more easily viewed in direct sunlight, may be utilized. It may also be useful to add an optical viewfinder such as those found in many conventional digital cameras.

[0045] FIG. 6 shows some examples of additional types of video-voice enabled viewing devices, with emphasis on the variation of aspect ratios among them. Previously, unit 16 in FIG. 3 showed a cellular phone device that opens into a wireless data enabled PDA, having a very wide format screen. Shown additionally in FIG. 6 are a conventional flip-phone 23 having a relatively standard aspect ratio screen, and a wireless enabled Palm Pilot PDA 24, having an aspect ratio that is unusually tall in the vertical direction.

[0046] FIG. 7 shows a conventional PC 25, in this case a notebook computer with a large high-resolution screen, having a fairly typical aspect ratio. The large variation in aspect ratio between the screens as shown in FIGS. 3, 6, and 7 provides an opportunity to pre-process video clip and image data at the service provider to better match the aspect ratio of a receiving device, before that information is sent to the particular receiving device. Essentially, when matching the aspect ratio of a particular receiving device, there is information that will not be displayed anyway, and removing (cropping) this extraneous information before transmitting, can reduce the amount of time acquired to transmit video clip and image information to a particular receiving device.

[0047] Therefore, a methods and apparatus for implementing a combination video/voicemail system especially useful in the construction industry and other industries requiring remote viewing with guidance and supervision, has been described

[0048] It should be understood that the particular embodiments described above are only illustrative of the principles of the present invention, and various modifications could be made by those skilled in the art without departing from the scope and spirit of the invention. Thus, the scope of the present invention is limited only by the claims that follow.