Autonomous computing and telecommunications head-up displays glasses

[0001] This application claims the benefit under 35 U.S.C §119(e) of U.S. provisional patent application No. 61/852,325, which was filed on Mar. 15, 2013 and which is incorporated here by reference in its entirety to provide continuity of disclosure.

[0002] A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0003] N/A

[0004] Field of Invention

[0005] This invention pertains to the fields of mobile wearable computing and digital eyeglasses and telecommunications devices that incorporate various electronic features such as sensors, computer processors, data storage, battery powered cameras and audio-visual feedback mechanisms.

[0006] Related Art

[0007] The emerging product category of heads up display eyeglasses are gaining media exposure. These products typically incorporate a single forward-facing camera and a digital display positioned proximate to the eyeglass lens and off to one side so as to minimize the occlusion of the wearer's field of view.

[0008] According to one aspect, there is provided an apparatus which can comprise a pair of eyeglasses having a frame and lenses mounted on said frame; a computer processor mounted on said eyeglasses; a digital display mounted in one of said lenses and operably coupled to said processor; a plurality of digital cameras mounted on said eyeglasses and operably coupled to said processor; and a digital projection system mounted on said eyeglasses and operably coupled to said processor.

[0009] In one embodiment, Smart Eye Glasses have stereo digital cameras, stereo human-audio interface, directional microphones, eye-tracking sensors, and digital projection system (“Smart Glasses”).

[0010] Purposes of the invention include enabling wearers of the device to use the autonomous computing and telecommunications glasses as primary phone, Internet to communicate wirelessly and to record and view live video feeds of their surroundings in the user's eyeglasses while interacting with these video feeds in a way that enables the wearer to be able to dynamically perform a variety of functions such as to automatically zoom in on objects of interest which are in the user's field of view or visible from one of the camera feeds and to communicate with said device using a variety of inputs such as eye movements, hand movements and finger gestures as well as voice commands and button selections.

[0011] The smart glasses can incorporate dual forward-facing cameras, enabling the camera to record wide angle and stereographic video or still image content. In addition, the smart glasses can incorporate side-facing and rear-facing cameras such that it is able to record a 360 degree surround video feed.

[0012] The cameras can incorporate electro magnetic, electro-static or electromahical-mechanical optical zoom capabilities as well as digital zoom capabilities. With infrared enabled active auto-zoom and autofocusing system. The IR sensor enables low light and night vision.

[0013] These cameras can also incorporate directional electro-mechanical gimbal mounts, allowing them to pitch and yaw for greater directional control, according to some embodiments.

[0014] The eyeglasses can be selectively transparent or partially transparent, or opaque, according to some embodiments. In addition to a heads-up OLED display or Pico-Project system

[0015] Heads-up display data can be projected into the eyeglasses such that it is superimposed onto the real-world view that the wearer sees through the glasses or it can replace the real-world view if the glasses opacity is increased.

[0016] Dual inward-facing eye motion tracking sensors are integrated into the frame of the eyeglasses above the left and right eyes. These give the smart glasses the ability to track the wearer's eye movements as well as determine the wearer's distant point of focus. Eye tracking can be used to control a wide variety of device functions including the cameras' zoom functions and the integrated telecommunications functions. For example, the user can direct the device to scroll a surround video feed left or right, simply by looking left or right. Alternatively, in another mode, the edges and corners of the user's heads-up display glasses can be logically mapped to represent a bird's eye view of the surrounding scene. To instruct the smart glasses to display the video feed from the rear-facing cameras, the user can briefly look down, to display video from the right, the user looks to the right of the display and to display video from the front, the user looks up.

[0017] IR Sensor, IR mapping, auto zoom adequately covered?

[0018] Gesture commands using one or both hands can be used to control the smart glasses various functions. For example, one hand can be held in front of the camera and views selected to be zoomed in or out of using finger gestures. Using the IR sensors and the dual zoom cameras the smart glasses are able to dynamically detect and adapt to this scenario, so that the zoom camera most proximate to the hand takes on the role of focusing on the hand while the other camera continues to feed video footage of the scene to the head-up display. In addition, the smart glasses are able to mask the user's hand(s) by detecting its position in front of the wearer and intelligently substituting video from the other camera to cover the arc of view obscured by the hand. The two cameras' video feeds can be automatically stitched together to provide a seamless wide angle or panoramic view of the scene without the hand appearing in it. Said video can be recorded or streamed live to the wearer's heads-up display.

[0019] The smart glasses can also incorporate multi-directional audio, as well as the ability to select and amplify audio from a specific target object or direction and to feed and store audio such that it can be associated with its associated zoom video. Both in playback and live streaming modes, the viewer can selectively focus on a particular subject area of the video and listen to the amplified sound recorded from that particular direction.

[0020] The wearer is able to optionally use head motion to activate the video scroll feature. The device is able to differentiate between when the user has turned his or her head and when he has turned his or her entire body to face a new direction. This can be done by detecting the orientation of the user's body or shoulders, using downward-facing cameras or IR sensors, according to some embodiments. To scroll in one direction using this mode, the user turns his or head in that direction and to pause the scrolling, the user turns his or her head back to the forward neutral position. The degree of head turn can control the speed of the scrolling action.

[0021] Audio Input Control

[0022] Gyroscopic Motion Sensors, Accelerometers, GPS

[0023] wifi-mobile communications device, a digital audio driver for sending audio feedback to the wearer's ears, a battery, a computer processor, a power management unit, and a forward-facing cameras and camera driver. Integrated audio-visual logic component or driver, a user interface logic component or driver, a computer processor, a mini SD memory card for external data storage, etc.

[0024] In another embodiment, the smart glasses can incorporate retractable, cameras that are on bendable or omni-directional mounts, enabling the user to custom orient each of these cameras in his or her preferred directions. For example, for certain sports a user may wish to capture video from above and/or below the head's natural orientation. These retractable cameras can also be oriented to provide an improved view behind the head (particularly if the rear camera support is not being used) and also around the wearer's hair.

[0025] In addition, the device has the ability to wirelessly stream surround video and audio or stereoscopic video or selected video feeds from individual cameras to the heads-up-display of another wearer of the one of these devices. Said wireless communication can take place though various communications protocols and systems, including but not limited to WiFi, Bluetooth, mobile phone cellular networks, etc.

[0048] Several embodiments are specifically illustrated and/or described herein. However, it will be appreciated that modifications and variations are covered by the above teachings and within the scope of the appended claims without departing from the spirit and intended scope thereof.

[0049] Technical features described in this application can be used to construct various embodiments of autonomous computing and telecommunications eye glasses.

[0050] According to one aspect, there is provided autonomous computing and multi-channel telecommunications head-up displays glasses with dual POV optical zoom and autofocusing cameras, infra-red light and sensor array and side and rear view multi-directional lens array for 360 degree image and video.

[0051] In one approach, embodiments comprise Smart Eye Glasses with stereo digital cameras, stereo human-audio interface, directional microphones, eye-tracking sensors, and digital projection system (“Smart Glasses”).

[0052] In one approach, purposes of the invention include enabling wearers of the device to use the autonomous computing and telecommunications glasses as primary phone, Internet to communicate wirelessly and to record and view live video feeds of their surroundings in the user's eyeglasses while interacting with these video feeds in a way that enables the wearer to be able to dynamically perform a variety of functions such as to automatically zoom in on objects of interest which are in the user's field of view or visible from one of the camera feeds and to communicate with said device using a variety of inputs such as eye movements, hand movements and finger gestures as well as voice commands and button selections.

[0053] According to some embodiments, the smart glasses can incorporate dual forward-facing cameras, enabling the camera to record wide angle and stereographic video or still image content. In addition, the smart glasses can incorporate side-facing and rear-facing cameras such that it is able to record a 360 degree surround video feed.

[0054] According to some embodiments, the cameras can incorporate electro magnetic, electro-static or electromechnical-mechanical optical zoom capabilities as well as digital zoom capabilities. With infrared enabled active auto-zoom and autofocusing system. The IR sensor enables low light and night vision. These cameras can also incorporate directional electro-mechanical gimbal mounts, allowing them to pitch and yaw for greater directional control, according to some embodiments.

[0055] The eyeglasses can be selectively transparent or partially transparent, or opaque, according to some embodiments. In addition to a heads-up OLED display or Pico-Project system

[0056] Heads-up display data can be projected into the eyeglasses such that it is superimposed onto the real-world view that the wearer sees through the glasses or it can replace the real-world view if the glasses opacity is increased.

[0057] Dual inward-facing eye motion tracking sensors are integrated into the frame of the eyeglasses above the left and right eyes. These give the smart glasses the ability to track the wearer's eye movements as well as determine the wearer's distant point of focus. Eye tracking can be used to control a wide variety of device functions including the cameras' zoom functions and the integrated telecommunications functions. For example, the user can direct the device to scroll a surround video feed left or right, simply by looking left or right. Alternatively, in another mode, the edges and corners of the user's heads-up display glasses can be logically mapped to represent a bird's eye view of the surrounding scene. To instruct the smart glasses to display the video feed from the rear-facing cameras, the user can briefly look down, to display video from the right, the user looks to the right of the display and to display video from the front, the user looks up.

[0058] Gesture commands using one or both hands can be used to control the smart glasses various functions. For example, one hand can be held in front of the camera and views selected to be zoomed in or out of using finger gestures. Using the IR sensors and the dual zoom cameras the smart glasses are able to dynamically detect and adapt to this scenario, so that the zoom camera most proximate to the hand takes on the role of focusing on the hand while the other camera continues to feed video footage of the scene to the head-up display. In addition, the smart glasses are able to mask the user's hand(s) by detecting its position in front of the wearer and intelligently substituting video from the other camera to cover the arc of view obscured by the hand. The two cameras' video feeds can be automatically stitched together to provide a seamless wide angle or panoramic view of the scene without the hand appearing in it. Said video can be recorded or streamed live to the wearer's heads-up display.

[0059] The smart glasses can also incorporate multi-directional audio, as well as the ability to select and amplify audio from a specific target object or direction and to feed and store audio such that it can be associated with its associated zoom video. Both in playback and live streaming modes, the viewer can selectively focus on a particular subject area of the video and listen to the amplified sound recorded from that particular direction.

[0060] The wearer is able to optionally use head motion to activate the video scroll feature. The device is able to differentiate between when the user has turned his or her head and when he has turned his or her entire body to face a new direction. This can be done by detecting the orientation of the user's body or shoulders, using downward-facing cameras or IR sensors, according to some embodiments. To scroll in one direction using this mode, the user turns his or head in that direction and to pause the scrolling, the user turns his or her head back to the forward neutral position. The degree of head turn can control the speed of the scrolling action.

[0061] In some embodiments, the smart glasses have audio input control, gyroscopic motion sensors, accelerometers, GPS wifi-mobile communications device, a digital audio driver for sending audio feedback to the wearer's ears, a battery, a computer processor, a power management unit, and a forward-facing cameras and camera driver. Integrated audio-visual logic component or driver, a user interface logic component or driver, a computer processor, a mini SD memory card for external data storage, etc.

[0062] In another embodiment, the smart glasses can incorporate retractable, cameras that are on bendable or omni-directional mounts, enabling the user to custom orient each of these cameras in his or her preferred directions. For example, for certain sports a user may wish to capture video from above and/or below the head's natural orientation. These retractable cameras can also be oriented to provide an improved view behind the head (particularly if the rear camera support is not being used) and also around the wearer's hair.

[0063] In addition, the device has the ability to wirelessly stream surround video and audio or stereoscopic video or selected video feeds from individual cameras to the heads-up-display of another wearer of the one of these devices. Said wireless communication can take place though various communications protocols and systems, including but not limited to WiFi, Bluetooth, mobile phone cellular networks, etc.

[0064] Reference will now be made to the drawings in which the various elements of embodiments will be given numerical designations and in which embodiments will be discussed so as to enable one skilled in the art to make and use the invention.

[0065] Specific reference to components, process steps, and other elements are not intended to be limiting. Further, it is understood that like parts bear the same reference numerals, when referring to alternate figures. It will be further noted that the figures are schematic and provided for guidance to the skilled reader and are not necessarily drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the figures may be purposely distorted to make certain features or relationships easier to understand.

[0066] Reference will now be made to FIGS. 1 through 2C which illustrate a pair of smart glasses according to some embodiments. A pair of smart glasses 100 are shown on a human head 800. The smart glasses integrate dual forward-facing digital cameras 105 as well as digital optical cameras which face to the sides and the rear of the wearer. The smart glasses can incorporate digital displays 102 in the eyeglasses, which, according to some embodiments are able to function as head-up displays, allowing to the wearer to see through said eyeglasses and or focus on digital content which can be displayed in the eyeglasses.

[0067] In addition there are detachable stereo wireless earphones 112, which can be stored in or on the glasses, according to some embodiments and inserted in the user's ears when needed. Said wireless earphones incorporate a power source (battery) and wireless communications electronics, such as a Bluetooth transceiver, which is able to communicate directly with similar communications electronics in the smart glasses, thereby allowing the smart glasses to transmit audio to the wearer's ears. In addition to the optical cameras, the front of each side of the smart glasses can incorporate an infra-red sensor and infra-red camera, according to some embodiments. The smart glasses also integrate eye tracking sensors 103 above each eye, so that the user is able to communicate with the smart glasses using eye movements and the smart glasses are able to continually monitor eye movement to determine the wearer's visual focus and point of interest and accordingly adapt its video feeds to on-board digital memory and to the wearer's display, according to some embodiments. Dual multi-directional microphones in each arm 104 are able to track and record sound in three dimensions. The ear phones 112 are able to reproduce three-dimensional surround sound so that the wearer is able to sense directionality related to video feeds that are projected in the eyeglass head-up-displays. Internal rechargeable batteries provide power to all of the integrated electronic components. The precise arrangement of the described electronic components can vary.

[0068] The smart glasses can also incorporate interface ports for charging the internal batteries as well as data download and upload. In addition, the arms 104 of the smart glasses can incorporate ports or slots 114 for the insertion of memory cards, such as SD cards. These interface ports and memory card slots 114 are depicted as integrated underneath each of the arms, according to some embodiments. Internal electronics and contacts integrated into the hinge areas between the arms and the eyeglass frame 101 provide for data communication and power feeds between the arms and the eyeglass frame.

[0069] Referring now to FIGS. 3A and 3B, which illustrate smart glasses including detachable wireless earphones according to some embodiments, the figures illustrate how the wireless earphones 112 can be detached by the wearer from their docking-charging stations in the smart eyeglasses and then worn in the ears.

[0070] FIGS. 4A through 4B illustrate smart glasses including attachable external batteries according to some embodiments. As shown in FIG. 4A through C, attachable external batteries 200 can attach to the arms 104 of the smart glasses, according to some embodiments. Said attachable external batteries can also support and power the detachable wireless earphones 112, according to some embodiments. These batteries 200 can also provide a electronic communications pass-through to the ear phones 112 so that transmission of audio data to them from the smart glasses 100 can be wired rather than wireless. Said external batteries 200 can use the same docking port that is used for storage and charging of the wireless earphones 112 to connect to the arms 104 of the smart glasses 100, according to some embodiments. The smart glasses 100 are able to intelligently and dynamically switch power sources, so that it will always deplete the external batteries 200 before depleting its internal batteries. In addition, the smart glasses 100 will automatically recharge the internal batteries 200 and the earphone batteries from the external batteries whenever they are connected.

[0071] FIGS. 5A and B through 6A and B illustrate an attachable external rear camera support for the smart glasses according to some embodiments. As depicted in FIGS. 5A and B through 6A and B, the attachable external rear camera support 300 wraps behind the neck and provides additional support for the smart glasses 100 and to provide rear video feeds from its integrated stereo cameras. Said rear camera support allows the user to secure the smart glasses to his or her head for active use such as during sports and to also record full 360 degree panoramic video. Said rear camera support 300 can attach to the external batteries 200, as depicted, according to some embodiments. In other embodiments not depicted here, the rear camera support 300 can attach directly to the smart glasses 100 and can optionally incorporate its own internal battery. Said rear camera support 300 can be adjustable to allow for people of different size and shape to customize the fit. It can adjust out enough to wrap around hair of varying volume. In the depicted embodiments, sheaths 302 allow for the rear camera support 300 to be extended in its width, as well as to be extended out and down.

[0072] FIGS. 7A and B depict a pair of smart glasses according to some embodiments

[0073] As depicted in FIGS. 7A and B, a pair of smart glasses 100 in which the arms of the devices integrate extendable, cameras 400 that are on multi-directional bendable supports 401, according to some embodiments. In the depicted embodiment, these extendable cameras 400 are able to be stored inside recessed tracks 403 in the arms of the smart glasses 100 when not in use and can be swiveled out and bent into the desired orientation as desired.

[0074] FIGS. 8A and 8B depict simplified embodiments of the smart glasses according to some embodiments. FIGS. 8A and 8B emphasize the head-up display eyeglasses 100 and the optical motion sensors 105 as well as the integrated earphones 112, according to some embodiments. In this particular embodiment, the earphones are connected to the arms of the smart glasses. These may also be separable and/or wired or wireless, as depicted in earlier figures.

[0075] FIGS. 9A and 9B illustrate right and left side views respectively of the smart glasses according to some embodiments. Internal electronic components integrated into the arms of the glasses are symbolized. The precise arrangement of said electronic components can vary. In the depicted embodiment in FIG. 9A, the right eyeglass arm can be seen with a wifi-mobile communications device 126, a digital audio driver 124 for sending audio feedback to the wearer's ears, a battery 128, a computer processor 130, a power management unit 122, and a forward-facing camera 105 and camera driver 120. In FIG. 9B, the left eyeglasses arm can be seen with integrated audio-visual logic component or driver 134, a user interface logic component or driver 132, a computer processor 130, a mini SD memory card 136 for external data storage, and a camera 105 and camera driver 120.

[0076] FIG. 10A depicts a frontal perspective view of the heads-up display glasses according to some embodiments. In FIG. 10A, the glasses 100 are shown in their fully transparent state, allowing the wearer an unobstructed view of the real world.

[0077] FIG. 10B depicts a frontal perspective view of the heads-up display eyeglasses according to some embodiments. In FIG. 10B, the glasses are shown in a translucent or partially opaque state. The heads-up display glasses are able to display video or computer-generated content such that it is translucent, providing the user with the ability to monitor the real world in front of him or her. In addition, the opacity of the lenses 102 can be dynamically altered based on ambient lighting conditions or to filter direct sunlight.

[0078] FIG. 10C depicts a frontal perspective view of the heads-up display eyeglasses according to some embodiments. In FIG. 10C, the glasses are in an opaque or mostly opaque state, for optimal viewing of heads-up display imagery.

[0079] FIGS. 11A through 11D depicts the eye tracking sensors with and without the smart glasses according to some embodiments. FIGS. 11A to 11D show how eye tracking sensor 103 can be used to enable the user to give instructions or cues to the smart glasses' internal computer processor so that it can act on said cues and accordingly perform pre-programmed functions. In the depictions, using the eye-tracking sensors 103, the smart glasses are able to track where on the screen the eyes 801,802 are focused both on the vertical and horizontal axes. In this way it is able to, for example, detect when the wearer has read toward the bottom of a page and automatically scroll the content up. Likewise, the user is able to scroll the page down by looking toward the top of the screen. For wide images, the user can similarly look toward the left or right of the screen and the display content will automatically scroll across to the right or left, respectively.

[0080] FIGS. 12A and 12 B depict wireless communication between the smart glasses and a head-mounted camera or camera-phone according to some embodiments. The depicted head-mounted camera phone 500 is able to dock next to the eyeglasses 100. In this case, the smart glasses 100 would not require an earphone 112 since one is provided by the camera-phone 500. The camera-phone is able to wirelessly transmit its video feed to the smart glasses, which in turn can display the video content in the head-up display glasses, according to some embodiments. The wireless communication 501 can be two-way, so that the smart glasses 100 are able to transmit instructions to the camera-phone. 500 For example, tracking of eye movements can be used to instruct the camera-phone as what target object to zoom in and focus on.

[0081] FIGS. 13A and 13B illustrate the smart glasses' ability to detect and track hand movements and gesture patterns according to some embodiments. The user is able to control the smart glasses functions by detecting and tracking the hand movements 804 and gesture patterns. In this depiction, the user has used a reverse-pinching gesture to isolate the clock 701 on a distant clock tower and zoom in on this target subject. The dual zoom cameras automatically zoom in on the clock and the heads-up display glasses display live video footage of the zoomed-in clock 701 so that the user can determine the time of day.

[0082] FIG. 14 depicts the ability of the smart glasses to wirelessly communicate with telecommunications networks using WiFi 503, mobile cellular connections 502, satellites and global positioning systems 501 according to some embodiments.

[0083] FIGS. 15A and 15B depict the ability of the smart glasses to measure distances 601,602,604 according to some embodiments. The smart glasses 100 are configured to measure remote objects using cameras and infra-red sensors and also measure the distance 604 between the head-up display glasses and the wearer's eyes.

[0084] FIGS. 16A and 16B depict the ability of the smart glasses to map the area 605 of a hand 804 and assign sectors to it according to some embodiments;

[0085] FIGS. 17A and 17B depict the ability of the smart glasses 100 to project a virtual keypad 606 onto a representation of the user's hand 804 in the head-up display according to some embodiments. The user is able to then “click” on the “buttons” in this virtual keypad 606 in real space by touching the surface of the hand with a finger of the other hand or by sequentially focusing his or her eyes and momentarily pausing on each of the virtual buttons to be selected. The eye-tracking sensors can track eye movement and the forward-facing IR sensors and cameras can track hand movements for this purpose. The smart glasses do not need to project an actual image onto the hand in front of them, but this is also an option according to some embodiments.

[0086] FIGS. 18A and 18B depict the ability of the smart glasses to track the movement of the user's hand according to some embodiments. The smart glasses 100 are configured to track the movement and dynamically update the position of the virtual keypad representation 606 of the user's hand 804 in the heads-up display.

[0087] FIGS. 19A and 19B depict the user using his or her free hand to dial a phone number on a virtual keypad according to some embodiments. The smart phones are configured to generate and project the virtual keypad 606 onto a representation of the user's hand in the head-up display. The heads-up display glasses then display the selected numbers as they are being called.

[0088] FIGS. 20A and 20B depict the ability of the smart glasses to project a virtual keyboard onto a representation of a surface according to some embodiments. The smart glasses 100 are configured to project 604 the representation 608 on surfaces 710 such as a desk or wall in front of the user. The smart glasses 100 are able to correctly adjust the virtual keyboard 608 to the distance and angle of the surface. The user's hands 804 can either been seen through the translucent heads-up display as they type, or the head's up display can project a video feed of the hands.

[0089] FIG. 21A through 21C depict the ability of the smart glasses to detect hand movements and gestures according to some embodiments. The smart glasses 100 are configured to detect movements and gestures so that the user can interact with virtual content that is displayed in the heads-up displays. In this depiction, the user is able to spin a representation of the earth 609 by turning his hand or by placing his hand 804 in position on the virtual globe as he sees it and spinning the globe.

[0090] FIG. 22A through 22C depict the ability of the smart glasses to add virtual representations of furniture to the head-up display according to some embodiments. The smart glasses 100 are configured to add the representations such that the furniture 722 looks like it is in the a real environment 720 in front of the glasses. The user can use this to “try out” different furniture styles, colors and positions in a room to see how they will look. Virtual imagery of said furniture can be selected and downloaded by the user by using the head-up display glasses to browse through an Internet-based database of photographs or 3D models.

[0091] FIGS. 23A through 23C and 24A through 24B show further examples of the smart glasses in use.

[0092] The embodiments discussed herein are illustrative of the present invention. As these embodiments of the present invention are described with reference to illustrations, various modifications or adaptations of the methods and or specific structures described may become apparent to those skilled in the art. All such modifications, adaptations, or variations that rely upon the teachings of the present invention, and through which these teachings have advanced the art, are considered to be within the spirit and scope of the present invention. Hence, these descriptions and drawings should not be considered in a limiting sense, as it is understood that the present invention is in no way limited to only the embodiments illustrated.