Three dimensional audio vision

[0002] 1. Field of the Invention

[0003] The present invention pertains to the field of vision enhancement. More particularly, this invention relates to the art of providing an optical vision substitute.

[0004] 2. Background

[0005] Eyesight is, for many people, the most important of all the senses. Unfortunately, not everyone enjoys perfect vision. Many visually impaired people have developed their other senses to reduce their reliance on optical vision. For instance, the visually impaired can learn to use a cane to detect objects in one's immediate vicinity. Braille provides a means by which visually impaired people can read text. Hearing can be developed to recognize the flow and direction of traffic at an intersection. Seeing eye dogs can be trained to provide excellent assistance.

[0006] Technology has sought to provide additional alternatives for the visually impaired. Corrective lenses can improve visual acuity for those with at least some degree of optical sensory perception. Surgery can often correct retinal or nerve damage, and remove cataracts. Sonar devices have also been used to provide the visually impaired with an audio warning signal when an object over a specified size is encountered within a specified distance.

[0007] A need remains, however, for an apparatus to provide an audio representation of one's surroundings.

[0008] In accordance with the teachings of the present invention, a method and apparatus to create an audio representation of a three dimensional environment is provided. One embodiment includes a plurality of video receptors, a plurality of audio output devices, and a converter. The converter receives multidimensional video data from the plurality of video receptors, converts the multidimensional video data into a multidimensional audio representation of the multidimensional video data, and outputs the multidimensional audio representation to the plurality of audio output devices.

[0009] Examples of the present invention are illustrated in the accompanying drawings. The accompanying drawings, however, do not limit the scope of the present invention in any way. Like references in the drawings indicate similar elements.

[0010] FIG. 1A is a block diagram illustrating one embodiment of the present invention;

[0011] FIG. 1B illustrates one embodiment of the present invention employed with a headset;

[0012] FIG. 2 is a flow chart illustrating the method of one embodiment of the present invention;

[0013] FIG. 3A is a block diagram illustrating one embodiment of video to audio landscaping;

[0014] FIG. 3B is a block diagram illustrating one embodiment of image recognition to audio recognition;

[0015] FIG. 4 is a block diagram of one embodiment of a hardware system suitable for use with the present invention.

[0016] In the following detailed description, exemplary embodiments are presented in connection with the figures and numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details, that the present invention is not limited to the depicted embodiments, and that the present invention may be practiced in a variety of alternate embodiments. Accordingly, the innovative features of the present invention may be practiced in a system of greater or lesser complexity than that of the system depicted in the figures. In other instances well known methods, procedures, components, and circuits have not been described in detail.

[0017] FIG. 1A is a block diagram of one embodiment of the present invention. Video receptors 105aand 105breceive light input and provide multidimensional video data to input ports A and B of converter 110. Converter 110 receives the multidimensional video data, converts it to a multidimensional audio representation, and provides the multidimensional audio representation to audio output devices 115aand 115bfrom output ports C and D. Audio output devices 115aand 115boutput the multidimensional audio representation.

[0018] FIG. 1B is an illustration of one embodiment of the present invention employed using a headset 120. Headset 120 is not a necessary element, and any number of other configures could be used to practice the present invention. In FIG. 1B, headset 120 is operative to fit over the head of a user so that audio output devices 115aand 115bare close enough to the user's ears so that the user can hear audio signals produced by audio output devices 115aand 115b. Audio output devices 115aand 115bcan be ear inserts that fit into the ear canal, or earphones that rest on the outside of the ear. Video receptors 105aand 105bcan be small video cameras affixed to headset 120 so that, when the headset is worn, video receptors 105aand 105bare on either side of the user's head, and receive light from the general direction in which the head is pointed. In alternate embodiments, three or more video receptors could be employed. With additional video receptors, the composite field of view for all of the video receptors together could provide a 360 degree perspective.

[0019] Converter 110 can be affixed to headset 120 as shown, or converter 110 can be located elsewhere, such as in a pocket, clipped to a belt, or located remotely. Wires can be used to couple converter 110 to the video receptors and audio output devices, or wireless communications can be used such as infra-red and radio frequency communications.

[0020] FIG. 2 is a flow chart illustrating the process of the present invention. Sensors 105aand 105bcontinually provide multidimensional video data in block 210. Converter 110 converts the multidimensional video data into multidimensional audio representations in block 220. The audio representations are provided by audio output devices 115aand 115bin block 230. The process is continually repeated, providing a real time audio representation of the surroundings.

[0021] When in use, video receptors 105aand 105beach provide a video image of the area in the direction the head is pointed. Converter 110 compiles and analyzes the two video images. As shown in FIG. 3A, video landscaping generator 310 generates a video landscape. The video landscape is provided to audio landscape generator 320 to generate an audio landscape based on the video landscape.

[0022] The video landscape comprises a body of data representing objects and distances to the objects with relation to video receptors 105aand 105bin three dimensional space. The invention can be calibrated initially, or on a continuing basis, to determine the distance between the cameras, and the relation of the cameras to the ground. For instance, video receptors 105aand 105bcould be equipped with inclination sensors (not shown). Converter 110 could calculate the angle of the video receptors with relation to an identified point on the ground using the angle of inclination from the inclination sensors and the angle of the identified point off the center of the field of view. Then converter 110 could calculate how high the video receptors are off the ground based on that angle and the distance to the identified point on the ground. The distance to the identified point, as with any object in the field of view, can be measured based on the two perspectives of the video receptors. Then, distances to objects and the relation of the objects to the video receptors can be calculated based on the distance between the two perspectives of the video receptors, the inclination of the video receptors, the position of the object in the field of view, and the height of the video receptors off the ground.

[0023] The distances and positions are converted into audio representations with differentiating frequencies and volumes for different objects at different distances. As the user turns his or her head from side to side, tilts his or her head up and down, and moves about a landscape, the audio representations change according to the video landscape.

[0024] Since the receptors are video receptors, converter 110 can also perform image recognition, as shown in FIG. 3B. A library of shapes and objects can be created, updated, and stored in image recognition element 330. The library could even be dynamically updated, adding new items to the library as they are encountered. The recognized images can then be mapped to specific audio signals in audio mapping element 340. Audio signals could be quickly recognizable tones for commonly encountered objects, or verbal descriptions of new or rare objects. In this way, tables, chairs, doors, and many other objects could be identified by the sound of the audio representation.

[0025] Image recognition, in connection with the video landscaping, could be used to identify the size and shape of an object. For instance, once a user becomes proficient with the device, the identity, dimensions, and location of a door, crosswalk, table top, or person could be ascertained from the audio representation of each. As a user walks toward a doorway, for instance, the user can “hear” that a door is just ahead. As the user gets closer, the height, width, and direction of the door relative to the video receptors are continually updated so the user can make course corrections to keep on path for the doorway. Converter 110 could be calibrated to provide several inches of clearance above the height of the video receptors and to either side to account for the user's head and body. If the user is too tall to walk upright through the doorway, converter 110 could provide a warning signal to the user to duck his head. Other warnings could be provided to avoid various other dangers. For instance, fast moving objects on a collision course with the user could be recognized and an audio signal could warn the user to duck or dodge to one side.

[0026] Text recognition could also be incorporated into the invention, allowing the user to hear audio representations of the text. In this way, a user could “hear” street signs, newspaper articles, or even the words on a computer screen. The converter could also include language translation, which would make the invention useful even for people with perfect eyesight when, for instance, traveling in a foreign country.

[0027] In alternate embodiments, the present invention could be employed on the frames of glasses. For instance, the video receptors could be affixed to the arms of the frames, pointing forward, and the audio output devices could be small ear inserts that fit in the ear canal. The converter could be located remotely, carried in the user's pocket, or incorporated into the frames. In other embodiments, the present invention could be incorporated in jewelry, decorative hair pins, or any number of inconspicuous and aesthetic settings.

[0028] Except for the teachings of the present invention, converter 110 may be represented by a broad category of computer systems known in the art. An example of such a computer system is a computer system equipped with a high performance microprocessor(s), such as the Pentium® processor, Pentium® Pro processor, or Pentium® II processor manufactured by and commonly available from Intel Corporation of Santa Clara, Calif., or the Alpha® processor manufactured by Digital Equipment Corporation of Manard, Mass.

[0029] It is to be appreciated that the housing size and design for converter 110 may be altered, allowing it to be incorporated into a headset, glasses frame, a piece of jewelry, or a pocket sized package. Alternately, in the case of the wireless communications connections between converter 110 and video receptors 105aand 105b,and between converter 110 and audio output device 115aand 115b,converter 110 could be located centrally, for instance, within the house or office. A separate, rechargeable portable converter could be used for travel outside the range of the centrally located converter. A network of converters or transmission stations could expand the coverage area. The centrally located converter could be incorporated into a standard desktop computer, for instance, reducing the amount of hardware the user must carry.

[0030] Such computer systems are commonly equipped with a number of audio and video input and output peripherals and interfaces, which are known in the art, for receiving, digitizing, and compressing audio and video signals. FIG. 4 illustrates one embodiment of a hardware system suitable for use with converter 110 of FIG. 1. In the illustrated embodiment, the hardware system includes processor 402 and cache memory 404 coupled to each other as shown. Additionally, the hardware system includes high performance input/output (I/O) bus 406 and standard I/O bus 408. Host bridge 410 couples processor 402 to high performance I/O bus 406, whereas I/O bus bridge 412 couples the two buses 406 and 408 to each other. System memo 414 is coupled to bus 406. Mass storage 420 is coupled to bus 408. Collectively, these elements are intended to represent a broad category of hardware systems, including but not limited to general purpose computer systems based on the Pentium® processor, Pentium® Pro processor, or Pentium® II processor, manufactured by Intel Corporation of Santa Clara, Calif.

[0031] In one embodiment, various electronic devices are also coupled to high performance I/O bus 406. As illustrated, video input device 430 and audio outputs 432 are also coupled to high performance I/O bus 406. These elements 402-432 perform their conventional functions known in the art.

[0032] Mass storage 420 is used to provide permanent storage for the data and programming instructions to implement the above described functions, whereas system memory 414 is used to provide temporary storage for the data and programming instructions when executed by processor 402.

[0033] It is to be appreciated that various components of the hardware system may be rearranged. For example, cache 404 may be on-chip with processor 402. Alternatively, cache 404 and processor 402 may be packed together as a “processor module”, with processor 402 being referred to as the “processor core”. Furthermore, certain implementations of the present invention may not require nor include all of the above components. For example, mass storage 420 may not be included in the system. Additionally, mass storage 420 shown coupled to standard I/O bus 408 may be coupled to high performance I/O bus 406; in addition, in some implementations only a single bus may exist with the components of the hardware system being coupled to the single bus. Furthermore, additional components may be included in the hardware system, such as additional processors, storage devices, or memories.

[0034] In one embodiment, converter 110 as discussed above is implemented as a series of software routines run by the hardware system of FIG. 4. These software routines comprise a plurality or series of instructions to be executed by a processor in a hardware system, such as processor 402 of FIG. 4. Initially, the series of instructions are stored on a storage device, such as mass storage 420. It is to be appreciated that the series of instructions can be stored using any conventional storage medium, such as a diskette, CD-ROM, magnetic tape, digital video or versatile disk (DVD), laser disk, ROM, Flash memory, etc. It is also to be appreciated that the series of instructions need not be stored locally, and could be received from a remote storage device, such as a server on a network. The instructions are copied from the storage device, such as mass storage 420, into memory 414 and then accessed and executed by processor 402. In one implementation, these software routines are written in the C++ programming language. It is to be appreciated, however, that these routines may be implemented in any of a wide variety of programming languages.

[0035] In alternate embodiments, the present invention is implemented in discrete hardware or firmware. For example, one or more application specific integrated circuits (ASICs) could be programmed with the above described functions of the present invention. By way of another example, converter 110 of FIG. 1 could be implemented in one or more ASICs of an additional circuit board for insertion into the hardware system of FIG. 4.

[0036] Thus, a method and apparatus for providing an audio representation of a three dimensional environment is described. Whereas many alterations and modifications of the present invention will be comprehended by a person skilled in the art after having read the foregoing description, it is to be understood that the particular embodiments shown and described by way of illustration are in no way intended to be considered limiting. Therefore, references to details of particular embodiments are not intended to limit the scope of the claims.