Inertial sensors integration

[0001] This application is related to U.S. Ser. No. 10/284,417, entitled Universal Computing Device, filed Oct. 31, 2002, invented by Jian Wang and Chui Hui Zhang, whose contents are hereby expressly incorporated by reference.

[0002] 1. Technical Field

[0003] Aspects of the present invention relate to an apparatus and method for generating data inputs including electronic ink. More particularly, aspects of the present invention relate to an input device including inertial sensors to accurately detect movement of the input device.

[0004] 2. Related Art

[0005] Typical computers systems, especially computer systems using graphical user interface (GUI) systems, such as Microsoft Windows, are optimized for accepting user input from one or more discrete input devices for entering text (such as a keyboard), and a pointing device (such as a mouse) with one or more buttons for activating user selections.

[0006] Some computing systems have expanded the input systems available to a user by providing a pen like stylus. The stylus may take the place of both the keyboard (for data entry) as well as the mouse (for control). Some computing systems preserve electronic ink in handwritten form. Other systems receive handwritten electronic information, or electronic ink, and immediately convert the electronic ink into text.

[0007] One shortcoming associated with the use of a stylus and sensor board system, is that use of the stylus is tied to the device incorporating the sensor board. In other words, the stylus may be used to generate inputs only when used in conjunction with the required sensor board. Additionally, accuracy of movement detection is affected by the proximity of the stylus to the sensing board. If the stylus is not positioned in contact with or within a very short distance from the screen and/or sensor board, the user's inputs may not be detected accurately. Further, intentional movements by a user may not be accurately captured using only a digitizer. One device that include conventional input devices includes Novel Device for Inputting Handwriting Trajectory, Ricoh Technical Report No. 27, 52 NOVEMBER, 2001, whose contents are located at http://www.ricoh.co.jp/rdc/techreport/No27/Ronbun/A2707.pdf.

[0008] There is a need in the art for an input device that can be used in combination with or independent of a host device to which input data is transferred.

[0009] Aspects of the present invention are accomplished using an input device including additional sensors for accurately capturing movement of the input device. Further aspects of the present invention permit the use of an input device in combination with any surface that will generate data based on movement of the input device. Aspects of the present invention provide an input device that may be used in combination with, or independent of, the device for which the data is intended. Aspects of the invention further involve improved movement sensing and analysis. The input device may be formed in the shape of a pen, and may include an ink cartridge for use of the input device in a manner familiar to users.

[0010] These and other aspects of the invention are addressed in relation to the figures and related descriptions.

[0011] The foregoing summary of aspects of the invention, as well as the following detailed description of various embodiments, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention.

[0012] FIG. 1 shows a schematic diagram of a general-purpose digital computing environment that can be used to implement various aspects of the invention.

[0013] FIG. 2 illustrates an input device for generating user inputs in accordance with an illustrative embodiment of the present invention.

[0014] FIG. 3 provides an illustration of the type of data generated by inertial sensors in accordance with aspects of the present invention.

[0015] FIG. 4 is a flowchart depicting a method for measuring and processing data representing movement of the input device in accordance with aspects of the present invention.

[0016] FIG. 5 illustrates an input device for enter user inputs in accordance with an additional illustrative embodiment of the present invention.

[0017] FIG. 6 shows a paper coordinate system and an inertial coordinate system in accordance with embodiments of the present invention.

[0018] FIG. 7 shows a pen with various dimensions in accordance with embodiments of the present invention.

[0019] FIG. 8 shows the pen of FIG. 7 with relation to a tip of the pen in accordance with embodiments of the present invention.

[0020] FIGS. 9-11 show various graphs of acceleration, velocity, and displacement in accordance with embodiments of the present invention.

[0021] Aspects of the present invention provide improved motion sensing using one or more inertial sensors. The input device may be equipped with sensors for detecting movement of the input device by measuring such indicators as velocity, acceleration, changes in electromagnetic fields or other such detected indicators. Further aspects of the invention involve use of a memory incorporated within the input device for storing the sensed movement data for transmission to the intended device. Processing of the sensed movement data to generate images representative of hand written strokes may also be performed.

[0022] Terms

[0023] Input Devicea device for entering information, which may be configured for generating and processing information.

[0024] Penany writing implement that may or may not include the ability to store ink. In some examples a stylus with no ink capability may be used as a pen in accordance with embodiments of the present invention.

[0025] Inertial Sensorany device or combination of devices that can be used to measure the movement of the input device in a variety of ways, including, but not limited to laterally, at the pen tip, or by a changing the orientation of the input device.

[0026] Accelerometera device or combination of devices that produce indications of the acceleration of the input device.

[0027] Gyroscopea device or combination of devices that produce indications of the angular velocity of the input device.

[0028] Magnitometera device or combination of devices that senses changes in a magnetic field.

[0029] Cameraan image capture system.

[0030] Inductive elementa device incorporated within the input device for use in combination with a sensor board or the like in which the sensor board detects movement and/or positioning of the input device based on changes in an electromagnetic field caused by the inductive or resistive element.

[0031] Pressure Sensora device for detecting force such as the force exerted against the surface over which the input device is positioned.

[0032] Electronic InkA sequence or set of strokes with properties, a stroke being a sequence or set of captured points. A sequence of strokes may include strokes in an ordered form. The sequence may be ordered by the time captured or by where the strokes appear on a page. Other orders are possible. A set of strokes may include sequences of strokes or unordered strokes or any combination thereof. Ink may be expanded to include additional properties, methods, and trigger events and the like. When combined with at least some of these events, it may be referred to as an ink object.

[0033] Inertial Coordinatesa local geographic coordinate which has its origin at the location where the input device is used, such as the left top of the paper, and axes aligned with the directions of north, east and the local vertical (down).

[0034] Paper Coordinatesthe coordinates of the surface over which the input device is moved, whether it is paper or some other surface.

[0035] Psuedo Standing Pointan estimation of the position at which, where the user reverses direction of the device, the velocity of the input device reduces to zero. This may also be interpreted as an approximate point on/in the device, typically selected to be located below the middle position of the input device as shown in FIG. 7. When the input device is used to write, this point moves less than other points. When the input device does not move, the output of the accelerometers contains only orientation information (information that may be used to determine the orientation of the input device in space and/or in relation to a surface). When the input device moves, the output value of accelerometers include the acceleration information (information that may be used to determine the acceleration) and the orientation information of the device. Both sets of information may be combined or related to complement each other. It is assumed that in most cases when the device is used to write, the major part of the acceleration of this point is orientation information, and the minor part of it is acceleration information. The minor part is small compared with the acceleration of gravity. The acceleration information may be used to compensate for the drift error of the gyroscope (as information in the measurement equations).

[0036] Host Computing Devicea desktop computer, a laptop computer, Tablet PC, a personal data assistant, a telephone, or any device which is configured to receive information from one or more input devices.

[0037] General Purpose Operating Environment

[0038] FIG. 1 illustrates a schematic diagram of an illustrative conventional general-purpose digital computing environment that can be used to implement various aspects of the present invention. In FIG. 1, a computer 100 includes a processing unit 110, a system memory 120, and a system bus 130 that couples various system components including the system memory to the processing unit 110. The system bus 130 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory 120 includes read only memory (ROM) 140 and random access memory (RAM) 150.

[0039] A basic input/output system 160 (BIOS), containing the basic routines that help to transfer information between elements within the computer 100, such as during start-up, is stored in the ROM 140. The computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to a removable optical disk 192 such as a CD ROM or other optical media. The hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are connected to the system bus 130 by a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive interface 194, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the personal computer 100. It will be appreciated by those skilled in the art that other types of computer readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the example operating environment.

[0040] A number of program modules can be stored on the hard disk drive 170, magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or more application programs 196, other program modules 197, and program data 198. A user can enter commands and information into the computer 100 through input devices such as a keyboard 101 and pointing device 102. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner or the like. These and other input devices are often connected to the processing unit 110 through a serial port interface 106 that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB). Further still, these devices may be coupled directly to the system bus 130 via an appropriate interface (not shown). A monitor 107 or other type of display device is also connected to the system bus 130 via an interface, such as a video adapter 108. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. In a preferred embodiment, a pen digitizer 165 and accompanying pen or stylus 166 are provided in order to digitally capture freehand input. Although a direct connection between the pen digitizer 165 and the serial port interface 106 is shown, in practice, the pen digitizer 165 may be coupled to the processing unit 110 directly, parallel port or other interface and the system bus 130 as known in the art. Furthermore, although the digitizer 165 is shown apart from the monitor 107, the usable input area of the digitizer 165 may be co-extensive with the display area of the monitor 107. Further still, the digitizer 165 may be integrated in the monitor 107, or may exist as a separate device overlaying or otherwise appended to the monitor 107.

[0041] The computer 100 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 109. The remote computer 109 can be a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 100, although only a memory storage device 111 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0042] When used in a LAN networking environment, the computer 100 is connected to the local network 112 through a network interface or adapter 114. When used in a WAN networking environment, the personal computer 100 typically includes a modem 115 or other means for establishing a communications over the wide area network 113, such as the Internet. The modem 115, which may be internal or external, is connected to the system bus 130 via the serial port interface 106. In a networked environment, program modules depicted relative to the personal computer 100, or portions thereof, may be stored in the remote memory storage device.

[0043] It will be appreciated that the network connections shown are illustrative and other techniques for establishing a communications link between the computers can be used. The existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like is presumed, and the system can be operated in a client-server configuration to permit a user to retrieve web pages from a web-based server. Any of various conventional web browsers can be used to display and manipulate data on web pages.

[0044] Inertial Sensor Input Device

[0045] FIG. 2 illustrates an illustrative embodiment of an input device for use in accordance with various aspects of the invention. In FIG. 2, input device 201 may include one or more of the following: ink cartridge 202, pressure sensor 203, accelerometer 204, magnetic sensor 205, gyroscope 206, processor 207, memory 208, transceiver 209, accelerometer 210, power supply 211, docking station adapter 212, cap 213, inductive element 215, and a display, not shown. The various components may be electrically coupled as necessary using, for example, a bus, not shown, or wirelessly. Using one or more of the sensors may enhance the accuracy of the determination of the location of the input device.

[0046] Ink cartridge 202 may be included to enable use of the input device in a manner typical of a standard writing instrument such as a pen. The ink cartridge 202 may provide a comfortable, familiar medium for generating handwriting strokes on paper while movement of the input device is detected, recorded, and/or used to generate electronic ink. Pressure sensor 203 may be included for designating an input, such as might be indicated when the input device 201 is depressed while positioned over an object, for example. Moreover, the pressure sensor 203 may be used to detect the depressive force with which the user makes strokes with the input device, and such data may be used in varying the width of electronic ink generated. In some examples, however, the pressure sensor may be eliminated

[0047] Processor 207 may be comprised of any known processor for performing functions associated with various aspects of the invention. For example, the processor may include an FPSLIC AT94S40, that particular device including a 20 MHz clock and operating at 20 MIPS. The processor 207 may further operate to perform steps that may reduce power consumption, conserve power stored in power supply 211, such as powering down various components when the input device is inactive. Such powering down may be determined based on data or lack of data indicating an absence of movement and/or positioning of the device in a given amount of time, for example.

[0048] The processor 207 may further operate to calibrate and regulate the performance of various components. The processor also may be programmed to select from among a plurality of ink and/or movement processing algorithms stored in memory. For example, the processor may select an algorithm most suitable for detecting movement, in accordance with, for example, characteristics associated with the surface over which the device is moved or by the impact on sensing caused by the environment in which the device is used. When used in conjunction with an LCD screen, for example, the algorithm selected may be the one most useful for accurately recording inputs entered using fast, smooth stokes, while a different algorithm may be selected for processing inputs generated on paper made with slower, less smooth stokes. Other factors for selecting the algorithm may also be considered such as, for example, movement of the input device relative to the surface in an unstable environment.

[0049] The algorithm may be selected automatically based on performance considerations programmed into the input device. Alternatively, the processing algorithm may be selected manually. For example, using the input device and the display of a host computer, the user may scroll through options for setting the functions of the input device until arriving at the option associated with the appropriate settings for the input device, in a manner similar to that already found for monitors and the like in the Control Panel of MICROSOFT operating systems.

[0050] Alternatively, input device settings, such as the processing algorithm selection described, may be achieved using the input device alone. User selections may be activated by depressing the input device, drawing symbols with the device that when detected are determined to indicate user inputs, and by other such means. In one example, the user may draw a symbol associated with the control function for identifying the surface over which the input device is about to be used. For example, the user may draw the symbol associated with the input device, the shape of a pen. Recognition of this symbol by the processor may cause the input device to enter a mode for enabling the user to next select the processing algorithm described, or any mode setting. Writing the word settings may, for example, indicate to the device that the next instruction is intended to set a mode associated with the pen. In this example, the user may enter another symbol, such as a number previously associated with the surface (describe in a manual, for example), such as a 2 corresponding to an indication that the input device is to be used with paper, or a predetermined graphical symbol indicating the use of the input device with paper, for selecting the processing algorithm to be used. Next, depression of the input device a certain number of times may then indicate that the mode setting procedure is complete, for example. While specific examples for interactively controlling the input device have been provided, use of any input and selection technique is well within the scope of the present invention.

[0051] In one embodiment, memory 208 may include one or more RAMs, ROMs, FLASH memories, or other memory device or devices for storing data, storing software for controlling the device, or for storing software for processing data, or otherwise. As noted, data representing positioning or the location of the input device may be processed within the input device 201 and stored in memory 208 for transfer to a host computer. Alternatively, the captured image data may be stored in memory 208 for transfer to a host device for processing or otherwise.

[0052] Transceiver, or communication unit, 209 may include a transmission unit and receiving unit. Information representing movement of the input device, either processed into a form suitable for generating and/or displaying electronic ink or otherwise, may be transmitted to a host computer, such as a desktop computer, laptop computer, Tablet PC, personal digital assistant, telephone, or other such device for which user inputs and/or electronic ink might be useful. The transceiver 209 may communicate with an external device using any wireless communication technique, including blue-tooth technology, for performing short-range wireless communications, infrared communications, or even cellular or other long-range wireless technologies. Alternatively, the transceiver 209 may control the transmission of data over a direct link to a host computer, such as over a USB connection, or indirectly through a connection with docking cradle 212, for example. The input device may also be linked directly to a particular host computer using a dedicated connection. The transceiver 209 may also be used to receive information and/or software, which, in one embodiment, may be downloaded for improving performance of the input device. For example, program information for updating the control functions of the processor may be downloaded from the host computer or otherwise via any of the previously described transmission/coupling techniques. Moreover, software may also be transmitted to or downloaded by the input device, including software for analyzing the image data and/or for calibrating the input device.

[0053] Power supply 211 may be included, particularly if the input device 201 is to be used independent of and remotely from a device in which the data is to be processed, stored and/or displayed. Power supply 211 may be incorporated into the input device 201 in any number of locations. For example, power supply 211 may be positioned for immediate replacement, should the power supply be replaceable, or to facilitate its recharging, should the power supply be rechargeable. Alternatively, the input device may be coupled to alternate power supplies, such as using an adapter for electrically coupling the input device 201 to a car battery, through a recharger connected to a wall outlet, to the power supply of a computer, or to any other power supply. Docking station adapter 212 may be comprised of known structure for electrically coupling the input device to other devices or to a power supply. The docking station adapter 211 also may be used for transferring information to an external device to which the docking station is coupled, directly or indirectly. Similarly, software, control information, or other data may be uploaded to the input device 201 via the docking station, which may also be used to recharge power supply 206.

[0054] Removable cap 213 may be used to cover the tip of the input device, which may include ink cartridge 202, and may also be equipped with a tip for facilitating resistive sensing if input device 201 is to be used with a device that includes a sensing board, for example. Similarly, inductive element 215 also may be included to enhance performance of the input device when used as a stylus in an inductive system. The shell of input device 201 may be comprised of plastic, metal, a resin, a combination thereof, or any material that may provide protection to the components of the device or provide overall structural integrity for the input device. While the input device shell may be made from a metal material, which may electronically shield sensitive electronic/magnetic components of the device, it need not be. For example, incorporation of a magnetic sensor may suggest that the structure of the input device be designed so that one or more sensor in the device is not shielded, so that the input device may, for example, detect magnetic fields.

[0055] Sensors for measuring movement or positioning of the input device consistent with various aspects of the present invention will now be described. Inertial sensors may include a number of sensor types. As illustrated in FIG. 2, gyroscope 206 may be included to provide indications of the angular velocity of the input device 201. Samplings of such indications over time t may reflect movement of the input device during that time. In one embodiment, the gyroscope may provide an indication of movement and/or positioning of the input device in two dimensions. To determine the orientation of the input device, data representing the angle of the input device in the third dimension is required. In one embodiment, data generated by the set of magnetometers may be used to supplement this missing data.

[0056] In another illustrative embodiment of the invention, the orientation of the device is obtained with measurement of a set of accelerometers and a set of magnetometers.

[0057] In yet another illustrative embodiment of the invention, such as if acceleration cannot be measured accurately or additional information is desired, data generated by the magnetic sensor 205 may be used to derive the angular velocity of the input device in the third dimension. Magnetic sensor 205 may measure movements of the input device by detecting variations in measurements of the earth's magnetic field or the magnetic field generated by digitizer 165. Thus, data from the gyroscope may be used in combination with data from the magnetic sensor to obtain the orientation of the input device. A comparison of the orientation of the input device at a first time with that at a second time may provide, in part, information necessary to determine movement of the input device, as will be described further below.

[0058] Measurements providing an indication of movement of the input device the tip also may be useful in approximating movement of the input device, and thereby aid in the accurate generation of inputs including ink. For example, a sensor for measuring movement of the tip of the input device may be placed near or at the tip. Such placement, however, may interfere with positioning and/or use of an ink cartridge, and may cause the exposed sensor to be subject to damage. In alternative embodiments, a single accelerometer or set of accelerometers may be located within the input device for generating data representing acceleration of the device at points other than its tip. Such data may translated into data representing acceleration of the tip of the input device using translation algorithms. In one embodiment, data acquired from accelerometer 204 and accelerometer 210 may be translated to represent acceleration of the tip of the input device.

[0059] As will be describe in more detail, data representing the orientation of the input device in combination with data representing the acceleration of the tip of the input device can be processed consistent with further aspects of the invention to accurately track movement of the input device at or near the surface over which the input device is moved and thereby produce data accurately representing the handwriting of the user. The resulting electronic ink will be of a higher quality with smoother transitions between incremental movements of the input device.

[0060] FIG. 3 represents the input device and depicts examples of the data generated by representative inertial sensors. As depicted, gyroscope 306 produces indications of the angular velocity of the input device in two axes. Accelerometer 304 and accelerometer 310 generate indications of acceleration in three axes. Similarly, the magnetometer 305 senses changes in the magnetic pull of the earth along three axes. In combination, these measurements may produce extremely accurate indications of movement of the input device and thereby facilitate accurate recording of handwriting, via electronic ink, and other inputs. Alternative sensors to those recited may also be used to generate inputs.

[0061] Accurate representations of handwritten strokes may also be obtained using less than all of the previously described sensors. For example, gyroscopes perform in a complimentary manner to the performance of accelerometers. Where the velocity of an object remains relatively constant, the acceleration of that object may be negligible. Thus, for example, where the input device's movements are few over an extended period of time, indications of acceleration may prove inaccurate. In those instances, representation of angular velocity obtained using the gyroscope which may alone produce sufficient data to generate accurate results. During the time that the acceleration of the input device is dramatic, however, the velocity of the input device may not be a good indicator of movement. At such times, data generated by the accelerometers may provide the best indication of movement.

[0062] Moreover, because the gyroscope only produces information representing the orientation of the input device along two axes, for improved results, data representing the third axis may be obtained using the accelerometer, the magnetometer, or another gyroscope for example. When the device moves, the 2-axis gyroscope is used to represent the orientation of the input device along two axes. The set of magnetometers may provide the orientation of along the remaining axis. The acceleration of the pseudo standing point may be used to compensate for the drift error (and other possible errors) of gyroscopes.

[0063] As demonstrated, aspects of the invention enable the use of data generated by fewer than all of the recited sensors, or alternative sensors, at different times to produce accurate results. Such accuracy and flexibility may add to the cost of the overall device. Alternatively, incorporation of fewer sensors may be desirable to reduce overall expense and complexity of the input device.

[0064] FIG. 4 is flowchart depicting one illustrative embodiment implementing various aspects of the present invention. As will be explained in greater detail to follow in this embodiment, the orientation of the input device in a next sampling time is determined using a revised Kalman filter. Values representing the angular velocity of the input device in two axes may be supplemented using data from, for example, either the accelerometers or the magnetometer. Those inputs may be processed by the Kalman filter, or other suitable processing technique, to generate a calculation of the orientation of the input device. Simultaneously, acceleration of the tip of the input device may be determined using a translation of data generated by two sets of accelerometers. The indication of the orientation of the input device and the values representing the acceleration of the tip of the input device are converted into data representing the acceleration of the tip of the input device in inertial coordinates. A transform matrix is generated for converting from inertial coordinates (the measurements taken along the input device) to paper coordinates (the coordinates of the surface over which the input device is moved, whether it is paper or some other surface). Using that transform, the acceleration of the tip of the input device in paper coordinates is determined. The acceleration of the tip of the input device in paper coordinates may then be twice integrated, the result providing the track of the input device over the course of the current sampling time. Drift correction may also be performed to improve the resultant track indications. Inertial coordinates may be used to define the location of the pen. For example, inertial coordinates and paper coordinates are shown in FIG. 6. The obtained orientation of the device in inertial coordinates may be transferred to the paper coordinate when the paper coordinate is not aligned with the inertial coordinate by a calibration process. A sample calibration process follows.

[0065] The input device is put in a special orientation C_pⁿ so that the pen coordinate is aligned with the paper coordinate. C_pⁿ is actually the orientation of the paper coordinate in inertial coordinate, and is calculated using the following algorithm. Any time when the orientation of the input device in inertial coordinate C_bⁿ is calculated, the orientation of the input device in paper coordinate is obtained as C_b^p(C_p ⁿ)¹C_bⁿ(C_pⁿ)^TC_bⁿ, where C_pⁿ, C_bⁿ and C_b^p are direction cosine matrix. (C_pⁿ)¹(C_pⁿ)^T is a basic property of direction cosine matrix. See, for example, D. H. Titterton and J. L. Weston, Strapdown Inertial Navigation Technology, IEE Radar, Sonar, Navigation and Avionics Series 5, 1997, Perter Peregrinus Ltd., whose contents are incorporated by reference.

[0066] In accordance with one embodiment of the present invention, as depicted in FIG. 4, electronic ink accurately representing handwriting is generated based on data output from one or a combination of inertial sensors. In the depicted embodiment, a two-axis gyroscope produces data representing the angular velocity of the input device as it moves from time 0 to time t, as indicated in step 401 of the flowchart. Those values may be transferred to the revised Kalman filter for processing, as represented by step 406. Information corresponding to movement of the input device in a third dimension, which may assist in estimating movement of the input device during time t, may supplement the information generated by the two-axis gyroscope. Therefore, in accordance with the illustrative embodiment illustrated, data representing movement of the input device in the third dimension may be determined from measurements of sensors in addition to a first sensor, such as the gyroscope illustrated.

[0067] In steps 402 and 403 of the flowchart, values representing the acceleration of the front portion of the input device, for example, generated by a first set of accelerometers, and values representing acceleration of the rear portion of the input device, generated by a second set of accelerometers, are used to supplement the missing data. To further improve accuracy, the acceleration values obtained may be calibrated using a pseudo-standing point calibration technique, represented in step 405, based on, for example, an estimation of the position at which, where the user reverses direction of the device, the acceleration of the input device reduces to zero. That calibration employs an algorithm that detects when the speed of the input device should be zero, and compensates for any offsets that might occur in detected data values. Alternative processing techniques for improving the accuracy with which acceleration data indicates movement may be used in place of, or in addition to, the calibration step 405, described. The acceleration values may then be input to the revised Kalman filter.

[0068] As shown in FIG. 7, when the input device moves, the orientation and acceleration information are coupled in the output of the accelerometers, the acceleration of the pseudo standing point of the input device contains less acceleration information and more orientation information, so the acceleration value of the pseudo standing point is used to estimate orientation or provide a compensation of the gyroscope output. The acceleration value of the pseudo standing point may be obtained from the two sets of $a_x=\frac{l_{2\mathrm{xy}}}{l_{1\mathrm{xy}}+l_{2\mathrm{xy}}}{a}_{1x}+\frac{l_{1\mathrm{xy}}}{l_{1\mathrm{xy}}+l_{2\mathrm{xy}}}{a}_{2x}$ $a_y=\frac{l_{2\mathrm{xy}}}{l_{1\mathrm{xy}}+l_{2\mathrm{xy}}}{a}_{1y}+\frac{l_{1\mathrm{xy}}}{l_{1\mathrm{xy}}+l_{2\mathrm{xy}}}{a}_{2y}$ $a_z=\frac{l_{2x}}{l_{1x}+l_{2x}}{a}_{1z}+\frac{l_{1z}}{l_{1z}+l_{2z}}{a}_{2z}$

[0069] where l_1xy and l_2xy are the distances from the two sets of x-y axis accelerometers to the pseudo standing point, l_1z and l_2z are the distances from the two sets of z axis accelerometers to the pseudo standing point, a_x, a_y and a_z are the accelerations of pseudo standing point in three axes respectively. a_1x, a_1y, a_1z are the 3-axis accelerations of the bottom set of accelerometers, a_2x, a_2y and a_2z are the 3-axis accelerations of the top set of accelerometers.

[0070] As previously noted, data generated by the accelerometers may not consistently provide accurate indications of movement of the input device. For example, when the input device is moved with substantially constant velocity, and the acceleration of the input device in any direction is approximately zero, the accelerometers may yield undesirable results. In such cases, measurements generated by alternate sensors may be used in place of acceleration data. For example, a three-axis magnetometer, such as that previously described, may aid in estimating movement of the input device by providing an indication of movement of the input device in a third dimension, for example. The magnetometer may sense movement of the input device by measuring changes in measurements corresponding to the earth's magnetic field, as shown in step 404. Those measurements may then be input into the revised Kalman filter for processing in step 406. Of course, such sensors are highly sensitive to magnetic interference and may require frequent recalibration of the device and/or processing of the measured data may be required to correct for interference.

[0071] In accordance with the illustrated embodiment, as depicted in step 406 of the flowchart, data from the gyroscope and the accelerometers or the magnetic sensors are all input to the processor for processing in accordance with a revised Kalman filter. The filter produces data representing the current 3D orientation of the input device, as shown in step 407. The illustrative embodiment describes use of a revised Kalman filter for estimating the orientation of the input device, in part, because the Kalman filter is a recursive feedback filter employing a least-squares method that may accurately predict future values based on current values. Alternatively, other predictive filters or processing techniques for generating an accurate representation of the orientation of the input device using any number of measured data indicative of movement and/or positioning of the device may also be employed. Other filtering techniques may also be used including but not limited to the extended Kalman filter

[0072] Simultaneously, as depicted in step 408, acceleration of the tip of the input device, such as a pen, in pen coordinates (the coordinates corresponding to the input device), may be determined from data output by the two sets of accelerometers, for example. As previously indicated, data from the two sets of accelerometers, each positioned at spaced locations along the axis of the input device, is transformed using a transformation process to provide an indication of the acceleration of the input device at the tip during time interval t. In one example, the transformation from coordinates along the axis of the input device to the tip of the device may be predetermined using previous calibrations, estimation techniques, or any technique for translating values to accurately represent movement of the tip of the input device. Data representing acceleration of the input device at its tip in pen coordinates may then be converted to data representing the acceleration of the input device at its tip in inertial coordinates, shown in step 409, using the previously described current 3D orientation of the input device determined in step 407.

[0073] One may determine the pen tip acceleration as follows. Pen tip acceleration is obtained as follows. $a_x^{\mathrm{tip}}=\frac{l_{4\mathrm{xy}}}{l_{4\mathrm{xy}}-l_{3\mathrm{xy}}}{a}_{1x}-\frac{l_{3\mathrm{xy}}}{l_{4\mathrm{xy}}-l_{3\mathrm{xy}}}{a}_{2x},a_y^{\mathrm{tip}}=\frac{l_{4\mathrm{xy}}}{l_{4\mathrm{xy}}-l_{3\mathrm{xy}}}{a}_{1y}-\frac{l_{3\mathrm{xy}}}{l_{4\mathrm{xy}}-l_{3\mathrm{xy}}}{a}_{2y},a_z^{\mathrm{tip}}=\frac{l_{4z}}{l_{4z}-l_{3z}}{a}_{1z}-\frac{l_{3z}}{l_{4z}-l_{3z}}{a}_{2z},$

[0074] where l_3xy, l_4xy are the distances from the 2 set of x-y axis accelerometers to the pen tip, l_3z and l_4z are the distances from the 2 set of z axis accelerometers to the pen tip, a_tip^x, a_tip^y and a_tip ^z are the accelerations of the pen tip in 3 axes respectively. FIG. 8 shows the dimensions relating to the above algorithms.

[0075] A transformation matrix for transforming from the inertial coordinates to the coordinates of the surface over which the input device is moved may be determined, for example, based on a calibration procedure performed previously. Such calibration steps may occur at the time of assembly, manufacture, before each use, at pauses during use, or at any time. Where the surface over which the input is moved is paper, the coordinates of such a surface have been described as paper coordinates. However, the input device may be moved over any number of surfaces, and reference to paper coordinates is not intended to limit use of the input device or the description of the transform to such surfaces.

[0076] As shown in step 410 of the flowchart, acceleration of the input device at the tip in paper coordinates, where the device is moved over paper, is determined by converting acceleration data into inertial coordinates using the transformation matrix described. The resultant acceleration may then be twice integrated, as depicted in step 411, thereby generating a track of the movement of the input device. Finally, the track data may further be processed by performing drift correction. As a result of the above-described operation, movement of the input device at the tip is tracked, producing data representing the electronic ink corresponding to movement of the input device.

[0077] The following is an example of how drift correction may function. Initially, the states of the input device are labeled as moving or not moving in each sampling time by judging the consistency of the acceleration obtained in the current sampling time and the adjacent several sampling points. For example, one may use 8 sampling points before and 8 sampling points after the current sampling time. It is appreciated that other numbers of sampling points may be used. If all the measured acceleration values during this 17 points period are near a fixed value, the state of this sampling time is labeled as not moving. The fixed value may then be treated as the acceleration drift error. (The sampling period is typically 10 ms.) When a new not moving state is detected, the acceleration values in the last continuous moving state are revised using the following algorithm.

[0078] It is assumed that the drift error is linearly increasing during a moving state period. The increasing ratio is calculated according to the total drift error in the labeling phase. Next, the drifting errors in each sampling point are subtracted from the acceleration values in the period to obtain the revised acceleration values in the period. By integrating the revised acceleration values, the revised velocity values are obtained, but the revised velocity value in the first not moving state after this moving period is still possibly not zero. It is further assumed the velocity drift error is linearly increasing in the last moving period. The revised revised velocity values are obtained by using the similar approach in the acceleration revising case. Consequently the revised revised velocity values are integrated to obtain the revised displacement value in the last moving period. The same algorithm is utilized for the drift correction of the input device in 2 directions on a surface, but only one direction case is illustrated here. FIGS. 9A-11C show the various applications of drift correction. FIG. 9 relates to acceleration. FIG. 10 relates to velocity. FIG. 11 relates to displacement. FIGS. 9A, 10A, and 11A relate to actual input values. FIGS. 9B, 10B, and 11B show measured values including the drift error. FIGS. 9C, 10C, and 11C show the corrected versions of each.

[0079] As previously described, one aspect of the invention involves processing of sensed data to determine movement of the input device, and determining electronic ink representing such movement. In one illustrative embodiment a revised Kalman filter was used for calculating the three dimensional orientation of the input device in the inertial coordinates. Derivation of such a processing algorithm is described below.

[0080] Equation (1) shows the basic system translation matrix:

[0081] where, C_bⁿ is the direction cosine matrix that translates from pen coordinates to inertial coordinates: ${}_b=\left[\begin{array}{ccc}0& -{}_z& {}_y\\ {}_z& 0& -{}_x\\ {}_y& {}_x& 0\end{array}\right],$

[0082] _{x y z}^T, where _x,y,_z is the angular rate vector in pen coordinates, which can be measured using the gyroscope.

[0083] Basic equation (1) can be simplified to:

[0084] {dot over (C)}C, where its discrete form is as follows:

[0085] where, $A\left(k\right)=I_{33}+\left[\right]t+I_{33}+\left[\right]t=\left[\begin{array}{ccc}1& -{}_z& {}_y\\ {}_z& 1& -{}_x\\ -{}_y& {}_x& 1\end{array}\right]$

[0086] where t, _x,_y,_z^T, where t is the sampling time.

[0087] And the state vector becomes x ₁ x₂ X₃^T_{x y z}^T

[0088] Then the state translation is broken into the following:

[0089] x₁(k1)_x(k1){overscore ()}_x(k,k1)t

[0090] x₂(k1)_y(k1){overscore ()}_y(k,k1)t

[0091] X₃(k1)_z(k1){overscore ()}_z(k,k1)t

[0092] where, {overscore ()}(k,k1) is the average value of the angular velocity from t(k) to t(k1). Because _z is unknown, however, a measurement from the accelerometers, or magnetic sensors, may be used to calculate x₃(k1). Equations for determining angular velocity in inertial coordinates are described as follows. The acceleration in inertial coordinates is calculated by transforming the measured acceleration of the input device in pen coordinates (b) to inertial coordinates using the C_bⁿ translation.

[0093] To determine the magnetism of the input device in inertial coordinates, which is equal to that of the acceleration in inertial coordinates the same calculation is performed.

[0094] Equation (3) can be rewritten to standard form as: ${\stackrel{}{a}}_n=C_b^n\left(k+1\right){\stackrel{}{a}}_b\left(k+1\right)=C\left(k\right)A\left(k\right){\stackrel{}{a}}_b$ $C^T\left(k\right){\stackrel{}{a}}_n=A\left(k\right){\stackrel{}{a}}_b={\stackrel{}{a}}_b-\left[{\stackrel{}{a}}_b\right]\stackrel{}{x}{\stackrel{}{a}}_b-C^T\left(k\right){\stackrel{}{a}}_n=\left[{\stackrel{}{a}}_b\right]\stackrel{}{x}$

[0095] C^T(k)C¹ (k), where ()^T means matrix transform, ()¹ means matrix inverse, this is a property of direction cosine matrix.

[0096] The measurement vector corresponding to measured acceleration becomes:

[0097] Substituting the equation becomes z_ab. Rewritten in detail becomes, $\{\begin{array}{cc}{z}_{a_1}=-a_z{x}_2+a_y{x}_3& 5\left(a\right)\\ z_{a_2}=a_z{x}_1-a_x{x}_3& 5\left(b\right)\\ z_{a_3}=-a_y{x}_1+a_x{x}_2& 5\left(c\right)\end{array}$

[0098] Similarly, the measurement vector corresponding to magnetic sensor is:

[0099] With substitution the equation is solved into, z_mb, which can be rewritten in detail as follows, $\{\begin{array}{cc}{z}_{m_1}=-m_z{x}_2+m_y{x}_3& 6\left(a\right)\\ z_{m_2}=-m_z{x}_1-m_x{x}_3& 6\left(b\right)\\ z_{m_3}=-m_y{x}_1+m_x{x}_2& 6\left(c\right)\end{array}$

[0100] The determination of state translation for x₃ (k1) can be obtained using the first and second sub-equations of equations (5a-5c), by solving for x₃. Then the measurement equations can be constructed by the rest of equation (6) and equation (1).

[0101] The determination of state translation for x₃ (k1) is obtained from equation (5a) and (5b). The measurement equations are constructed by the equation (5c), (6a), (6b) and (6c) as follows. $\left[\begin{array}{c}{z}_{a_3}\\ z_{m_1}\\ z_{m_2}\\ z_{m_3}\end{array}\right]=\left[\begin{array}{ccc}-a_y& a_x& 0\\ 0& -m_z& m_y\\ m_z& 0& -m_x\\ m_y& m_x& 0\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]$

[0102] When the input device is parallel to the magnetic field of the earth, both m_x and m_y will be extremely small, and the error in the value X₃ (k1) obtained from the equation (5a) and (5b) will become too great. In that case, the equation (6a) and (6b) are used to obtain x₃ (k1), consequently, the equation (5a) (5b) (5c) and (6c) are used to construct measurement like the equation (7) in the comment 13.

[0103] In this revised Kalman filter, the output of gyroscope, as well as part of the output of magnetometers or accelerometers provide the state prediction, the other part of the output of magnetometers and accelerometers provide the state measurement, and correct the prediction error.

[0104] As described in one embodiment of the present invention, using a modified Kalman filter, the current attitude of the input device in inertial coordinates can be determined. Having determined the translation matrix for conversion from the pen coordinates to the inertial coordinates, and having previously obtained the transformation matrix corresponding to the transformation from inertial coordinates to paper coordinates by calibration, the transformation matrix from pen coordinates to paper coordinates can be determined. Accordingly, the track of the input device's tip can be calculated by integrating the acceleration of the tip transformed into the spatial coordinates of the surface over which the input device is moved. Accurately determining movement of the pen will provide data necessary for generating accurate inputs, by accurately measuring movement of the input device tip for recording strokes in a manner similar to that generated by conventional writing instruments, with the exception that the input device need not contact the surface for such movements to be recorded.

[0105] While the illustrative embodiment described employed a single gyroscope, plural acceleration sensors, and a magnetic sensor, that description is in no way intended to limit the invention to systems using those components. Rather, any substitution, supplementation, removal or other modification of the illustrative embodiment described is within the scope of the present invention.

[0106] FIG. 5 illustrates an illustrative embodiment of an input device incorporating a camera for capturing images for use in accordance with various aspects of the invention. In FIG. 5, input device 501 includes ink cartridge 502, pressure sensor 503, accelerometer 504, magnetic sensor 505, gyroscope 506, processor 507, memory 508, transceiver 509, accelerometer 510, power supply 511, docking station adapter 512, cap 513, camera 514, inductive element 215 and a display, not shown. Some of the elements may be omitted. For example, the ink cartridge may be eliminated if reliance solely on the display is desired.

[0107] In this embodiment, a camera may be added to complement, or in place of, one or more of the sensors described, to assist in tracking lateral movement or the orientation of the input device. Camera 514 may be included to capture images of the surface over which the input device is moved, and thereby to detect movement of the input device over the surface being scanned. In that case, processing of optical information may be performed and combined with, or used in place of, data generated by one or more sensors described, with modifications to the processing algorithm made as necessary to accurately determine movement of the input device. Measurement of positioning or movement of the input device may be performed based on, at least in part, detection of patterns or changes in features within the images captured depicting the surface over which the input device is moved, using an image sensor built into the tip of the input device and the appropriate image processing techniques, for example. Such additional information may be factored into the algorithm employed to determine position and/or movement of the input device for the generation of electronic ink. As elements may be added and/or removed, processing techniques may be replaced, updated, or otherwise modified to compensate for the loss of information or the addition of information generated by those devices. Modifications to the Kalman filter described, or other processing algorithm used, may be made consistent with the present invention, and the use of different processing techniques for estimating pen tip movement from data generated by one or more sensors for measuring movement of the input device is within the scope of the present invention.

[0108] While the illustrative embodiments illustrated depict devices utilizing specific components, the addition of components and/or removal of the component depicted, with modification to the device consistent with such changes, is within the scope of the present invention. Similarly, the relocation of components, within or external to the input device body, is further within the scope of the present invention.

[0109] The illustrative embodiments have described an input device implemented in the shape of a writing instrument such as a pen. Aspects of the present invention are applicable, however, to input devices of any number of shapes and sizes. For example, the input device may take on an ergonomic shape and may include indentations for improved comfort and control.

[0110] Although the invention has been defined using the appended claims, these claims are illustrative in that the invention may be intended to include the elements and steps described herein in any combination or sub combination. Accordingly, there are any number of alternative combinations for defining the invention, which incorporate one or more elements from the specification, including the description, claims, and drawings, in various combinations or sub combinations. It will be apparent to those skilled in the relevant technology, in light of the present specification, that alternate combinations of aspects of the invention, either alone or in combination with one or more elements or steps defined herein, may be utilized as modifications or alterations of the invention or as part of the invention. It may be intended that the written description of the invention contained herein covers all such modifications and alterations. For instance, in various embodiments, a certain order to the data has been shown. However, any reordering of the data is encompassed by the present invention. Also, where certain units of properties such as size (e.g., in bytes or bits) are used, any other units are also envisioned.