Method of generating digital ink thickness information

[0001] The invention relates generally to digital ink, and more particularly to a method for generating digital ink information.

[0002] Digital ink technology enables a user to write and draw on the touch-sensitive screen of a handheld PC or other writing tablet with a stylus or other pointing device, providing a convenient means for applications to accept input from a user without using a keyboard. For a user, taking notes or drawing sketches with using digital ink technology is very much like writing or drawing on paper.

[0003] Contemporary digital ink technology is capable of extracting a variety of information from a user's handwriting, including vector, timing, coordinates, angle of the stylus, and additional information. The digital ink information may be provided to an application, and may be used for many purposes, such as for handwriting recognition. The digital ink information may also be used for improving the way handwriting is displayed, including providing higher resolution, editing, smoothing, and alteration of individual elements, for example.

[0004] Contemporary digital ink technology is not, however, effective in generating line thickness information. Thickness information may be helpful, for example, to provide high-resolution display of calligraphy, or for font generation of some characters, such as Chinese characters. Some writing tablets have attempted to utilize pressure sensors in an effort to obtain line thickness information. However, the dynamic range of the pressure sensors in the writing tablets is limited, especially at the pressures at which people tend to write. Thus, it has been found that the pressure differentiation information provided by the existing pressure sensors is not sufficient to provide line thickness information, especially for fountain pen simulation and font generation of Chinese characters.

[0005] The present invention provides a thickness conversion component that converts movement of a pen (e.g., ballistic movement of a pen point across a surface or tilting of a pen) into thickness information for digital ink data. In accordance with one aspect of the invention, the pen includes at least one accelerometer that is used to generate either ballistic movement or ballistic pen tilting information.

[0006] In one implementation of the invention, the thickness conversion component is located on a computer that is remote from the pen. In this implementation, the ballistic movement or tilt information is transmitted to the computer via a hardwired or wireless connection. For example, the accelerometer generates the ballistic movement or tilt information in the form of pulses, the width of each pulse being directly related to the acceleration of the pen movements or the tilt of the pen, respectively. The thickness conversion component converts the acceleration information, with or without additional information such as coordinate information, available pressure information, pen angle information, and vector information, into thickness information for digital ink. This thickness information may be used to generate variably thick lines, which may be useful for a variety of applications, for example, better display and improved recognition.

[0007] Other advantages will become apparent from the following detailed description when taken in conjunction with the drawings, in which:

Exemplary Operating Environment

[0015] FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented. The computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100.

[0016] The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microcontroller-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0017] The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0018] With reference to FIG. 1, an exemplary system for implementing the invention includes a general-purpose computing device in the form of a computer 110. Components of the computer 110 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

[0019] Computer 110 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computer 110 and includes both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by the computer 110. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer-readable media.

[0020] The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation, FIG. 1 illustrates operating system 134, application programs 135, other program modules 136, and program data 137.

[0021] The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive 140 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152, and an optical disk drive 155 that reads from or writes to a removable, nonvolatile optical disk 156 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140, and magnetic disk drive 151 and optical disk drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150.

[0022] The drives and their associated computer storage media, discussed above and illustrated in FIG. 1, provide storage of computer-readable instructions, data structures, program modules, and other data for the computer 110. In FIG. 1, for example, hard disk drive 141 is illustrated as storing operating system 144, application programs 145, other program modules 146, and program data 147. Note that these components can either be the same as or different from operating system 134, application programs 135, other program modules 136, and program data 137. Operating system 144, application programs 145, other program modules 146, and program data 147 are given different numbers herein to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 20 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, a touch-sensitive screen of an handheld PC or other writing tablet, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 190.

[0023] The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0024] When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs 185 as residing on memory device 181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

Generating Digital Ink Thickness Information

[0025] Typically, a user writing on a touch-sensitive screen or tablet generates digital ink information with a stylus or other writing instrument. Generally, digital ink technology stores and/or processes information about stylus, mouse, or other pointer movements, along with enhanced information such as calculated vector information, pressure, timing, strokes, angle of stylus, italic and bold states, and the like. There are a variety of different digital ink formats, and the additional information that the format can store or process with the pointer movements varies for the different applications.

[0026] In summary, the present invention is directed to a system and method for generating additional digital ink information in the form of thickness information. The thickness information permits the digital ink information to better match the user's actual writing movements. To this end, a thickness conversion component is provided that converts ballistic pen movement or ballistic pen tilt information generated by movements of a writing instrument (e.g., a pen or a stylus) into thickness information. The thickness information may be used, for example, for enhanced display or recognition purposes. Moreover, the ballistic movement or tilt information may be generated from a writing instrument, such as a pen, independent of a touch-sensitive screen. Thus, a user may generate the ballistic movement or tilt information while writing on paper or another surface, which is beneficial because writing on such surfaces is familiar to most users.

[0027] Turning now to the drawings, FIG. 2 shows generally an architecture for a system 200 for generating digital ink thickness information in accordance with one aspect of the invention. The system 200 includes a computer 202 (e.g., the computer 110) having a digital ink receiver 204. The digital ink receiver 204 receives and stores the raw data generated by a user's writing movements, processes that data if necessary, and forwards corresponding appropriate data to the appropriate software, such as the operating system or an application. In this manner, the digital ink receiver enables a user to input information into a computer utilizing a pointer device, and without having to use a keyboard.

[0028] In accordance with one aspect of the present invention, the digital ink receiver 204 includes, or alternatively is associated with, a thickness conversion component 206, which is configured to convert ballistic movement or tilt information into thickness information, as is further described below. The digital ink receiver 204 and the thickness conversion component 206 may be provided on a single PC (e.g., the personal computer 202), or the thickness conversion component 206 may be provided on a separate machine from the digital ink receiver 204. In addition, their various functions may be performed by a single device or by several devices.

[0029] The thickness conversion component 206 may be connected to a recognizer 208 and/or a display generation mechanism 209, each of which may be integrated with, or separate from, the computer 202. As further described below, one or both of these components, or other software, may utilize the output of the thickness conversion component 206.

[0030] The computer 202 is connected via a connection 210 to a writing instrument 220, such as a pen, stylus, pencil, paintbrush, stick, a pointer device, or any other mechanism through which a user may perform writing movements. Information about the writing instrument's movements is transmitted to the computer via the connection 210.

[0031] The connection 210 may be hardwired or wireless (wherein if wireless, the connection is conceptual, e.g., line-of-sight for infrared, or within range for FM transmissions, and so forth). As some examples, the computer 202 may be located remotely from the writing instrument 220, and transmission of information regarding movements of the writing instrument 220 to the computer may occur via a wireless transmission, a local area network (e.g., the LAN 171), a wide area network (e.g., the WAN 173), the Internet, or through another network or similar connection. Alternatively, writing information may be stored in memory in the writing instrument, and may be later downloaded to the remote computer. In addition, some or all of the functions of the digital ink receiver 204 and/or the thickness conversion component 206 may be provided in the writing instrument, although in practice, such a design may result in a mechanism that may be too cumbersome for comfortable writing.

[0032] The writing instrument 220 includes a ballistic generator 222 that is configured to generate motion information, such as acceleration and/or tilt information, as a result of writing movements. The ballistic generator 222 may be, for example, a dual-axis accelerometer, or a pair of accelerometers, or any other mechanism that is capable of generating data regarding information about movement or tilt of the writing instrument 220.

[0033] The writing instrument 220 also preferably includes an appropriate tip 224 for writing on a surface 230 (e.g., paper, a touch-sensitive screen, or a writing tablet). The tip 224 preferably is designed for optimum writing on the surface 230, such as a blunt end for a touch-sensitive screen or tablet, or a pen point or nib for writing on paper.

[0034] In practice, a user grips the writing instrument 220 and writes in a standard fashion, either on a writing tablet, or on an ordinary writing surface (e.g., paper). The ballistic generator 222 generates ballistic information regarding the writing instrument's movements or tilt, such as may be caused by a user's writing. If a touch-sensitive screen is utilized, additional digital ink information, such as calculated vector information, pressure, timing, strokes, angle of stylus, and the like, may be generated by the touch-sensitive screen or tablet.

[0035] The ballistic information is transferred to the computer 202 via the connection 210 and is received by the digital ink receiver 204. As described below, the thickness conversion component 206 converts the ballistic information, with or without additional information such as coordinate information, available pressure information, pen angle information, and vector information, into thickness information for digital ink. Software (e.g., applications, the operating system, the recognizer 208, or the display generation mechanism 209) that receive the line thickness information may utilize the thickness information, for example, to generate variably thick lines, which may be useful for a variety of purposes, for example, improved display of calligraphy and/or Chinese characters (e.g., via the display generation mechanism 209), and/or better recognition accuracy (e.g., via the recognizer 208).

[0036] FIG. 3 shows a representation of an embodiment of a digital ink generation system 300 that incorporates the present invention and that utilizes a wireless connection 310. The wireless connection 310 includes a receiver 312 at the computer 202, and a transmitter 314 at the pen 220. The writing instrument in the example shown in FIG. 3 is a pen 320.

[0037] The ballistic generator in the embodiment of FIG. 3 is an accelerometer 322, which is mounted adjacent to the tip 324 of the pen 220. It has been found that mounting the accelerometer 322 in this location permits the accelerometer to more accurately reflect the pen tip's trajectory, however other locations may work better with other types of components.

[0038] The accelerometer 322 is wired to an analog-to-digital (A/D) converter 326, which in turn is connected to a microcontroller 328. Alternatively, a digital output accelerometer may be used, whereby the A/D converter 326 is not needed. The microcontroller 328 is wired to the transmitter 314. A battery 330 is attached to the microcontroller 320 to provide power to the circuit.

[0039] For the embodiment shown in FIG. 3, a user grasps the pen 320 and writes on a surface such as paper or a writing tablet. The accelerometer 322 generates analog ballistic information, in the form of pulses, and forwards that information to the A/D converter 322, where the analog ballistic information is converted to digital ballistic signals. Alternatively, the accelerometer 322 may output digital signals, whereby the A/D converter 322 is not needed. The microcontroller 320 controls the transfer of data from the pen 220 to the computer 202. In the embodiment shown in FIG. 3, the transmitter 314 sends the digital ballistic signals to the receiver 312. The digital ballistic signals are converted by the thickness conversion component 206 into thickness information, as is further described below.

[0040] It can be understood that some the components of the pen may be distributed between the computer 202 and the pen 320. For example, analog ballistic information may be transmitted to the computer, and the analog-to-digital converter 326 may be located at the computer.

[0041] FIG. 4 is a representation of a graph of data generated by an accelerometer (e.g., the accelerometer 322) as a result of movements of a writing instrument (e.g., the writing instrument 220). The graph maps acceleration, as wavelength of pulses generated by the accelerometer, versus time for both X and Y movements of the pen 220.

[0042] The relative acceleration and deceleration of the pen in the X and Y axes that is measured by the ballistic generator 222 may be used to provide dynamic information regarding the motion of the pen and hence form a two-dimensional vector stream representing the pen movement. Other pen movement information may be extracted from, for example, a handwriting tablet, or sensors placed in the writing instrument 220. The additional pen movement information may also be used to extract vector information for handwriting recognition.

[0043] Applicants have also found that thickness information can be determined from the relative acceleration and deceleration of the pen 220 in the X and Y-axes. In essence, pulses generated by the accelerometer have a larger wavelength as a result of slowing down of movement of the pen in the particular dimension. The slower movements by the pen generate the larger the wavelength of the pulses, which generally represent positions where a user's handwriting pattern would be thicker. Thus, there is a direct correlation between the wavelength and pen thickness information. Other pen movement information may be used to enhance the ballistic information that is extracted by the ballistic generator 222, e.g., by adding thickness information such as in the form of pressure information.

[0044] Using the thickness information extracted by the ballistic generator (and pressure information, if available), the thickness conversion component 206 can provide instructions or other information to software or the display generation mechanism 209 that instructs the software or display generation mechanism to amplify the digital ink (e.g., by adding additional bits to a bitmap, or by increasing the thickness of lines in an application) in places where pulse widths are read to be larger, or alternatively may thin lines where pulses are read to be smaller. In this manner, thickness information may be produced from accelerometer pulse information.

[0045] In another embodiment shown in FIG. 5, the ballistic generator 522 is a dual-axis accelerometer that is arranged to produce tilt data (i.e., the accelerometer is arranged to be used as a tilt sensor). When used as a tilt sensor, an accelerometer uses the force of gravity as an input vector to determine the orientation of an object. An accelerometer is most sensitive to tilt when its sensitive axis is perpendicular to the force of gravity, i.e., parallel to the earth's surface. At this orientation its sensitivity to changes in tilt is highest. Thus, for the dual-axis accelerometer to be used as a tilt sensor, it is preferred that the X and Y axes for the accelerometer be aligned perpendicular to the axis of the writing instrument 220, and parallel with the ground when the pen is in an upright position. An example of a dual-axis accelerometer that works well for this purpose is the +/−2 g Dual-Axis Accelerometer with Duty Cycle Output produced by Analog Devices, Inc. of Norwood Mass. under the model number ADXL202E.

[0046] For the tilt sensor embodiment, mounting the tilt sensor adjacent to the top end of the writing instrument 220 may produce the most accurate information, but the tilt sensor may be mounted in other locations as desired. The remaining components of the tilt sensor are essentially the same as the previous embodiment (i.e., the microcontroller 528, the transmitter 514, and battery 550). The ADXL202E accelerometer produces digital output, so an analog-to-digital converter is not needed.

[0047] The dual-axis digital accelerometer (e.g., the ADXL202e accelerometer) generates ballistic angle movement information regarding the writing instrument 220. Pen position data is generated, for example, at 60 samples/sec/channel. The x and y channel data is read into the microcontroller 528 and is combined into a single channel of ASCII data for transmission by the transmitter 314, e.g., at 1200 Baud. The transmitter 314 may be, for example, a 433 MHz, AM transmitter having a range up to 100 meters. The transmission may or may not be encrypted.

[0048] The raw data from the accelerometer is received by the digital ink receiver 504, and is converted to pitch and roll angles in a manner known in the art (see, for example, the product literature for the ADXL202E, incorporated herein by reference). The pitch and roll angles may then be used to plot relative X and Y components, which represent the pen movements that are generated by a user.

[0049] As an example, FIG. 6 shows a representation of a parallelogram 600 that may be drawn by a user with the pen 520. The computer 202 generates a graph 610 that represents a plot of the X and Y positions of the data generated by the tilt sensor 520. Although the sides do not align vertically and horizontally in the graph plot shown in FIG. 6, rotating the pen 520 in the user's hand may generate another set of data points that align with the vertical and horizontal axes in the graph. In any event, alignment of the data plots is not critical, because the data points may be properly interpreted because they are read relative to one another, and not to the orientation of the graph 610.

[0050] Applicants have found that the data plots in the graph 610 may be utilized to generate thickness information. As the user draws faster (and thus with less pressure and less thickness), there are less samples/unit distance (see sides of parallelogram). As the user draws more slowly (and thus with greater pressure and more thickness), there are more samples/unit distance (see corners). Using this information, thickness information may be generated by directly correlating the proximity of adjacent plots to thickness information. That is, closely packed plots are generated around thicker lines, and separated plots are generated on thin lines.

[0051] Using this information, the thickness conversion component 206 can provide instructions or other information to software or the display generation mechanism 209 that instructs the software or display generation mechanism to amplify the digital ink (e.g., by adding additional bits to a bitmap, or by increasing the thickness of lines in an application) in places where pulse widths are read to be larger, or alternatively may thin lines where pulses are read to be smaller. In this manner, digital ink thickness information may be generated from a tilt sensor.

[0052] Turning now to an explanation of the operation of the present invention, FIG. 7 shows a general overview of a process for generating line thickness information from a writing input of a user in accordance with one aspect of the present invention. Beginning at step 700, a user utilizes a writing instrument (e.g., the pen 320) to generate ballistic information. As discussed above, the ballistic information may be generated by the accelerometer 322 or by the tilt sensor 522 as a result of writing movements by the pen 320 or the pen 520.

[0053] At step 702, the ballistic information is received by the computer 202 (e.g., through the connection 310). The thickness conversion component then converts the ballistic information to line thickness information (step 704). As discussed above, this conversion may be, for example, based upon pulse wavelength thickness for the use of an accelerometer, or proximity of plots if a tilt sensor is used. The line thickness information is forwarded to the appropriate application (step 706). The application may be, for example, recognition or display software.

[0054] While the invention is susceptible to various modifications and alternative constructions, a certain illustrated embodiment thereof is shown in the drawings and has been described above in detail. It should be understood, however, that there is no intention to limit the invention to the specific form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention.