US 7,499,589 B1Grant
Method and apparatus for using pressure information for improved computer controlled handwriting recognition, data entry and user authentication
Issue Date:2009-03-03
•19 Claims
•18 Drawing Sheets
Abstract
A method and system utilizing both (x, y) coordinate (“spatial”) stroke data and associated pressure information for improved handwriting recognition. The method and system can also be applied to all types of handwriting-based data entry applications and also to user authentication. The digitizer pad used in the computer system gives both spatial information and associated pressure data when a stroke is being drawn thereon, e.g., by a stylus. Pressure information can be used to differentiate between different character sets, e.g., upper case and lower case characters for certain alphabetic characters. The spatial stroke data then identifies the particular character. The pressure information can also be used to adjust any display attribute, such as character font size, font selection, color, italic, bold, underline, shadow, language, etc. The associated pressure information can also be used for recognizing a signature. In this case, a user is allowed to sign a name on the digitizer pad. This provides non-character based user authentication that relies not only on the spatial stroke data but also on the pressure applied at different points in the signed name or image. Pressure information can also be used to provide improved handwriting-based data entry. For instance, in a drafting program, the pressure of a drawn line can be used to determine its width. Generally, pressure data can also be used to improve handwriting recognition tasks and heuristics.
Metadata
Assignee
- PalmSource, Inc.
Inventors
- Michael Cortopassi
- Edward Endejan
Application Information
Application Number:US 10/672,382
Filing Date:2003-09-25
Priority Date:2003-03-01
Art Unit:2624
Classifications
IPC:
G06K9/18
Patent Drawings (18 sheets)
Description
[0001] This is a continuation of application Ser. No. 09/516,656 filed on Mar. 1, 2003 now U.S. Pat. No. 6,707,942 which is hereby incorporated by reference to this specification
BACKGROUND OF THE INVENTION
[0002] 1. Field of the Invention
[0003] The present invention relates to the field of data entry methods and mechanisms for computer systems. The present invention is directed to the field of palmtop computers and handwriting recognition systems and handwriting-based data entry and user authentication. Specifically, the present invention discloses an efficient mechanism for automatic character recognition and display attribute recognition.
[0004] 2. Related Art
[0005] As the components required to build a computer system have reduced in size, new categories of computer systems have emerged. One of the new categories of computer systems is the “palmtop” computer system. A palmtop computer system is a computer that is small enough to be held in the hand of a user and can therefore be “palm-sized.” Most palmtop computer systems are used to implement various Personal Information Management (PIM) applications such as an address book, a daily organizer and electronic notepads, to name a few. Palmtop computers with PIM software have been know as Personal Digital Assistants (PDAs).
[0006] Data entry on a palmtop computer has been a challenge. Since palmtop computer systems are very small, full-sized keyboards are generally not efficient input devices. Palmtop computers using keyboards have keyboard devices that are so small that a user cannot touch-type. Furthermore, to use a keyboard device, a user must either place the palmtop computer system down onto a flat surface, so the user can type with both hands, or the user holds the palmtop computer system with two hands and types with thumbs only.
[0007] Instead of a mechanical keyboard device, some palmtop computers utilize a touch screen and display an image of a small keyboard thereon. When a particular button is pressed or tapped, a small keyboard image is displayed on the display screen. The user then interacts with the on-screen small keyboard image to enter characters, usually one character at a time. To interact with the displayed keyboard image (e.g., “virtual keyboard”), the user taps the screen location of a character with a pen or stylus. That corresponding character is then recognized and added to a data entry field, also displayed on the screen. However, for experienced users, the virtual keyboard input system can be a tedious input process.
[0008] Instead of using a mechanical keyboard device or a displayed keyboard, many palmtop computers employ a pen and a digitizer pad as an input system. The pen and digitizer pad combination works well for palmtop computers because the arrangement allows a user to hold the palmtop computer system in one hand while writing with the pen onto the digitizer pad with the other hand.
[0009] A number of palmtop computer systems that rely on the pen and digitizer pad combination as the primary means of input have been introduced to the market. Most of these pen-based palmtop computer systems provide some type of handwriting recognition system whereby the user can write words and letters on the digitizer pad with a stylus. The palmtop computer system then converts the user's handwriting into a machine readable format such as ASCII code characters. Examples of pen-based palmtop computer systems that provide handwriting recognition include the Apple Newton (trademark) device and the Tandy Zoomer (trademark) device.
[0010] Consumer acceptance of many pen based palmtop computer systems has been limited due to the performance of handwriting recognition systems. When a human reads a handwritten message, the human uses various clues to decipher the handwriting such as the known spelling of words, the known subject of the message, the writer's character writing style, and knowledge of English grammar. Since it is very difficult to teach a computer system all these subtle handwriting recognition heuristics, most handwriting recognition systems only rely on the writer's input strokes and a dictionary of common words. Using such limited information, such handwriting recognition systems often yield inaccurate results.
[0011] Some handwriting recognition techniques divide the recognition task into steps for identifying individual characters. Another handwriting recognition system is described in U.S. Pat. No. 5,889,888, issued on Mar. 30, 1999 entitled, “Method and Apparatus for Immediate Response Handwriting Recognition System that Handles Multiple Character Sets.” This character recognition system recognizes a large number of different characters with less effort on the user's part (e.g., fewer “mode change” strokes). This U.S. Pat. No. 5,889,888 is incorporated herein by reference.
[0012] However, one of the great difficulties in machine-based handwriting recognition is selecting between different, yet similarly shaped (or drawn) characters. For example, when using the handwriting recognition processes of the U.S. Pat. No. 5,889,888 patent it is difficult to enter the letter “V.” The character based handwriting recognition typically mistakenly determines the input character to be the letter “U” for “V” strokes. In other instances, it is difficult to differentiate between a lower case character and an upper case character in cases when the upper case and the lower case characters are similar in shape, e.g., in the case of “o” versus “O” and “t” versus “T” and “F” versus “f,” etc. What is done instead is to require a special “prestroke” to indicate the case of the character. However, sometimes the prestroke is drawn incorrectly and it improperly becomes recognized as a character itself, thereby causing confusion and difficulty for the user. It would be advantageous to be able to eliminate this prestroke requirement to offer a more natural writing style for selecting between character sets. It would be advantageous, then, to provide a more effective handwriting recognition mechanism for a computer system.
[0013] Also, in many graphics applications and word processors, a user wants to select between different character sets, e.g., to select fonts, character sizes, line widths, colors, etc. Normally, small buttons or icons are placed on the screen by the application allowing a user to select these features. However, in the case of palmtop computers, the screen is very small and these buttons and icons become very small indeed and therefore hard to see and select. It would be advantageous to provide a mechanism for selecting character sets without requiring an on-screen button or icon to be selected first.
[0014] Authentication is also an important issue with respect to palmtop computers because they can so readily be stolen, concealed and then used by an unauthorized person. Heretofore, a character based password has been used for locking down the computer system until the password is correctly entered. It would be advantageous to provide a more secure authentication system that was not character based.
SUMMARY OF THE INVENTION
[0015] Accordingly, what is needed is an effective and natural mechanism and method for discriminating between the case of characters within a handwriting recognition system. What is also needed is a mechanism for allowing a user to select different character sets or display attributes within an application program without requiring that a small icon or window be first selected. What is needed is a palm sized computer system that offers an improved authentication mechanism that is not entirely character-based. The present invention provides these advantages and others not specifically mentioned above but described in the sections to follow.
[0016] A method and system are described utilizing both (x, y) coordinate (“spatial”) stroke data and associated pressure information for improved handwriting recognition and other uses. The method and system can also be applied to all types of handwriting-based data entry applications and also to user authentication. The digitizer pad used in the palmtop computer system gives both spatial information and associated pressure data when a stroke is being drawn thereon, e.g., by a stylus. Pressure information can be used to differentiate between different character sets, e.g., upper case and lower case characters for certain alphabetic characters. The spatial stroke data then identifies the particular character within the selected character set.
[0017] The pressure information can also be used to adjust any display attribute, such as character font size, font selection, color, italic, bold, underline, shadow, language, etc. Again, the spatial stroke data then identifies the particular character within the selected character set. The associated pressure information can also be used for recognizing a signature. In this case, a user is allowed to sign a name on the digitizer pad. This provides non-character based user authentication that relies not only on the spatial stroke data but also on the pressure applied at different points in the signed name or image. Pressure information can also be used to provide improved handwriting-based data entry. For instance, in a drafting program, the pressure of a drawn line can be used to determine its width. Generally, pressure data can also be used to improve handwriting recognition tasks and heuristics.
[0018] More specifically, an embodiment of the present invention includes a method and system of recognizing data representing a user-drawn character, the method comprising the steps of: a) accessing spatial stroke data and pressure data captured by a digitizer and representing the user-drawn character wherein respective pressure data is associated with respective spatial stroke data; b) storing the spatial stroke data and pressure data into a computer memory wherein pressure data of a first range represents a first character set and pressure data of a second range represents a second character set; c) performing character recognition on the spatial stroke data and the pressure data, the step c) comprising the steps of: c1) using the pressure data to identify a character set; and c2) using the spatial stroke data to identify a particular character within the identified character set; and d) displaying the particular character on a display screen of a computer system. Embodiments include the above and wherein the first character set comprises upper case alphabetic characters and wherein the second character set comprises lower case alphabetic characters.
[0019] Embodiments also include a method and system for using pressure data and spatial stroke data for drawing a graphics element, e.g., a line and differentiating between a thick element and a thin element for use within a graphics application. Embodiments also include method and system for using pressure data and spatial stroke data for performing user authentication. Embodiments also include method and system for using pressure data and spatial stroke data for improved handwriting recognition.
BRIEF DESCRIPTION OF THE DRAWINGS
[0020] FIG. 1 is system illustration of a palmtop or “palm sized” computer system connected to other computer systems and the Internet via a cradle device.
[0021] FIG. 2A is a top side perspective view of a palmtop computer system that can be used as a platform for the data entry and authentication embodiments of the present invention.
[0022] FIG. 2B is a bottom side perspective view of the palmtop computer system of FIG. 2A .
[0023] FIG. 3 is an exploded view of the components of the palmtop computer system of FIG. 2A .
[0024] FIG. 4 is a perspective view of the cradle device for connecting the palmtop computer system to other systems via a communication interface.
[0025] FIG. 5 is a logical block diagram of the palmtop computer system in accordance with an embodiment of the present invention.
[0026] FIG. 6 is a front view of a palm top computer system illustrating the display screen, digitizer regions and an exemplary menu in accordance with the present invention.
[0027] FIG. 7 illustrates a display screen showing a data entry window and a virtual keyboard window and also illustrates a handwriting recognition digitizer on the palmtop computer system in accordance with embodiments of the present invention.
[0028] FIG. 8 illustrates a three dimensional logical diagram of a volume used for performing stroke recognition with associated pressure data in accordance with an embodiment of the present invention.
[0029] FIG. 9 illustrates an exemplary sample of (x, y) coordinate (“spatial”) stroke data along with associated pressure data in accordance with an embodiment of the present invention.
[0030] FIG. 10 is a table illustrating spatial stroke data and pressure ranges for recognizing lower case alphabetic characters in accordance with an embodiment of the present invention.
[0031] FIG. 11 is a table illustrating spatial stroke data and pressure ranges for recognizing upper case alphabetic characters in accordance with an embodiment of the present invention.
[0032] FIG. 12 is a table illustrating spatial stroke data and pressure ranges for recognizing case-less numeric characters in accordance with an embodiment of the present invention.
[0033] FIG. 13A is a flow diagram illustrating steps in a computer implemented method for using (x, y) coordinate stroke data along with associated pressure data for performing character recognition in accordance with one embodiment of the present invention.
[0034] FIG. 13B is a flow diagram illustrating steps in a computer implemented method for using (x, y) coordinate stroke data along with associated pressure data for identifying a character set and a character in accordance with one embodiment of the present invention.
[0035] FIG. 14 is a flow diagram illustrating steps in a computer implemented method for using (x, y) coordinate stroke data along with associated pressure data for performing user, e.g., signature, authentication in accordance with one embodiment of the present invention.
[0036] FIG. 15 illustrates a display of a palmtop computer system running a drawing or word processing application and using pressure related information for selecting an object width.
DETAILED DESCRIPTION OF THE INVENTION
[0037] In the following detailed description of the present invention, a method and system for using (x, y) stroke data with associated pressure data for performing improved handwriting recognition, user authentication and also handwriting-based data entry in a computer system, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be recognized by one skilled in the art that the present invention may be practiced without these specific details or with equivalents thereof. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present invention.
Notation and Nomenclature
[0038] Some portions of the detailed descriptions which follow are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits that can be performed on computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer executed step, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
[0039] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as “accessing” “processing” or “computing” or “translating” or “calculating” or “determining” or “scrolling” or “displaying” or “recognizing” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Exemplary Palmtop Computer System Platform
[0040] FIG. 1 illustrates a system 50 that can be used in conjunction with the recognition systems of the present invention that are based on using spatial coordinate stroke data with associated pressure data. System 50 comprises a host computer system 56 which can either be a desktop unit as shown, or, alternatively, can be a laptop system 58. Optionally, one or more host computer systems can be used within system 50. Host computer systems 58 and 56 are shown connected to a communication bus 54, which in one embodiment can be a serial communication bus, but could be of any of a number of well known designs, e.g., a parallel bus, Ethernet Local Area Network (LAN), etc. Optionally, bus 54 can provide communication with the Internet 52 using a number of well known protocols.
[0041] Importantly, bus 54 is also coupled to a cradle 60 for receiving and initiating communication with a palm top (“palm-sized”) portable computer system 100 of the present invention. Cradle 60 provides an electrical and mechanical communication interface between bus 54 (and anything coupled to bus 54) and the computer system 100 for two way communications. Computer system 100 also contains a wireless infrared communication mechanism 64 for sending and receiving information from other devices.
[0042] FIG. 2A is a perspective illustration of the top face 100a of one embodiment of the palmtop computer system of the present invention. The top face 110a contains a display screen 105 surrounded by a bezel or cover. A removable stylus 80 is also shown. The display screen 105 is a touch screen able to register contact between the screen and the tip of the stylus 80. The stylus 80 can be of any material to make contact with the screen 105. The top face 100a also contains one or more dedicated and/or programmable buttons 75 for selecting information and causing the computer system to implement functions. The on/off button 95 is also shown.
[0043] Importantly, FIG. 2A also illustrates a handwriting recognition pad or “digitizer” containing two regions 106a and 106b. Region 106a is for the drawing of alpha characters therein for automatic recognition (and generally not used for recognizing numeric characters) and region 106b is for the drawing of numeric characters therein for automatic recognition (and generally not used for recognizing numeric characters). The stylus 80 is used for stroking a character within one of the regions 106a and 106b. The stroke information is then fed to an internal processor for automatic character recognition. Once characters are recognized, they are typically displayed on the screen 105 for verification and/or modification.
[0044] The digitizer 106 (106a and 106b) records both the (x, y) coordinate value of the current location of the stylus and also simultaneously records the pressure that the stylus exerts on the face of the digitizer pad. The coordinate values (spatial information) and pressure data are then output on separate channels for sampling by the processor 101 (FIG. 5 ). In one implementation, there are roughly 256 different discrete levels of pressure that can be detected by the digitizer 106. Since the digitizer's channels are sampled serially by the processor, the stroke spatial data are sampled “pseudo” simultaneously with the associated pressure data. In an alternate embodiment, the channels could be sampled simultaneously, converted and then read back separately in either a parallel or serial manner. The sampled data is then stored in a memory by the processor 101 (FIG. 5 ) for later analysis.
[0045] FIG. 2B illustrates the bottom side 100b of one embodiment of the palmtop computer system of the present invention. An optional extendible antenna 85 is shown and also a battery storage compartment door 90 is shown. A communication interface 108 is also shown. In one embodiment of the present invention, the communication interface 108 is a serial communication port, but could also alternatively be of any of a number of well known communication standards and protocols, e.g., parallel, USB, SCSI, Firewire (IEEE 1394), Ethernet, etc. It is appreciated that interface 108 can also be used for charging current when using rechargeable batteries.
[0046] FIG. 3 is an exploded view of the palmtop computer system 100 in accordance with one implementation. System 100 contains a front cover 210 having an outline of region 106 and holes 75a for receiving buttons 75b. A flat panel display 105 (both liquid crystal display and touch screen) fits into front cover 210. Any of a number of display technologies can be used, e.g., LCD, FED, plasma, etc., for the flat panel display 105. The touch screen can be a digitizer. A battery 215 provides electrical power. A contrast adjustment (potentiometer) 220 is also shown. On/off button 95 is shown along with an infrared emitter and detector device 64. A flex circuit 230 is shown along with a PC board 225 containing electronics and logic (e.g., memory, communication bus, processor, etc.) for implementing computer system functionality. The digitizer pad can be part of the display assembly or it can also be included in PC board 225. A midframe 235 is shown along with stylus 80. Position adjustable antenna 85 is shown.
[0047] A radio receiver/transmitter device 240 is also shown between the midframe and the rear cover 245 of FIG. 3 . The receiver/transmitter device 240 is coupled to the antenna 85 and also coupled to communicate with the PC board 225. In one implementation, the Mobitex wireless communication system is used to provide two way communication between system 100 and other networked computers and/or the Internet via a proxy server. In other embodiments, TCP protocol can be used.
[0048] FIG. 4 is a perspective illustration of one embodiment of the cradle 60 for receiving the palmtop computer system 100. Cradle 60 contains a mechanical and electrical interface 260 for interfacing with communication interface 108 (FIG. 2B ) of computer system 100 when system 100 is slid into the cradle 60 in an upright position. Once inserted, button 270 can be pressed to initiate two way communication between system 100 and other computer systems coupled to electrical interface cable 265.
[0049] FIG. 5 illustrates circuitry of computer system 100, some of which can be implemented on PC board 225. The computer system 100 can be used to perform character recognition processes and authentication of the present invention, e.g., processes 600 and 640 (FIG. 13A and FIG. 13B ) and process 650 (FIG. 14 ). Computer system 100 includes an address/data bus 99 for communicating information, a central processor 101 coupled with the bus for processing information and instructions, a volatile memory 102 (e.g., random access memory RAM) coupled with the bus 99 for storing information and instructions for the central processor 101 and a non-volatile memory 103 (e.g., read only memory ROM) coupled with the bus 99 for storing static information and instructions for the processor 101. Computer system 100 also includes an optional data storage device 104 (e.g., memory stick) coupled with the bus 99 for storing information and instructions. Device 104 can be removable. As described above, system 100 also contains a display device 105 coupled to the bus 99 for displaying information to the computer user. PC board 225 can contain the processor 101, the bus 99, the ROM 103 and the RAM 102.
[0050] Also included in computer system 100 of FIG. 5 is an alphanumeric input device 106 which in one implementation is a handwriting recognition pad (“digitizer”) having regions 106a and 106b (FIG. 2A ), for instance. Device 106 can communicate information (spatial data and pressure data) and command selections to the central processor 101. System 100 also includes an optional cursor control or directing device 107 coupled to the bus for communicating user input information and command selections to the central processor 101. In one implementation, device 107 is a touch screen device incorporated with screen 105. Device 107 is capable of registering a position on the screen 105 where the stylus makes contact and the pressure of the contact. The display device 105 utilized with the computer system 100 may be a liquid crystal device, cathode ray tube (CRT), field emission device (FED, also called flat panel CRT) or other display device suitable for creating graphic images and alphanumeric characters recognizable to the user. In the preferred embodiment, display 105 is a flat panel display.
[0051] Signal communication device 108, also coupled to bus 99, can be a serial port for communicating with the cradle 60. Device 108 can also include an infrared communication port.
[0052] FIG. 6 is a front view of the palmtop computer system 100 with a menu bar 305 open displaying a pull down window. Also shown are two regions of digitizer 106a and 106b. Region 106a is for receiving user stroke data (and pressure data) for alphabetic characters, and typically not numeric characters, and region 106b is for receiving user stroke data (and pressure data) for numeric data, and typically not for alphabetic characters. Physical buttons 75 are also shown. Although different regions are shown for alphabetic and numeric characters, the present invention is also operable within a single region that recognizes both alphabetic and numeric characters.
[0053] FIG. 7 is a front view of the palmtop computer system 100 with several display windows open on screen 105 including a virtual keyboard window 315. The user has the choice of displaying a virtual keyboard image on display 105 and entering characters by selecting one of the displayed characters of the virtual keyboard image. Window 310 is part of a generic application program executing on system 100. Window 310 is a data entry window within which the generic application program is requesting information to be entered by a user. The information requested and the application program could be any information and any program. Typically, data entry window 310 has at least one data entry field 312 for accepting character data therein. When not entering data using the virtual keyboard 315, the user can stroke a character within pad 106 (either region 106a or 106b) or on screen 105. The recognized character is then also placed into the displayed data entry field for user verification and use. This process can be repeated.
[0054] It is appreciated that, in one embodiment, the digitizer region 106a and 106b is separate from the display screen 105 and therefore does not consume any display area.
Using (x, y) Spatial Stroke Data and Associated Pressure Information for Improved Handwriting Recognition Data Entry and User Authentication
[0055] As described above, the digitizer 106 of computer system 100 provides both pressure data and also (x, y) spatial stroke data when stylus 80 makes contact with the digitizer pad 106. It is also appreciated that screen 105 also contains a digitizer associated with it and capability of providing the same type of information as provided by digitizer 106. Therefore, the screen 105 also provides both pressure data and also (x, y) spatial stroke data when stylus 80 makes contact with the screen 105. Either digitizer can be implemented using well known devices, for instance, using the ADS-7846 device by Burr-Brown that provides separate channels for spatial stroke information and pressure information. The embodiments of the present invention described below, make use of both pressure data and (x, y) spatial stroke data (from either digitizer) in performing various functions, e.g., character recognition, character set differentiation, character size differentiation, graphic element size differentiation and user authentication.
[0056] FIG. 8 illustrates an exemplary format 430 in which digitizers of the present invention provide both pressure data and also (x, y) spatial stroke data. Discrete units of time, T1-TN, are shown in row 438. Associated with each discrete unit of time, is a discrete x-coordinate shown in the x-channel 432. Also associated with each discrete unit of time, is a discrete y-coordinate shown in the y-channel 434. In addition, associated with each discrete unit of time, is a discrete pressure data shown in the p-channel 436. Each of the channels can be sampled by processor 101 over bus 99 (FIG. 5 ). A particular sample, e.g., at time T4, represents three respective data pieces that are associated with the sample period, e.g., an x-coordinate (X4), a y-coordinate (Y4) and a pressure value (P4). The (x, y) data is “spatial” stroke data because it indicates a position on the digitizer where contact is made with the stylus 80. The pressure channel contains “pressure data.” Data of a same sample period are said to be associated. In one implementation, the processor 101 samples the channels in series (not in parallel) and therefore while the respective data pieces for a given sample period are sampled at a high rate of speed, they are not sampled purely simultaneously. Rather, they are sampled “pseudo simultaneously.” In one implementation, all of the spatial data and pressure data of FIG. 8 are digital values.
[0057] FIG. 9 and FIG. 13A illustrate a diagram and flowchart of one manner in which pressure data can be used along with spatial stroke data in performing handwriting recognition. Volume 410 of FIG. 9 represents a three dimensional map. Volume 410 is referenced by three dimensions which can be associated with the y-coordinate 412, the x-coordinate 414 and the pressure data 416. Volume 410 can contain a plurality of sub-volumes, each representing a different character. One exemplary sub-volume 418 is shown. Sub-volume 418 represents the volume defined for an exemplary character, “q.” A vector 420 or radial projection (obtained from spatial stroke and pressure data) can be placed into the volume 410. Character recognition is then performed by identifying the sub-volume to which the radial projection points. Handwriting recognition using a vector and volume is described in a work entitled, “Introduction to Radial Basis Function Networks,” by Mark J. L. Orr, published by the Centre for Cognitive Science, University of Edinburgh, 2, Buccleuch Place, Edinburgh EH8 9LW, which is incorporated herein by reference as background to illustrate the state of the art.
[0058] A handwriting recognition process 600 using pressure data, spatial stroke data, a volume 410 and a radial projection is shown in FIG. 13A . The process 600 commences when the stylus 80 or pen is detected as touching the digitizer, step 605. During the user-drawn stroke, steps 610, 615 and 620 capture the spatial stroke data and associated pressure data and store this information in memory. This continues until the stylus 80 is detected as no longer making contact with the digitizer, step 625.
[0059] Once spatial stroke data and pressure data for a user-drawn character or stroke is received, it is processed by computer system 100 to produce a single vector 420 that has coordinates matching the volume 410, step 630 of FIG. 13A . The vector 420 is then mathematically placed into volume 410 to determine the sub-volume to which the vector 420 points, step 635. The spatial stroke data and pressure data (obtained by steps 610-625) is then recognized to be the character associated with the sub-volume to which the vector 420 points. In this exemplary case, the vector 420 points to the sub-volume defined for the character, “q.” Therefore, the spatial stroke data and the pressure data associated with this exemplary user-drawn stroke is recognized as a “q.” Once given spatial stroke data and associated pressure data, it is appreciated that a number of well known character recognition techniques can be modified to analyze this data, including the technique described in U.S. Pat. No. 5,889,888, issued on Mar. 30, 1999 to Michael Albanese and Jeff Hawkins, which is incorporated herein by reference.
[0060] It is appreciated, then, that this embodiment of the present invention utilizes pressure data of the user-drawn stroke in creating the vectors that are then used to determine or “recognize” the characters. A number of various well known handwriting recognition methods can be used in accordance with the present invention, as long as pressure data is used in the character recognition analysis.
[0061] FIG. 10 , FIG. 11 and FIG. 12 illustrate another embodiment of the present invention that utilizes pressure data to determine which of a number of different character sets to select from when recognizing a character. After the character set is determined, then spatial stroke data is used to identify a particular character of the identified character set. The exemplary character sets of upper and lower case are shown below as an example, but any character set differentiation can be done.
[0062] FIG. 10 illustrates a character set definition 445 of lower case alphabetic characters, a-z. The characters are shown in column 450. Associated with each character is the spatial stroke data, column 470, which is used to recognize the character. Also associated with each character is a pressure range, P1, column 460, used to recognize the character set 445. In one embodiment, the pressure range, P1, is a low pressure range but could be of any pressure range.
[0063] Within this same embodiment, FIG. 11 illustrates a character set definition 472 of upper case alphabetic characters, A-Z. The characters are shown in column 475. Associated with each character is the spatial stroke data, column 485, which is used to recognize the character. In this embodiment, spatial stroke data of column 485 is the same as column 470 (FIG. 10 ). Also associated with each character is a pressure range, P2, column 480, used to recognize the character set 472. In one embodiment, the pressure range, P2, is a pressure range that is higher than pressure range P1. Pressure ranges P1 and P2 do not overlap.
[0064] FIG. 12 illustrates a character set 500 of the numeric characters 0-9, column 510. Both pressure ranges P1 and P2, column 515, can be used for this character set since upper/lower case differentiation is not required. Associated with each character is the spatial stroke data, column 520, which is used to recognized the character.
[0065] FIG. 13B illustrates a flowchart of steps in accordance with a handwriting recognition process 640 that uses the character set information of FIG. 10 , FIG. 11 and FIG. 12 . Using the process 640, a user can enter lower case characters using a low pressure stroke and can enter upper case character using a high pressure stroke. For a given character, the stroke is the same, only the pressure differentiates the character set. Alternatively, low pressure can identify upper case and high pressure can identify lower case. Steps 605-625 of FIG. 13B are the same as those described with respect to FIG. 13A . At step 642, the present invention uses the captured pressure data to identify a pressure range, either P1 or P2, that is associated with the pressure data. This can be done using a number of different methods. The maximum pressure value of the pressure data can be used to determine its pressure range. Alternatively, the minimum pressure value of the pressure data can be used to determine its pressure range. The average, or arithmetic mean or any of a number of other data smoothing or manipulation functions can be used to determine the pressure range of the pressure data. In the preferred embodiment, the pressure data is averaged to determine which pressure range, as between P1 and P2, the pressure data belongs in.
[0066] At step 642, after the pressure range is determined, the present invention then determines the character set associated with the identified pressure range. For instance, if the determined pressure range is P1, then the character set of FIG. 10 is selected. Alternatively, if the determined pressure range is P2, then the character set of FIG. 11 is selected. At step 644, the present invention then uses the spatial stroke data obtained from steps 610-625 to determine a particular character of the identified character set, e.g., the character, “q” of the character set of FIG. 10 . Steps 644 and 642 can be done in any order. It is appreciated that a number of well known character recognition techniques can be used to analyze the spatial stroke data, including the technique described in U.S. Pat. No. 5,889,888, issued on Mar. 30, 1999 to Michael Albanese and Jeff Hawkins, which is incorporated herein by reference.
[0067] In addition to selecting between upper and lower case character sets, the process 640 of FIG. 13B can also be used to select between other types of character sets. For instance, pressure data can be used to select between character sets of different character fonts, or character sets of different character sizes, or character sets of different languages, or character sets of different character attributes (e.g., italic, bold, underline, shadow, color selection, superscript, etc.). It is appreciated that the application in which the user is selecting character sets and characters can be any application program that accepts character entry from a user, such as, for example, a graphics drafting program, a computer aided design program, an electronic spread sheet program, a word processor, an Internet browser, etc.
[0068] FIG. 14 illustrates that pressure data and spatial stroke data can be used together for user authentication. Since palm sized computers are portable and readily concealed, they can fall into the hands of unauthorized people. Using spatial stroke data and pressure, a user can write a signature on the digitizer and store the resulting information into memory. This can be used as a key for authentication into the palm sized computer 100. This type of key is very secure because while a person's signature can be found on paper, the pressure made at the different points in the signature will not be known from the paper and therefore very hard to duplicate by the unauthorized user. The rate of which various portions of the signature are written are also recorded by this method (by more or less data points being captured), which can also be used to distinguish authenticity and which cannot be determined by a signature on paper.
[0069] At FIG. 14 , authentication process 650 assumes that a reference signature has already been entered into computer memory. Authentication is triggered at step 655, perhaps by a password application within computer 100 that is executed whenever the user turns on the computer 100 or wants to enter a new password. Authentication triggering is a well known step and can be invoked by a user pressing an on-screen icon or a mechanical button on system 100 (e.g., the power-on button). After authentication is requested, when the stylus 80 meets contact with the digitizer, step 660 is then entered. Steps 660-670 are analogous to steps 610-620 of FIG. 13A and FIG. 13B .
[0070] Data capturing can end at step 675 when the stylus 80 is lifted off of the digitizer or when a special button or icon is pressed by the user. At step 680, computer 100 compares the captured spatial stroke data and pressure data from steps 660-670 against stored spatial stroke data and pressure data from a reference signature (from memory). It is appreciated that a number of well known preprocessing steps can be performed before the comparison to prepare the data for the comparison. At step 685, if the data match is within a preselected threshold, then an authentication determination is made. In other words, if the reference data matches the captured data within a certain percentage, then the signature is determined to be authentic and access to the computer 100 is allowed, otherwise, access is prohibited.
[0071] FIG. 15 illustrates that pressure data along with spatial data can be used to select certain display attributes within a graphics application. For instance, instead of selecting a particular icon or small button on the display screen 105 in order to change the width of a drawn line, the user instead uses pressure. For instance, soft pressure on the stylus to digitizer interface yields a thin line as shown in line 710. Line 710 is a representation of a user stroke drawn on the digitizer, either 106 or directly on the screen 105. A medium level pressure yields a thicker line shown as line 730. Lastly, a very high pressure yields a very thick line 720. Pressure can be used to differentiate a number of different display attributes in addition to object thickness, such as object fading, shading, and dotted or dashed graphics.
[0072] It is appreciated that because the icons and buttons 750 that are provided by the graphics application are small, it may be very hard to select these windows easily. Sometimes the buttons are hard to see and other times the wrong button is accidentally selected. Therefore, the present invention allows the user to utilize pressure on the digitizer to select certain display attributes within the graphics program rather than use the icons 750. This can be applied to any application program that allows character set selection or visual attribute selection, such as the selection between character sizes, character fonts, character attributes (italic, bold, underline, superscript, color selection, shadowing, etc.) and can be applied to a number of different applications, such as word processors, computer aided design, electronic spread sheets, Internet browsers, etc.
[0073] The preferred embodiment of the present invention, a method and system for using (x, y) stroke data with associated pressure data for performing improved handwriting recognition, user authentication and also handwriting-based data entry in a computer system, is thus described. While the present invention has been described in particular embodiments, it should be appreciated that the present invention should not be construed as limited by such embodiments, but rather construed according to the below claims.
Claims
What is claimed is:
1. A method of recognizing stroke-based data entry comprising:
a) accessing spatial stroke data and pressure data captured by a digitizer of a computer system and representing said user-drawn stroke wherein respective pressure data is associated with respective spatial data;
b) storing said spatial stroke data and pressure data into a computer memory wherein pressure data of a first range can represents a first object attribute and pressure data of a second range can represents a second object attribute;
c) determining a user-selectable object attribute, where said attribute is modulated by said pressure data;
d) drawing a representation of said user-drawn stroke on a display screen of said computer system simultaneously with said spatial stroke data being accessed by said digitizer wherein said representation of said user-drawn stroke is drawn with said object attribute as determined at said c); and
e) repeating said a) through said d) until said stroke is complete;
wherein the object attribute.
2. A method as described in claim 1 wherein said first object attribute is a first line width and wherein said second object attribute is a second line width.
3. A method as described in claim 1 wherein said stroke is a line.
4. A method as described in claim 1 wherein said computer system is a palm sized computer system.
5. A method as described in claim 1 wherein said computer system is a portable computer system.
6. A method as described in claim 1 wherein said digitizer is separate in area from said display screen.
7. A secure stroke-based data entry recognition system comprising:
means for accessing spatial stroke data and pressure data captured by a digitizer of a computer system and representing said user-drawn stroke wherein respective pressure data is associated with respective spatial data;
means for storing said spatial stroke data and pressure data into a computer memory wherein pressure data of a first range can represents a first object attribute and pressure data of a second range can represents a second object attribute;
means for determining a user-selectable object display attribute, where said attribute is modulated by said pressure data; and
means for drawing a representation of said user-drawn stroke on a display screen of said computer system simultaneously with said spatial stroke data being accessed by said digitizer wherein said representation of said user-drawn stroke is drawn with said object attribute as determined at said means for determining an object attribute.
8. The secure handwriting-based data entry recognition system as described in claim 7 wherein said first object attribute is a first line width and wherein said second object attribute is a second line width.
9. The secure handwriting-based data entry recognition system as described in claim 7 wherein said stroke is a line.
10. The secure handwriting-based data entry recognition system as described in claim 7 wherein said computer system is a palm sized computer system.
11. The secure handwriting-based data entry recognition system as described in claim 7 wherein said computer system is a portable computer system.
12. The secure handwriting-based data entry recognition system as described in claim 7 wherein said digitizer is separate in area from said display screen.
13. A method of recognizing shape entry, said method comprising the steps of:
accessing spatial stroke data and pressure data captured by a digitizer wherein respective pressure data is associated with respective spatial stroke data;
storing said spatial stroke data and pressure data into a computer memory;
building a set of vectors from the spatial stroke data and associated pressure data;
delineating sets of vectors based on where and when the stylus is placed onto the surface and lifted from the surface, where data is only collected when the stylus is contacting the surface;
performing shape recognition by applying a mathematical model to the set of vectors to identify a recognized shape with a shape set; and
displaying said recognized shape on a display screen of a computer system.
14. The method of claim 13 wherein said shape set includes a circle.
15. The method of claim 13, wherein the mathematical model includes a neural network.
16. The method of claim 15, wherein the neural network is a radial basis function network.
17. The method of claim 1, where the stroke and pressure data is utilizable by any application or routine running on said computing platform.
18. The method of claim 7, where the stroke and pressure data is utilizable by any application or routine running on said computing platform.
19. The method of claim 13, where the stroke and pressure data is utilizable by any application or routine running on said computing platform.
Patent Citations (23)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4318096(A) | 1982-03-01 | Thornburg et al. | Examiner |
| US5062143(A) | 1991-10-01 | Schmitt | Applicant |
| US5128672(A) | 1992-07-01 | Kaehler | Applicant |
| US5434959(A) | 1995-07-01 | Von Ehr et al. | Examiner |
| US5479536(A) | 1995-12-01 | Comerford | Applicant |
| US5533141(A) | 1996-07-01 | Futatsugi et al. | Examiner |
| US5615285(A) | 1997-03-01 | Beernink | Examiner |
| US5724449(A) | 1998-03-01 | Cornerford | Applicant |
| US5724457(A) | 1998-03-01 | Fukushima | Applicant |
| US5730468(A) | 1998-03-01 | Wirtz | Examiner |
| US5734749(A) | 1998-03-01 | Yamada et al. | Applicant |
| US5734750(A) | 1998-03-01 | Arai et al. | Applicant |
| US5818963(A) | 1998-10-01 | Murdock et al. | Applicant |
| US5864635(A) | 1999-01-01 | Zetts et al. | Applicant |
| US5889888(A) | 1999-03-01 | Marianetti, II et al. | Applicant |
| US5896321(A) | 1999-04-01 | Miller et al. | Applicant |
| US5900875(A) | 1999-05-01 | Haitani et al. | Applicant |
| US5940532(A) | 1999-08-01 | Tanaka | Applicant |
| US5963666(A) | 1999-10-01 | Fujisaki et al. | Applicant |
| US6021218(A) | 2000-02-01 | Capps et al. | Applicant |
| US6081621(A) | 2000-06-01 | Ackner | Applicant |
| US6212299(B1) | 2001-04-01 | Yuge | Applicant |
| US6408092(B1) | 2002-06-01 | Sites | Examiner |